Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication — Detailed Summary

Yijun Ma, Siyuan Shen, Tiancheng Chen, Akhil Langer, Jiri Kraus, Benjamin Glick, Craig Belusar, Jeff Hammond, Torsten Hoefler | ETH Zürich / NVIDIA Corporation | Preprint, arXiv:2606.05951v1 [cs.DC], 4 Jun 2026 | (* Ma and Shen contributed equally)**

Per-section summary organized by the paper's headings. Each section includes paragraph-level bullet points and exact quantitative results where the paper provides them. The study targets NVSHMEM version 3.3.9 throughout.

Abstract

• NVSHMEM is NVIDIA's OpenSHMEM-based PGAS communication library for GPU clusters; it enables GPU-initiated, one-sided communication through symmetric memory.
• Despite growing adoption, a system-level understanding of its design and behavior remains scattered across documentation, source code, and application experience.
• The paper presents a concise study of NVSHMEM's programming model, implementation, and performance characteristics, focusing on symmetric memory, one-sided operations, and device-side collectives.
• DeepEP is examined as a case study of NVSHMEM in performance-critical sparse deep learning workloads.
• The analysis argues NVSHMEM pioneered a device-side symmetric-memory programming model that enables fine-grained GPU-driven communication and is important for approaching hardware performance limits.
• The work defines NVSHMEM's role as a systems building block, highlights its design tradeoffs, and identifies opportunities for improving GPU communication runtimes.

I. Introduction

• Distributed deep learning and GPU-accelerated scientific computing make GPU communication a central challenge. Distributed DL relies on data, tensor, and pipeline parallelism; HPC applications use distributed decompositions to scale beyond a single device. Performance depends on data exchange over PCIe, NVLink, and InfiniBand. NCCL emerged as the most widely used solution and is the recommended backend for PyTorch, Megatron-LM, and vLLM.
• NCCL was originally host-driven: the CPU enqueues collectives and the library selects algorithms and schedules. This is effective for bulk-synchronous collectives but less suitable for fine-grained point-to-point communication — data-dependent patterns, subsets of threads, or cases where CPU coordination causes unacceptable latency (stencil halo exchange, irregular graph processing, sparse / expert-parallel workloads, custom compute-communication kernels).
• NVSHMEM addressed this gap with a complementary programming model exposing a PGAS directly to GPU code, letting CUDA kernels access symmetric memory on remote PEs via one-sided put/get, atomics, and explicit synchronization. It complements rather than replaces NCCL, enabling irregular exchanges and fine-grained overlap.
• As NVSHMEM proved successful, NCCL incorporated related ideas through its new device API (device-initiated operations + symmetric memory). The two target different abstractions: NVSHMEM provides a flat remote-memory view, while NCCL makes the scale-up/scale-out distinction explicit. NCCL's device API also addresses an NVSHMEM limitation — lack of hierarchical composition through communicators. NVSHMEM supports teams, but its OpenSHMEM roots still expose a single symmetric heap shared across all GPUs in an instance.
• Many key NVSHMEM behaviors are not apparent from the API alone. Research questions: How does NVSHMEM realize symmetric memory on modern GPUs? Which communication paths avoid CPU involvement, and when? What mechanisms underlie its collectives? The paper answers via source-level analysis, tracing design from programming model to runtime internals and transport. NCCL comparisons clarify design differences rather than provide an official ranking.
• Scope: the analysis primarily targets NVSHMEM v3.3.9. Future releases may change features, but core architectural components (symmetric memory, communication paths) are fundamental and likely stable, so the insights should remain relevant.

II. Background

II.A Essential Concepts

• NVSHMEM builds on three ideas: limitations of host-initiated communication, the motivation for device-initiated communication, and the PGAS/OpenSHMEM model based on symmetric memory and one-sided operations.
• NCCL is the standard multi-GPU library but its conventional interface is largely host-initiated: communicator setup and collective launch are CPU-performed, scheduling NCCL kernels onto CUDA streams. Optimized for regular collectives, but keeps the host in the control path, introducing coordination overhead between computation and communication.
• This motivates device-initiated communication, where communication and synchronization launch directly from GPU code, reducing CPU involvement and enabling finer-grained overlap — especially important for latency-sensitive GPU workloads. It motivates interfaces like NVSHMEM that support GPU-side primitives and GPU-initiated network operations that bypass CPU proxy threads.
• Peer-to-peer (P2P) memory access is one GPU directly accessing another GPU's memory (distinct from point-to-point communication and from peer-to-peer network architectures). It enables low-overhead remote access within a scale-up domain — usually node-level NVLink/NVSwitch, or a multi-node NVLink (MNNVL) domain. GPUDirect RDMA (GDRDMA) extends P2P to scale-out domains via an RDMA-capable NIC. GPUDirect Async Kernel-Initiated (GDA-KI) lets GPU kernels initiate and control network operations without CPU intervention. InfiniBand GPUDirect Async (IBGDA) is the InfiniBand-specific realization of GDA-KI.
• Partitioned Global Address Space (PGAS) exposes a logically shared address space partitioned across processing elements (PEs); it underlies OpenSHMEM, which NVSHMEM adapts for GPUs. It is closely related to MPI-3 RMA one-sided communication, but OpenSHMEM/NVSHMEM provide a lighter-weight call path by encoding operation properties (data type, transfer size, operation kind) directly into the API name, whereas MPI RMA passes these as arguments. The SHMEM-style interface reduces dispatch and argument-processing overhead — useful for fine-grained communication.
• In NVSHMEM, a PE is an OS process mapped to one GPU; since v2.4.1 there is limited support for multiple PEs per GPU. Communication is built on symmetric memory — a symmetric heap on each PE plus symmetric objects allocated with the same type, size, and layout across all PEs. Remote PEs access objects via one-sided put/get; the initiator specifies source, destination, and location without explicit receives, enabling asynchronous progress and often direct placement into the destination's symmetric buffer.

II.B GPU Communication Libraries

• CUDA-aware MPI lets MPI routines operate directly on GPU buffers, avoiding host staging; effective for porting HPC apps while preserving rank-based two-sided semantics, but remains fundamentally host-initiated.
• NCCL is the main alternative to NVSHMEM, optimized for dense/regular collectives; the natural baseline for distributed DL and bulk-synchronous workloads, but host-centric. Recent NCCL Device API work reduces the gap via GPU-initiated communication, but its AI-ecosystem use remains collective-centric. NVSHMEM was designed around fine-grained one-sided communication and currently has functionality not in NCCL's GPU-Initiated Networking (GIN) interface.
• UPC++ and GASNet-EX provide portable PGAS-style one-sided access to a partitioned address space; their portability and C++ integration suit host-driven HPC. The key distinction is where communication is initiated — UPC++/GASNet-EX can support GPU transfers via device-aware transports but the control path is still typically host-managed, whereas NVSHMEM exposes device-callable operations so CUDA threads, warps, or thread blocks initiate communication directly.
• rocSHMEM (AMD ROCm) and Intel SHMEM (oneAPI/SYCL) bring OpenSHMEM-style GPU one-sided communication to other vendors, showing device-initiated PGAS is a broader trend, not unique to NVSHMEM. Each is shaped by its own compiler stack, memory model, interconnects, and transports. GICC implements GPU-initiated communication for the Slingshot network using triggered operations.

III. NVSHMEM Overview

III.A Teams

• Teams define subsets of PEs that participate in communication. Unlike MPI/NCCL communicators, teams do not contain most communication-related runtime state; they are lightweight handles identifying PE groups and may encode topology information. The default team NVSHMEM_TEAM_WORLD contains all PEs and is used implicitly when a team argument is omitted.
• Teams carry semantic and runtime constraints: a collective on a team involves exactly the PEs in that team, and team-relative operations must translate PE indices correctly. Arbitrary teams can be built at runtime. However, a team cannot safely be used by multiple concurrent collective invocations from the same PE, because the runtime associates a single set of internal resources (synchronization state, scratch storage) with each team.

III.B API and Dual-Interface Design

• The dual host + device interface is a key strength but makes NVSHMEM far more complex than single-control-path libraries. The split is not uniform: some components remain fundamentally host-managed while the core communication path is exposed on both CPU and GPU. APIs are grouped in Table I.
• Naming convention: nvshmem_ is the standard OpenSHMEM API; nvshmemx_ is for extensions (stream-ordered operations, threadgroup-scoped variants); internal helpers use nvshmemi_ (the i denotes internal routines, not public API).
• Within device-side APIs, NVSHMEM distinguishes operations by execution scope. Default APIs are thread-scoped (one CUDA thread issues the operation). Many nvshmemx_ extensions add _warp and _block variants, where the calling warp or block collectively provides the resources to execute one operation.

Table I — NVSHMEM API groups by availability and functionality

• Setup/exit: initialization, bootstrap via MPI or unique IDs, finalization. Primarily host-side; nvshmem_global_exit is the main exception also exposed on device.
• Memory management: allocate/manage symmetric objects (nvshmem_malloc, nvshmem_align, nvshmem_free) plus buffer registration; host-driven since it manipulates global runtime state and collective memory layout.
• Team management: define subgroups and rank-translation. Queries like nvshmem_team_my_pe exist on both host and device; team creation/destruction remain host-managed.
• One-sided RMA: core interface. Typed/untyped put/get, scalar p/g, strided iput/iget, and nonblocking variants. Naming: typed generalized nvshmem_<TYPENAME>_{put,get}; size-specialized nvshmem_{put,get}<SIZE>; untyped nvshmem_{putmem,getmem}. Scalar p/g cover single-element access; iput/iget are strided shorthand. Fully dual-interface; extended with device warp/block-scoped variants and host on-stream forms.
• Atomics: remote read-modify-write (fetch, add). Naming: non-fetching nvshmem_<TYPENAME>_atomic_<op>; fetching nvshmem_<TYPENAME>_atomic_fetch_<op>; plus compare_swap, swap, fetch.
• Memory ordering: nvshmem_fence orders previously issued operations to the same destination PE but does not guarantee completion; nvshmem_quiet is stronger, waiting until prior operations complete and become visible at the destination. Host side adds nvshmemx_quiet_on_stream, placing completion semantics into CUDA stream order.
• Synchronization: wait/test on the value of a symmetric variable, plus signal-based waiting. Less uniformly dual-interface — ordinary wait/test are primarily device-side; the host accesses this mainly via on-stream variants.
• Collectives: team-wide communication and synchronization — Barrier, Sync, Broadcast, AlltoAll, FCollect, Reduce, ReduceScatter (host and device). FCollect is the counterpart of MPI AllGather; NVSHMEM Reduce is the counterpart of MPI AllReduce. Memory-management and team-creation functions are also collective and internally call nvshmem_barrier_all.

IV. Memory Management

• NVSHMEM implements the symmetric heap on top of CUDA's low-level virtual memory management (VMM) API, with alternative backends using system memory or pinned device memory (selectable via environment options). With VMM, the scheme separates virtual-address (VA) reservation from physical allocation: the heap's VA range is reserved eagerly at init; physical pages are committed on demand. Figure 1 illustrates the VA-based heap setup for two nodes (two GPUs and one NIC each); markers 1–5 show the steps for allocating new physical memory.

IV.A Initialization

• During nvshmem_init, NVSHMEM creates a symmetric heap per PE. It chooses the allocation granularity and sets per-PE heap size to the larger of MAX_MEMORY_PER_GPU and the internal runtime memory overhead, rounded up to the granularity. Each PE reserves a contiguous VA range of size p2p_npes_ × heap_size_ via cuMemAddressReserve(), where p2p_npes_ is the number of P2P-reachable GPUs. The first segment is the local heap; remaining equal-sized segments are reserved for peer mappings. A host-side mspace allocator is created without committing physical memory. After transport init, PEs exchange local heap base addresses so cross-node remote access can be established.
• The reserved VA range covers only P2P-mappable peers. For non-P2P peers, NVSHMEM allgathers each PE's heap_base_ into a separate table peer_heap_base_remote_ storing the actual base of every remote heap; the slow path uses this table with transport metadata and remote memory handles.

IV.B Memory Allocation and Registration

• On the reserved VA range, NVSHMEM manages the heap with a simple host-side allocator using three std::map structures: two for free chunks indexed by start and end addresses, and one for allocated chunks indexed by start address. This makes coalescing straightforward (freeing checks both neighbors and merges adjacent free chunks). Allocation uses first-fit: scan the free list in address order, take the first sufficiently large block, split or consume it. All sizes align to NVSHMEMI_MALLOC_ALIGNMENT.
• The allocator is deliberately simple and deterministic; allocation is O(n) in the number of free chunks. Overhead is acceptable since allocations typically run during initialization. Unlike jemalloc, it avoids per-thread arenas, size classes, or other optimizations.
• Physical GPU memory for the heap is allocated only on demand, when a request cannot be satisfied from already-backed VA space. On heap growth, allocate_physical_memory_to_heap() performs: (1) create a physical handle with cuMemCreate(); (2) map the heap subrange with cuMemMap() and install permissions with cuMemSetAccess() for the local and eligible peer GPUs; (3) register the region with P2P transport; (4) insert the region into the mspace allocator; (5) register the region with network transports for non-P2P peers.
• NVSHMEM then records allocation metadata (offset, size), exchanges required handles across PEs, and synchronizes with a barrier.

IV.C Mapping and Remote Address Computation

• NVSHMEM reserves a VA range large enough to cover the heaps of all P2P-reachable GPUs and places each peer heap at a fixed offset within that range, so remote addresses derive from heap-relative offsets.
• It exports the local CUDA memory handle, exchanges handles across PEs, and maps peer memory into the reserved virtual segments. For non-P2P transports, the region is additionally registered with the appropriate network interface.
• Remote address computation follows the same symmetric-offset rule in fast and slow paths; only the per-PE base address used (and whether the result is dereferenceable) differs. In the P2P fast path, with heap_base_ the local heap base and peer_heap_base_p2p_[remote_pe] the remote heap's base in local VA space, for a local symmetric pointer dest_local:

Equation (1):

dest_remote = peer_heap_base_p2p_[remote_pe] + (dest_local − heap_base_)

• NVSHMEM preserves the object's offset within the local heap and applies it to the remote base. For non-P2P GPUs the same rule uses peer_heap_base_remote_[remote_pe], passing the computed address to the transport with the required handle/registration metadata/NIC key. Only the P2P case yields a directly mapped peer virtual address. The device-side fast path uses this mapping for SM-issued loads/stores from within the kernel (distinct from host-initiated CUDA peer copies, which may use async copy engines). On heap growth, device-resident metadata is refreshed so GPU kernels observe the updated layout.

V. One-Sided Communication

• NVSHMEM uses either a fast path or a slow path depending on whether the target peer is P2P-reachable.

V.A Fast Path: Direct GPU Memory Access

• Fast-path eligibility is decided by host-side P2P transport logic (Figure 2 shows the two common cases). NVSHMEM verifies the two PEs are on the same host, identifies the peer as a locally visible CUDA device, and uses cudaDeviceCanAccessPeer to check direct peer access. If available, it queries native GPU atomics; only then does it populate peer_heap_base_p2p_[pe] for that PE.
• Device-side RMA: once the peer heap is directly mapped, device-side RMA reduces to ordinary memory access on the computed remote address (Eq. 1). Scalar helpers nvshmemi_p and nvshmemi_g check the target PE has a mapped heap base, then issue a direct store/load on the remote pointer. Bulk interfaces like nvshmemi_put_threadgroup perform threadgroup-wide memcpy to/from the mapped address.
• Host-side RMA: same mapped address space, but the host orchestrates CUDA copies. nvshmem_put and nvshmemx_put_on_stream delegate to nvshmemi_prepare_and_post_rma, which selects a mapped P2P peer when the heap is locally accessible, using copies like cudaMemcpyAsync. For device-local symmetric buffers it may use helpers like nvshmemi_p2p_rma_optimized, selecting explicit copy kinds for stream-ordered and single-word operations.

V.B Slow Path: IBGDA and Proxy Execution

• Used when the target heap is not directly P2P-mapped — inter-node, and intra-node peers that are not CUDA P2P-accessible. The same offset-based addressing applies but the computed address cannot be dereferenced directly; the operation goes through a network transport (Figure 3). Depending on system support, NVSHMEM uses IBGDA if available, or a host proxy path where a CPU thread executes the request on behalf of the GPU.
• Network transport and connection management: the slow path uses a pluggable remote-transport layer. NVSHMEM can select InfiniBand transports such as IBRC and IBDEVX, plus higher-level backends UCX and libfabric. InfiniBand transports maintain PE-to-PE connectivity through Reliable Connection (RC) queue pairs (QPs); IBGDA additionally supports Dynamic Connection (DC) QPs. This RC-QP description is not universal — on HPE Slingshot, NVSHMEM uses libfabric with the cxi provider, with connectivity through libfabric endpoints; UCX similarly builds PE-to-PE endpoints from exchanged worker addresses. Recent versions expose QP-specific APIs for finer control over QP selection when IB-specific paths are active.
• Device-side RMA (slow path): scalar put/get use nvshmemi_transfer_rma_p and nvshmemi_transfer_rma_g; bulk uses nvshmemi_transfer_rma and nvshmemi_transfer_rma_nbi; signals and atomics use nvshmemi_transfer_put_signal and nvshmemi_transfer_amo_*. Each operation checks whether IBGDA is available; if so, it is offloaded to nvshmemi_ibgda_rma_*, which constructs and posts RDMA work directly from GPU code to the NIC. Otherwise the request is encoded as a work request written into a proxy buffer in host-pinned memory; a dedicated host proxy thread consumes the descriptor and executes via nvshmemi_proxy_rma_p / nvshmemi_proxy_rma_g. The slow path remains GPU-initiated at the API level, but completion is driven by the NIC (IBGDA) or by the host proxy (descriptor queue).
• Host-side RMA (slow path): nvshmem_put delegates to nvshmemi_prepare_and_post_rma, which selects the transport; on the slow path it is a remote network transfer. Off-stream: NVSHMEM selects a remote RMA-capable transport (IBRC or libfabric), constructs local/remote memory descriptors, and posts through the transport's host-side RMA entry point; bulk transfers go through nvshmemi_process_multisend_rma. On-stream: the request is encoded into the proxy command structure and launched on the specified CUDA stream via nvshmemi_proxy_rma_launcher, after which the host proxy thread drives the network operations.
• Host-side slow-path limitations: remote strided RMA and host-side g operations are unsupported, and put_signal is available only through the on-stream proxy path. These reflect that the current remote transport and proxy mechanisms are primarily optimized for contiguous transfers and device-side signaling.

VI. Collective Communication

• NVSHMEM's collectives build on the lower-level components above and implement several algorithms targeting different message sizes and system configurations. Individual algorithms are not detailed; Table II summarizes the designs and reports communication volume and latency (in number of synchronization steps).

Table II — Collective algorithms in NVSHMEM (v3.3.9). M = message size, N = number of PEs, S = ⌈M/B_seg⌉ = number of scratch-limited segments in segmented AllReduce. NVLS-named algorithms rely on NVSwitch and NVLink SHARP (NVLS) for in-network operations.

VI.A pSync Buffer

• Each team owns a pSync ("persistent synchronization") buffer — a symmetric memory region coordinating collectives (and sometimes point-to-point) across PEs. NVSHMEM lays out pSync regions in a strided manner across teams so synchronization states of different teams do not fall on the same cache line. Within a team's pSync region, different collectives are assigned fixed subregions via hardcoded offsets. Some collectives are double-buffered so consecutive invocations alternate buffers and avoid an extra barrier between uses.

VI.B Low Latency (LL) and LL128 Protocols

• To reduce the cost of explicit post-transfer synchronization (a major bottleneck for small messages), NVSHMEM implements LL and LL128 protocols, following a similar design to NCCL. The core idea is coupling data movement with lightweight arrival notification.
• In LL, each data unit is paired with synchronization flags: the sender packs two data elements and two flags into a single 16-byte atomic write; the receiver polls the flags to know when data is ready. LL128 uses the same idea with a larger unit, grouping 120 bytes of data with an 8-byte flag into 128 bytes to improve bandwidth utilization. LL128 is only safe over NVLink because it relies on 128-byte atomic stores, not generally guaranteed on interconnects such as PCIe.
• LL and LL128 require additional pSync storage since the reception buffer must hold both payload and flags. Not all algorithms support these protocols; Table II highlights the ones that do.

VI.C Algorithm Selection

• NVSHMEM selects collective algorithms at runtime. Instead of an explicit analytical model, it uses a rule-based decision tree per collective. Capability checks, datatype/scope constraints, scratch-space availability, and fixed message-size thresholds determine which algorithm is allowed and preferred, with unsupported cases falling back to more general implementations.

VI.D Support for Multiple Cooperative Thread Arrays (CTAs)

• Device-side collective APIs are thread-group collectives (thread-/warp-/ block-scoped), but a single collective invocation is executed by one participating threadgroup, not by multiple CTAs. The absence of public _grid variants is deliberate: a grid-scoped operation would require cross-CTA synchronization inside a running kernel, more expensive than ending the kernel and launching a stream-ordered communication kernel. It is also consistent with the team model — each team owns one set of internal collective resources (pSync), so it cannot safely host multiple concurrent collective invocations without duplication.
• NVSHMEM exposes built-in multi-CTA execution only through host-side _on_stream wrappers, and only for FCollect, AllReduce, ReduceScatter. On the first multi-CTA call, NVSHMEM creates duplicate teams in a per-team team_dups[] array, copies them to device, and assigns each CTA a separate team and payload slice; each duplicate team has its own pSync region. This path is enabled only when NVLS resources are available; otherwise NVSHMEM uses a single CTA or may fall back to NCCL when NCCL support is enabled. Users needing more parallelism typically issue independent block-scoped operations from multiple CTAs or launch a separate communication kernel via an on-stream API.

VI.E Barrier and Synchronization

• Synchronization collectives are implemented separately from the data collectives in Table II. In device code, both nvshmem_sync and nvshmem_barrier invoke a dissemination-style routine over the team's pSync region. At each phase, a PE signals a set of PEs then waits for notifications from another set of partners. The radix k comes from the runtime parameter barrier_tg_dissem_kval, clamped by team size; for block-scoped collectives on fully P2P-connected teams, k may be raised to the team size. Synchronization requires log_k N rounds for a team of N PEs.
• The difference between sync and barrier: barrier additionally enforces operation completion and visibility before the dissemination rounds via quiet or __threadfence_system (depending on the path), and may enforce target-side consistency afterward. Both have the same synchronization depth, but barrier provides stronger ordering and completion guarantees.

VII. Microbenchmarking

• Selected microbenchmark results for one-sided RMA and collectives. Performance evaluation is not the primary focus; results highlight a few performance characteristics.

VII.A Experimental Setup

VII.B One-Sided RMA

• Figure 4 reports device-initiated one-sided RMA for two intra-node P2P-reachable H200 and for one H200 GPU per node (axes = bandwidth; insets = effective latency).
• Intra-node: bulk put reaches 313 GB/s; get peaks at 141 GB/s; scalar p reaches 172 GB/s (many GPU threads issue independent remote stores, buffered and overlapped); scalar g stays below 9 GB/s (each remote load needs the returned value before completing, limiting outstanding operations and preventing deep pipelining). All remain below the 450 GB/s NVLink reference due to address translation, synchronization, work partitioning, and protocol overheads.
• Inter-node (tuned IBGDA): bulk put and get both approach the single-rail IB reference at 48.0 GB/s and 48.2 GB/s; scalar p improves substantially, peaking at 15.6 GB/s; scalar g remains lower at 1.28 GB/s.
• Latency insets: intra-node ~1.8–2.5 µs for bulk put/get and ~1.3–2.2 µs for scalar p/g. With IBGDA, inter-node put/get ~9.4–9.5 µs at 256 B and near 9.7 µs at 64 KiB; scalar p/g ~7.5 µs and 25.3 µs at 256 B. NVSHMEM is most effective for bulk or aggregated write-style RMA; scalar operations should be treated as latency/control primitives and batched when bandwidth matters.
• Figure 4 reference labels: intra-node "NVLink: 450 GB/s" (peak ~313.2); inter-node "IB: 50 GB/s" (peak ~48.2).

VII.C Collective: AllReduce

• Considers only AllReduce — the most performance-critical collective in many applications, with both NVSHMEM and NCCL providing highly optimized implementations. Figure 5 compares intra/inter-node performance. Variants:
  • NVSHMEM [On-stream] — host-side _on_stream path, enabling multi-CTA and NVLS-based algorithms intra-node.
  • NVSHMEM [Device Block] — fully device-initiated single-CTA path.
  • NCCL [Ring] — standard Ring AllReduce.
  • NCCL [NVLS] — RSxLDMC_AGxSTMC symmetric kernel intra-node and NVLS Tree inter-node.
• Intra-node (8 H200 GPUs): the host-side on-stream path reaches 264 GB/s algorithm bandwidth, outperforming the forced NCCL Ring baseline and approaching NCCL's NVLS path at 276 GB/s. The device-side block-scoped path peaks at only 30 GB/s (single CTA). For small messages the device path is latency-competitive: ~3.8–7.1 µs up to 64 KiB, vs ~4.7–8.9 µs for NCCL Ring and 5.6–5.9 µs for NCCL NVLS.
• Inter-node (16 H200 GPUs): both NVSHMEM variants remain below 0.20 GB/s, while NCCL reaches 180 GB/s (Ring) and 252 GB/s (NVLS Tree). NVSHMEM's optimized collective use has focused mainly on NVLS-based algorithms within a node or MNNVL domain. Unlike intra-node, inter-node shows no small-message advantage for NVSHMEM, and its latency grows to milliseconds by 64 KiB while NCCL stays in tens of microseconds. (A few inter-node NVSHMEM measurements are missing due to timeouts.) Inset peak labels: intra ~274.8 (NVLink ref 450 GB/s); inter ~246.4 (ref 400 GB/s for 8×50 GB/s IB NICs).

VIII. Case Study: DeepEP

• Examines real-world NVSHMEM use via DeepEP, an open-source library from DeepSeek for Mixture-of-Experts (MoE) workloads under expert parallelism (EP). In EP, experts of each MoE layer are sharded across GPUs; per layer, tokens are dispatched to router-selected experts and outputs are combined back to the originating GPUs. Both phases are sparse all-to-all exchanges where communication volume between rank pairs is data-dependent, making them hard to implement efficiently with off-the-shelf collectives. DeepEP uses custom dispatch/combine kernels with NVSHMEM as the cross-node RDMA substrate.
• DeepEP exposes a high-throughput (HT) path for training and a low-latency (LL) path for inference. The analysis focuses on DeepEP V1, since the newer V2 backend is based on NCCL GIN and is beyond scope.

VIII.A High-Throughput (HT) Kernels

• The HT path targets training, where large token counts make bandwidth more important than per-token latency. Key idea: a two-stage pipeline — tokens first sent across nodes by RDMA only between GPUs with the same local index, then redistributed within the destination node over NVLink to GPUs hosting the selected experts (Figure 6).
• The HT path uses eight parallel NVSHMEM world teams, one per GPU slot in a node. Each world contains one PE per node, so cross-node RDMA happens only between GPUs with the same local index. It assumes exactly eight P2P-accessible GPUs per node, so the design is portable to other such systems.
• Communication has two phases. First, notify_dispatch exchanges metadata (not payloads): it gathers per-rank and per-expert token counts, computes prefix matrices, and determines how each channel partitions its outgoing traffic. Second, dispatch (and symmetrically combine) moves actual payloads; each dispatched token is packed into a wire format containing hidden state, optional scaling metadata, and routing information.
• The dispatch kernel is organized into logical channels, each spanning a pair of SMs and handling a contiguous slice of input tokens. Within each SM, DeepEP uses warp specialization across its 16 warps. On the RDMA-side SM, 7 warps act as senders, 1 as a sender coordinator, and 8 as NVLink receivers (one per local GPU slot). On the paired SM, 8 warps serve as RDMA-to-NVLink forwarders (one per destination GPU slot) and 8 are forwarder coordinators (only one active in practice).
• During dispatch, sender warps place tokens into per-peer RDMA ring buffers on the symmetric NVSHMEM heap; the sender-coordinator warp periodically batches these writes into larger RDMA transfers via nvshmemi_ibgda_put_nbi_warp, updating the remote tail with nvshmemi_ibgda_amo_nonfetch_add. On receive, the 8 forwarder warps poll for arrivals, decode token metadata, and copy tokens into intra-node NVLink ring buffers; a single forwarder-coordinator warp tracks progress and returns RDMA credits, while the 8 NVLink-receiver warps place tokens into the output tensor.
• The combine kernel mirrors this in reverse: expert outputs are first moved locally over NVLink into RDMA buffers, then sent across nodes, then reduced into the destination tensor on the token's original GPU. Warp roles differ slightly but the overall design is the same.
• A key point: DeepEP does not express most work as NVSHMEM collectives or generic RMA. It uses NVSHMEM only at critical points in the inter-node RDMA path — metadata exchange in notify_dispatch, chunked RDMA puts from sender coordinators, and atomic credit updates for remote tail/head counters. NVSHMEM is the cross-node substrate while DeepEP builds its own multi-stage transport pipeline on top.

VIII.B Low-Latency (LL) Kernels

• The LL path targets inference, where batch sizes are small and end-to-end layer latency matters more than peak bandwidth. It removes the intra-node NVLink forwarding stage of the HT path and relies on RDMA for inter-node delivery.
• Instead of partitioning the cluster into eight independent NVSHMEM world teams, the LL path uses a single global NVSHMEM world team and overlays a strided team for the GPUs with the same slot across nodes.
• The kernel structure is much simpler: no logical channels and no warp specialization. A single kernel grid handles the whole dispatch, and SMs are partitioned by local experts rather than by token range. Within each block, warps are partitioned into fixed-size groups, each responsible for one expert.
• Both dispatch and combine follow this structure. In the send phase, each warp iterates over tokens and handles one top-k destination per warp. After optional FP8 conversion, lane 0 reserves a slot in the destination buffer and computes the target address in the remote receive buffer. If the destination is on another node, the warp sends the packed message with nvshmemi_ibgda_put_nbi_warp; if on the same node and P2P-accessible, it performs a direct local copy into the mapped receive buffer. After all messages for an expert are sent, one atomic update publishes the final token count to the destination.
• In the receive phase, the corresponding warp group waits until the count becomes nonzero, reads the number of received tokens, reserves space in the output buffer for that expert, then copies messages from the receive slots into the output tensor, unpacking source indices and optional FP8 scales. The critical-path NVSHMEM activity is minimal — an IBGDA put for payload transfer and an atomic update for the final count notification.

VIII.C Performance (DeepEP)

• NVIDIA already published a direct comparison of NCCL GIN and NVSHMEM on DeepEP; since this paper aims to explain NVSHMEM rather than add another benchmarking study, those experiments are not repeated. In the GIN paper, DeepEP was integrated with NCCL GIN while the original NVSHMEM-based version was retained as the baseline; both HT and LL dispatch/combine kernels were evaluated. Main takeaway: NCCL GIN matches NVSHMEM closely — typically within about 1–2% — while offering the same style of device-initiated communication inside NCCL's runtime.

IX. Related Work and Discussion

• The closest work is Hu et al., a source-level analysis of NCCL. Among prior works on NVSHMEM itself, the most directly related is Langer et al., which explains dynamic symmetric heap allocation; that study is limited to the memory subsystem, whereas this work analyzes the library more broadly.
• A separate line studies NVSHMEM from applications/runtimes: Hsu et al. evaluate NVSHMEM for usability and functionality; later works use it in systems like GROMACS and CharmNG to improve scaling. More broadly, related GPU-centric programming-model work includes Hamidouche et al. and Unat et al. (design space around GPU-resident communication). This contribution is a detailed study of NVSHMEM rather than a new runtime or full survey.
• One clear takeaway: limited multi-CTA support remains a weakness of NVSHMEM's built-in collective implementation (Section VI-D), while NCCL is narrowing the gap through its device API. However, the GIN paper also shows NVSHMEM still provides a meaningful advantage as a lower-level one-sided RMA substrate. NVSHMEM remains relevant for applications needing fine-grained one-sided communication and direct device-side control.

X. Conclusion

• NVSHMEM occupies a distinct point in the GPU communication design space by enabling GPU-initiated, one-sided communication through a PGAS model. The source-level analysis shows this is implemented through symmetric memory, multiple transport paths, and a layered runtime connecting device-side operations to network mechanisms. NVSHMEM's main limitations today come from its execution model and collective implementations, which often do not fully exploit GPU parallelism. The DeepEP case study highlights its practical value in modern sparse deep learning workloads. Overall, NVSHMEM made an important contribution by exposing a flexible, device-driven substrate for fine-grained data movement, and its design continues to drive the evolution of modern GPU communication libraries.

Limitations (as stated by the authors)

• Limited multi-CTA collective support is the key weakness: built-in multi-CTA execution is gated by NVLS availability and exposed only via host-side _on_stream wrappers for FCollect/AllReduce/ReduceScatter. Device-block collectives are single-CTA (AllReduce only 30 GB/s).
• Host-side slow-path RMA limitations: remote strided RMA unsupported, host-side scalar g unsupported, put_signal only via on-stream proxy path.
• Inter-node collectives underperform drastically (<0.20 GB/s vs NCCL's 180–252 GB/s); optimized collectives focus on NVLS within a node/MNNVL domain.
• Scalar g operations are bandwidth-poor (<9 GB/s intra, 1.28 GB/s inter) due to the load-completion dependency; they should be batched.
• Single shared symmetric heap across all GPUs in an instance (OpenSHMEM roots) limits hierarchical composition vs NCCL's communicator model.
• Analysis pinned to v3.3.9; future releases may change details (authors argue the core architecture is stable). DeepEP V2 (NCCL GIN-based) is out of scope.

Open Problems / Discussion Points

1. NCCL is narrowing the gap via its device API and GIN interface; whether NVSHMEM retains a long-term advantage as a lower-level one-sided RMA substrate is an open question (the GIN paper suggests it still has a meaningful advantage as a low-level substrate).
2. NVSHMEM's execution model does not fully exploit GPU parallelism (single-threadgroup collectives, no public _grid variants) — a design tension between grid-wide synchronization cost and kernel-relaunch cost.
3. Lack of hierarchical composition (single symmetric heap, flat remote-memory view) vs NCCL's explicit scale-up/scale-out distinction.
4. Device-initiated PGAS is a broader vendor trend (rocSHMEM/AMD, Intel SHMEM, GICC/Slingshot) — portability and convergence across compiler stacks, memory models, and transports is an open area.
5. NVSHMEM remains relevant for fine-grained, irregular, data-dependent communication (sparse/MoE/expert-parallel, stencil halo, irregular graph) where collective libraries are inefficient — as shown by DeepEP, which uses NVSHMEM only at critical RDMA-path points while building its own transport pipeline on top.

Note on NCCL Tuning

The paper directly documents NVSHMEM's runtime collective-algorithm choice as a rule-based decision tree keyed on capability checks, datatype/scope constraints, scratch-space availability, and fixed message-size thresholds (Section VI-C) — the same kind of static threshold logic NCCL uses to pick Ring vs. NVLS vs. Tree. The microbenchmarks make the selection stakes concrete: intra-node, the on-stream NVLS path reaches 264 GB/s while the single-CTA device-block AllReduce caps at 30 GB/s, yet that device path is the latency winner for small messages (3.8–7.1 µs up to 64 KiB vs NCCL Ring's 4.7–8.9 µs). It also reaffirms that LL128's 128-byte atomic stores are safe only over NVLink and unsafe on PCIe, a protocol-eligibility constraint that any algorithm/protocol selector must respect when the interconnect changes.