GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture

Zaid Qureshi, Vikram Sharma Mailthody (equal contribution), Isaac Gelado, Seungwon Min, Amna Masood, Jeongmin Park, Jinjun Xiong, C. J. Newburn, Dmitri Vainbrand, I-Hsin Chung, Michael Garland, William Dally, Wen-mei Hwu | NVIDIA / UIUC / AMD / University at Buffalo / IBM Research / Stanford | ASPLOS '23, Vancouver, BC, pp. 325–339 | DOI: 10.1145/3575693.3575748

"BaM" stands for Big accelerator Memory.

Problem

Emerging data-center workloads — graph and data analytics, recommender systems, and graph neural networks — operate on massive array datasets (tens of GBs to tens of TBs) that far exceed GPU memory capacity (an A100 has only 80GB despite a 53× growth from G80 to A100). They also need fine-grained, data-dependent access to storage. Existing solutions are CPU-centric: the CPU either tiles the dataset and orchestrates storage→GPU transfers, or services GPU page faults via the OS page-fault handler. Both are bottlenecked by CPU software, CPU–GPU synchronization, OS kernel crossings, long CPU latencies, and I/O traffic amplification (fetching unneeded bytes). The DRAM-only alternative — host memory or pooling many GPUs — is prohibitively expensive (a 10TB GPU pool needs 125 A100s) and still requires preloading data that may go unused. No system architecture or software stack existed for efficient GPU-initiated storage access, and prior GPU-initiated attempts (ActivePointers, GPUfs) reached only ~823K IOPs on an A100 because they still relied on the CPU to serve data.

Core Insight

BaM is the first accelerator-centric architecture letting GPU threads directly initiate and orchestrate on-demand, fine-grained, high-throughput storage accesses — via a highly concurrent GPU-resident software cache plus lock-minimized NVMe submission/completion queues mapped into GPU memory — which eliminates the CPU from the storage control path. It introduces a new GPUDirect Async variant, GPUDirect Async KI (Kernel Initiated) Storage.

Method

BaM presents an mmap-like bam::array abstraction. A GPU thread accessing array[tid] computes the cache line, warp-coalesces with peers via the __match_any_sync primitive (a per-group leader probes the cache, broadcasting via __shfl_sync), and probes a software-defined write-back cache kept entirely in GPU memory. On a hit it reads GPU memory directly; on a miss it locks the line, evicts a victim by a clock replacement algorithm, and issues an NVMe I/O request.

The I/O path uses GPU-resident NVMe submission (SQ) and completion (CQ) queues with lock-minimized enqueue: per-SQ metadata holds local head/tail, an atomic ticket counter, a turn_counter array, a mark bit-vector, and a lock. A thread atomically increments the ticket by two, deriving an entry and turn into a 2³² virtual queue, polls until its turn, writes its command, and sets its mark bit; a single thread then runs move_tail to advance the tail, reset_marks, and ring the doorbell once (coalescing expensive PCIe doorbell writes). Completion is polled lock-free.

The prototype uses a custom Linux driver (character device per SSD), GPUDirect RDMA to pin NVMe queues and I/O buffers in GPU memory for peer-to-peer DMA, and GPUDirect Async + cudaHostRegister to map SSD doorbell registers into the CUDA address space. Queue depth follows Little's Law (T × L = Q_d); for PCIe ×16 Gen4 at 26 GBps, sustaining 51M 512B accesses needs Q_d = 561 for Optane (L = 11 μs) and 16,524 for Samsung 980pro (L = 324 μs).

Experimental Setup

Headline Quantitative Results

Raw throughput (10 Optane SSDs, 512B):

• 45.8M random read IOPs = 22.9 GBps (90% of measured peak Gen4 ×16 bandwidth)
• 10.6M random write IOPs = 5.3 GBps
• A single Optane needs only ~16K–64K GPU threads to reach near-peak IOPs.

Versus GDS and ActivePointers:

• GDS reaches only 23.6% of PCIe bandwidth at 4KB; BaM reaches 25 GBps (peak PCIe) with four SSDs.
• ActivePointers peak miss-handling = 823 KIOPs; BaM = 17 MIOPS (85% of 4-SSD peak) even at 512B lines.
• On hits, BaM delivers up to 430 Gbps — 11.2× ActivePointers' peak; >1 order of magnitude better in both miss handling and hit delivery.

Graph analytics vs host-memory baseline T:

• 4-SSD BaM is storage-I/O bound at 5–6.2M IOPs (>80% of peak); cache overhead 2–15% (1 SSD), 4–45% (4 SSDs).
• Samsung DC PM1735 ≈ Optane; Samsung 980pro 3.21× (BFS) / 2.68× (CC) slower than Optane at 21.8× lower $/GB.

Data analytics (NYC taxi vs RAPIDS v21.12):

• Even single-SSD BaM beats RAPIDS on all queries; Q0 up to 1.22× (no I/O-amplification benefit, pure CPU-overhead win).
• Advantage grows from Q1→Q5 as on-demand fetch avoids the column-transfer amplification RAPIDS incurs.
• Scales 1.46× (2 SSDs) and 1.62× (4 SSDs) over single-SSD BaM; up to 5.3× over RAPIDS with 4 SSDs.

Cost: BaM gives a 4.3–21.8× cost-per-GB advantage over DRAM-only; on-par with a host-memory solution up to 21.7× more expensive.

Limitations

• No overlap of read-miss handling with write-back, exposing full write latency and making write-intensive vectorAdd 1.51× slower than the tiling baseline.
• GPUDirect RDMA write-ordering needs either a dependent I/O request (100% overhead) or a coalesced shared-queue scheme (<8% overhead).
• Cache overhead (4–45% with four SSDs) from metadata contention, long-latency atomics, and warp scheduling among polling threads.
• Four-SSD BaM is bounded by I/O request generation rate (5–6.2M IOPs); beyond ~80% of peak, adding SSDs doesn't help without application changes.
• No crash-consistency guarantees unless the application self-checkpoints; CPU–GPU sharing across two BaM caches needs application-managed synchronization.
• Higher per-thread register usage (up to 255 for RAPIDS Q5) causes register spilling in RAPIDS, though not performance-limiting since workloads are I/O bound.
• The prototype needs a custom PCIe expansion chassis to match ×16 bandwidth; the provided artifact system is limited to ~2 SSDs.

Open Problems

1. Enable asynchronous write-back to overlap read-miss handling with write-back and hide write latency.
2. Application-level work-assignment/scheduling changes to trigger BaM cache misses earlier and improve I/O-request generation rate and SSD scaling.
3. Use emerging disaggregated storage hardware components for very large real-world datasets.
4. Broader adoption of zero-trust / trusted-service security models to secure user-level storage queues at data-center scale.