Quantifying Performance Gains of GPUDirect Storage

Devasena Inupakutika, Bridget Davis, Qirui Yang, Daniel Kim (Samsung, San Diego) · David Akopian (UTSA, San Antonio) | 2022 IEEE International Conference on Networking, Architecture and Storage (NAS) | DOI: 10.1109/NAS55553.2022.9925516

Problem

As data-intensive ML and large-scale simulations shift computation from CPUs to GPUs, the IO path between storage and GPU memory has become a new bottleneck. World data is projected to grow from 33 ZB (2018) to 175 ZB by 2025. In the traditional architecture, devices cannot touch GPU memory directly: data must be staged through a CPU bounce buffer in host RAM (and often the kernel page cache) before it reaches the GPU. This lengthens the IO path, burns CPU cycles on transfers the CPU does not need to compute on, raises latency, and turns a compute-bound problem into an IO-bound one — degradation that worsens in supercomputers and multi-GPU systems.

Core Insight

NVIDIA GPUDirect Storage (GDS), an extension of GPUDirect RDMA from the MAGNUM IO family, establishes a direct DMA path between NVMe (local or remote) storage and GPU memory via the cuFile API, skipping the CPU bounce buffer entirely. The paper's contribution is to quantify that direct path: it establishes a baseline GDS performance level and measures throughput, IOPS, latency, and CPU utilization for both local direct-attached storage and a Weka remote disaggregated cluster, across synthetic random-read and real ResNet50 training workloads.

Method

WITHOUT GDS, the read path is open → malloc → pread → cudaMalloc → cudaMemcpy(HostToDevice) (NVMe → CPU → RAM → GPU). WITH GDS, files open with O_DIRECT and cuFileRead(fh, d_buf, size, 0, 0) moves data NVMe → PCIe switch → GPU memory with no bounce buffer. The GDS software stack runs cuFile / libcufile.so in user space over CUDA, the nvidia-fs.ko kernel driver on top of the Linux VFS, then filesystem/block/storage drivers down to the SSD DMA engine; GDS connects GPU and storage DMA over PCIe using BAR addressing (as in GPUDirect RDMA). Two configurations are compared throughout: cuda_io=cufile (GDS path) vs. cuda_io=posix (traditional CPU-GPU path). Benchmarks use GDSIO (ships with CUDA 11.4, libcufile engine) and FIO 3.26-59; DL uses the DALI pipeline with a TensorFlow reader. SSDs are preconditioned twice (sequential 128K + random 4K writes) to steady state before each run.

Experimental Setup

Headline Quantitative Results

Local direct-attached storage (single NVMe, EXT4):

• Throughput: +14% with GDS for medium-to-high transfer sizes; GDS beats CPU-GPU in all cases in Fig. 4(d).
• IOPS: ~800K out-of-box (≈900K theoretical for PM9A3), decreasing with larger IO size; 4K random-read IOPS higher with GDS.
• CPU utilization: -18% at 32K–256K (2-core advantage); -12% overall.
• ResNet50 training: throughput +1–4% (batch 4–16), +12–15% (batch≥32); CPU savings 5–10 cores (batch 4–32) and ~15 cores (batch≥64); latency >~1000 µs lower from 16K block size and beyond.

Weka remote storage:

• ~2.8x throughput advantage with GDS at 4K block size.
• WekaFS nearly saturates the 100-Gbit link: ~11 GiB/s peak read to a single client (independent of 2/4/6 cores); >740,000 IOPS to a single DSS8440-GDS system (4-core mount; IOPS rise from ~400K at 2 cores to ~740K at 4 cores; ~675K at 6 cores).
• Latency: 45% lower overall; 88% / 49% lower vs. CPU-GPU at 32K / 256K block sizes; lowest 256 µs at 4K (kernel bypassed).
• CPU utilization: ~2–3x lower (-18–21%) for ≥32K block sizes (helped by Weka's user-space container + DPDK networking).
• ResNet50 training: throughput +2–8% (batch≤16), +12–15% (batch≥32); CPU utilization -39–45% for batch≥32; GPU utilization comparable to CPU-GPU.

Cross-cutting: GPU utilization shows no noticeable improvement under GDS in any workload — gains are concentrated in CPU offload, latency, and throughput at medium-to-high transfer sizes (≥32K). At small IO sizes, GDS read rates can actually dip because of the overhead of querying the CUDA runtime for the GPU buffer pointer.

Limitations

• GPU utilization is unimproved by GDS; the benefit is CPU offload + latency + throughput, not GPU efficiency.
• Benefit is concentrated at ≥32K transfer sizes; at small IO sizes GDS can be slower due to CUDA-runtime buffer-pointer lookup overhead.
• Limited scaling: adding processes/threads across multiple GPUs does not help random read (gains come from threads on a single device); more than 3 GPUs gave no random-read improvement.
• Single-client testbed on one DSS8440 and a single 100-Gbit IB link; multi-client linear scaling is expected but not measured.
• Only EXT4 (local) and WekaFS (remote) file systems and one DL model (ResNet50 v1.5) / one synthetic pattern (4K random read) are evaluated.

Open Problems

1. A methodology to tune application-specific optimal GDS configurations (IO threads, client-mount cores, block size, queue depth).
2. Extend evaluation to MLPerf benchmark models beyond ResNet50.
3. Quantify GDS on other HPC workloads beyond DL training and synthetic random read.
4. Evaluate GDS over additional remote backends (Lustre, NFS, S3 on WekaFS).