Quantifying Performance Gains of GPUDirect Storage — Detailed Summary

Devasena Inupakutika, Bridget Davis, Qirui Yang, Daniel Kim (Samsung, San Diego) · David Akopian (UTSA, San Antonio) | 2022 IEEE International Conference on Networking, Architecture and Storage (NAS) | DOI: 10.1109/NAS55553.2022.9925516

Per-section summary organized by the paper's headings. Each section includes paragraph-level bullet points and exact quantitative results where the paper provides them. Figures 4 and 5 (the core results) are bar charts with no in-text numeric tables, so figure-derived numbers below are the magnitudes the authors state in prose plus chart-axis values.

Abstract

• Rapid growth in data collection forces a rethink of the underlying computation/storage stack; in data-intensive workloads (ML, large-scale simulation) computation is shifting from CPUs to GPUs.
• Input/output (IO) overhead during data transfer between GPU and storage has become a new performance bottleneck.
• The traditional data-transfer path between GPU memory and device storage uses the CPU as a buffer, which limits the GPU's ability to use its resources; the bounce-buffer approach wastes CPU cycles on transferring data.
• These traditional paths add latency and reduce overall GPU processing performance, especially when the CPU does not need to touch the data.
• NVIDIA GPUDirect Storage (GDS) solves this by establishing a direct path between local NVMe / remote storage and GPU memory.
• The work quantifies high throughput gains, low-latency achievements, and CPU utilization savings of GDS, using Weka as the remote storage cluster.
• They experiment with various workloads and identify which workloads benefit most; they systematically measure synthetic representative data-read workloads and real ML workloads optimized with GDS, then discuss correlated implications for systems using GDS.
• Two demonstration results: (a) a testing methodology for evaluating a GPU client with GDS-supported file systems for local NVMe and Weka; (b) establishing a baseline level of GDS performance for local and remote storage devices.
• Keywords: NVIDIA GPUDirect Storage (GDS), Weka, NVMe Storage, GPU, Workload performance analysis, Deep Learning, GPU I/O.

I. Introduction

Data-growth motivation:

• Data generation rates are huge in next-generation enterprise systems; AI and Big Data place high demands on storage and scale-out compute.
• Survey [1] expects world data to grow from 33 ZB (2018) to 175 ZB by 2025; during this period the storage industry needs to ship 42 ZB of capacity.
• HPC systems increasingly use accelerators (GPUs) to speed up ML data-intensive applications, large-scale simulations, advanced analytics, and visualizations.
• This shift from CPUs to faster GPUs has made IO operations between storage and the GPU more significant.

The CPU bounce-buffer problem:

• GPUs accelerate AI/ML, NoSQL/SQL apps but introduce data-transfer overhead between storage devices and GPUs.
• Devices cannot access GPU memory directly in traditional system architectures; data must be copied to host memory before GPUs/devices can access it, involving the CPU in the initiation of every transfer.
• Traditional reads/writes to GPU memory use POSIX APIs to read/write data from host memory as an intermediate bounce buffer.
• Some file systems also need additional memory in the kernel page cache.
• The IO path becomes longer because CPU cycles are spent moving content to/from GPU memory before read/write; the IO handling load increases on the CPU, and GPU compute nodes wait for IO completion — so application performance suffers.
• Such IO bottlenecks cause performance degradation that grows in supercomputers and multi-GPU systems, turning a compute-bound problem into an IO-bound one; the extra copy also costs GPU IO bandwidth, raising overall CPU utilization and latency.

Path to direct access:

• To address indirect data transfer and enable flexible transfers to/from GPU, CUDA 4.0 was released with Unified Virtual Addressing [3].
• GPUDirect was later proposed to transfer data directly between multiple GPUs without involving CPU bounce buffers, using peer-to-peer (P2P) techniques.
• The latest result is the experimental GPUDirect Storage (GDS) [4] from the MAGNUM IO family.
• GDS connects NVMe SSDs directly to the GPU to enable a direct data path between storage and GPU memory and avoid extra copies through the CPU in host memory.

This paper's scope:

• The paper integrates the latest GDS technology with an IO workload-performance analysis testbed to understand its impact when used with Samsung PM9A3 devices [5] as local storage and as remote storage on the Weka cluster [6] (the reference platform).
• Analysis is in two stages: Phase 1 — single-client GDS workload on direct attached storage; Phase 2 — single-client GDS workload on remote disaggregated storage.
• Results assess GDS impact on both synthetic read/write [7] and real AI/ML workloads.

Contributions (stated):

1. Establish a baseline level of GDS performance and report comparative analyses.
2. Study the impact of GDS on synthetic and real AI/ML workloads.
3. Analyze system resource consumption: device IO operations, CPU utilization, latency, and throughput.
4. Investigate workloads with local direct-attached storage using the EXT4 local file system and Weka distributed file system as remote storage.

Paper organization: Section II — GDS technology, related work, GDS software stack; Section III — testbed (local and remote), workloads, metrics; Section IV — evaluation with synthetic and real AI/ML workloads; Section V — discussion and system implications; Section VI — conclusions and future work.

II. Background

II.A. GPUDirect Storage

Why the CPU is a bottleneck:

• GPUs gain speedup from data parallelism but still rely on the CPU for data transfer and for launching compute kernels; multiple copies of data cause severe performance bottlenecks.
• Fig. 1(a) shows traditional data flow: for a standard ML read operation, data is first transferred from the file system to host memory using CPU cycles, then sent to GPU memory. With CUDA, apps use CudaMemcpy to move data between host (CPU) and GPU memory; some apps also need CPU-side data pre-processing.

GDS as the P2P answer:

• GPUDirect Access enabling a P2P connection between storage and GPU is an active research problem because of GPU data-access delays.
• With NVIDIA GDS there is a direct path between local/remote file system and GPU memory; GDS is an extension of GPUDirect RDMA [8] from MAGNUM IO.
• GPUDirect RDMA uses GPU remote DMA to move data directly between a NIC and GPU memory, providing direct communication between NVIDIA GPUs in remote systems.
• Fig. 1(b): data is sent directly between the storage device file system and GPU memory.
• Sequence WITHOUT GDS: (a) fd = open("data_file", O_RDONLY, ...); (b) buf = malloc(size); (c) pread(fd, buf, size, 0); (d) cudaMalloc(d_buf, size); (e) cudaMemcpy(d_buf, buf, size, cudaMemcpyHostToDevice).
• Sequence WITH GDS: files are opened with O_DIRECT: (a) fh = open("data_file", O_RDONLY | O_DIRECT, ...) — skipping the bounce buffers; (b) cudaMalloc(d_buf, size); (c) cuFileRead(fh, d_buf, size, 0, 0) — avoids copying data to CPU memory.
• Current GDS supports NVIDIA Tesla or Quadro GPUs based on Volta, Turing, Ampere, or Pascal.
• GDS expects a GDS-supported file system with NVMe / NVMe-over-Fabrics (NVMe-oF) storage devices or distributed file systems. If the file system is unsupported, cuFile provides a compatibility mode that delivers GPU IO using the internal CPU bounce buffers.

II.B. Related Work

• GPUDrive [9] integrates an all-flash storage array with the GPU for IO-intensive applications.
• Donard project [10] was an initial P2P method between NVMe and GPU.
• Morpheus [11] focuses on low-level P2P data transfer between SSDs and GPU, which is challenging for a user.
• NVIDIA GPUDirect RDMA [8] transfers data between multiple GPUs using GPU memory mapping to CPU.
• The Dolphin Interconnect Company proposed efficient GPU IO access to distributed storage [12], but the design requires specialized hardware and library constructs.
• Recent work [13] proposes a Spark-GPU platform with direct access to storage, giving 35% performance improvement vs. an architecture involving bounce buffers.
• SPIN [14] uses GPUDirect to exclude the CPU from the data path between GPU and NVMe; implemented in the standard OS file IO stack and aware of page cache and read-ahead for sequential access.
• GDS [4] (NVIDIA) demonstrates the feasibility of enabling direct memory access from storage; the storage DMA engine can be programmed to target GPU memory through the file system, especially useful for large datasets that do not fit entirely in host memory.
• These works show recent research attention to direct GPU↔︎NVMe data paths.
• Manufacturers (Microsoft + Weka [15], Mellanox [16], VAST [17]) have released GDS integration results with their NVMe SSDs; bandwidth improvements of orders of magnitude, reduced latency, and linear scaling with more clients (especially with larger IO transfer sizes) are expected. In some scenarios the compatibility POSIX mode can perform better than the cudaMemcpy GDS path.
• Similar to [13], the authors intend to exploit two of the three stages of performance improvement in big-data processing; to address required computational throughput across workloads, they use GPUs and NVMe SSDs for improved IO-intensive performance plus a distributed parallel file system (Weka) as remote storage.

II.C. GPUDirect Storage Architecture Layout

• Fig. 2 presents the GDS software-architecture layout that lets the GPU access storage directly and lowers data-transfer delay.
• One part is the cuFile user API for applications and frameworks.
• The second part is OS kernel drivers and third-party alternatives: the Nvidia-fs driver API is built on top of the virtual file system (VFS) in the Linux I/O stack, adding mechanisms to enable GPU memory buffers and take advantage of direct file system, block I/O drivers, and disk I/O interface.
• Read and write requests to the SSD devices are handled by this API.
• Vendor proprietary solutions for distributed file systems do not require patching, enabling Linux compatibility.
• In GPUDirect RDMA, all PCIe devices view the same physical address and communicate via reads/writes to a peer device's Base Address Register (BAR) address.
• GDS utilizes a similar technique and connects the GPU to the storage DMA with a PCIe interface like NVMe SSDs.
• Software-stack layers (Fig. 2, top→bottom): Applications on CPU → cuFile API / libcufile.so → CUDA → [kernel boundary] → nvidia-fs.ko kernel driver → Virtual file system → Filesystem driver / Block IO driver / Storage driver (third-party / OS kernel alternatives) → Storage (SSDs) DMA engine; a proprietary distributed file system path connects directly to the storage DMA engine.

III. Experimental Testbed

• Both local and remote storage testbed configurations are used.
• Testbed: a single client machine with GPUs and CPUs as compute, and NVMe drives as storage. The client GPU server is common to both local and remote testbeds.
• Local storage attached to the client is the Samsung PM9A3 device.
• For remote storage, the Weka distributed storage cluster is set up with 6 CPU servers and 4 Samsung PM9A3 NVMe devices per server.

III.A. Local Storage Testbed Configuration

• The client with 8 GPUs is used as the local storage testbed; client configuration is in Table I.

III.B. Remote Storage Testbed Configuration

• Client configuration is similar to Table I. Experiments run on Weka remote storage (in-house cluster at Samsung).
• Rationale for choosing Weka: (a) Weka is a software-designed solution to fully harness fast shared file system performance to feed data-starved applications; (b) it breaks the data dam, eliminating traditional islands of storage so data flows without pipeline constraints; (c) it enables flexible application storage access including POSIX, NFS, S3, SMB, and NVIDIA GDS.
• Remote storage consists of 6 nodes with 4 drives each (Table II).
• The gds rdma write support is enabled for the Weka file system on the client side; the client/host IP address must be added to the cufile configuration file at /etc/cufile.json.
• Fig. 3 shows the remote-storage experimental system (Weka remote storage cluster testbed).

TABLE I — GPU Machine Software and Hardware Configuration

TABLE II — Remote Storage Testbed Configuration

• Fig. 3 cluster detail: Management network 1 GbE; Application/Client Servers (Dell DSS8440, Dual Intel Xeon 8268 @2.90 GHz, 768 GB RAM, 8x Nvidia Tesla V100S, Weka v3.12.0, ConnectX-5 100GbE dual-port NIC); 100 GbE x1 connection to each client; Data network 100 GbE; 100 GbE x1 connection to each target; Weka Target Cluster (6 targets; each Dell R730xd, Dual Intel Xeon E5-2640 v3 @2.60 GHz, 128 GB RAM, 4x PM9A3 disks, Weka v3.12.0, ConnectX-5 100GbE NIC).

III.C. Workloads and Benchmarking Utilities

III.C.1. Microbenchmarks: Synthetic Random Read IO workloads

• Random (small-IO) read workloads are generated using NVIDIA GDS utilities with GDS support enabled and POSIX reads/writes.
• GDSIO is an in-house benchmarking utility shipping with NVIDIA GDS and the latest CUDA toolkit version 11.4; the I/O engine libcufile conducts FIO tests using the NVIDIA GDS cuFile API to perform asynchronous I/O directly against GPU buffers via NVIDIA-fs and a GDS-enabled file system.
• Installing the CUDA toolkit, NVIDIA-fs, and NVIDIA GDS is a prerequisite for GDSIO tests with GDS support.
• Synthetic workloads allow precise control of IO requests w.r.t. read/write mix, block size, random vs. sequential, and queue depth.

III.C.2. Real-world Deep Learning workloads

• DL is an emerging HPC workload; DL must go through a learning phase involving a huge volume of data, and the growing data-to-ingest has shifted the bottleneck from compute to storage.
• To understand GDS's impact on shortening the data path and accelerating DL, they collect performance for ResNet50 training with the ILSVRC12 [19] dataset.
• Given the smaller ImageNet size (~150 GB), it could fit fully in memory; to reduce data-preparation stalls [20], they use the DALI pipeline with GDS integration and a TensorFlow reader to set up DL benchmarking workloads.

TABLE III — Workload Descriptions

III.D. Performance Metrics

• The authors focus on key IO-workload characteristics that affect performance:
  1. I/O Operations (IOPS): how many operations/requests per second; reads per second and writes per second are typically expressed as IOPS for disks.
  2. Throughput: the size of each I/O request and resource availability, reporting how many MB/s the workload performs and how much data really moves in a given period.
  3. I/O Latency: time to complete a disk read or write operation.
  4. CPU Utilization: amount of workload handled by the CPU in % of a processing core's usage; GDS workloads are expected to show lower CPU utilization; the paper specifically studies CPU utilization by user space (USR) and kernel processes (SYS).
  5. GPU Utilization: amount of workload handled by GPU engines in %.

III.E. Precondition

• A preconditioning process is performed twice to ensure the NVMe SSD reaches steady-state performance before each measurement.
• Both Sequential and Random precondition are conducted: the SSD is filled completely with large 128K-granularity sequential and 4K random write IOs, respectively.
• The workloads are generated with the FIO IO testing tool to observe ideal write steady-state performance.
• Around 3.365 GB/s throughput for sequential write and 199K IOPS for random write are observed, achieving the theoretical reported values for Samsung PM9A3.

IV. Results and Analysis

• Before actual workloads, both local and remote SSDs are preconditioned for reliable results.
• Application workloads are characterized between the file system and GPU memory (labels start with GDS_) and the traditional path through CPU buffers (labels start with CPU_GPU_).
• For read workloads, without GDS, applications read data from the file system using a POSIX read call and transfer it explicitly from CPU memory to the GPU memory buffer.

IV.A. Performance on the Local Direct-Attached Storage

• Direct Attached Storage (DAS) is controlled by a single server (server-side); storage devices are NVMe SSDs. This section reports synthetic random read and DL workloads with varying IO transfer sizes for DAS on the client.
• The local Samsung PM9A3 device is mounted with EXT4 file system formatting and logical blocks of 512 bytes.

IV.A.1. GDS performance for Random Read workload (Fig. 4 (a)–(d)):

• Random read varies from 4K to 256K. Observed read rates are higher for medium to higher read sizes (64K to 256K) when enabling O_DIRECT.
• For lower IO sizes, read rates are reduced because of the extra overhead from querying the CUDA runtime for the memory-buffer location (memory-pointer info) inside GDS.
• For larger read sizes with GDS, higher read speeds are achieved. If O_DIRECT is not enabled (traditional CPU-GPU path), the OS will try to read ahead by prefetching and caching data.
• Read latencies with GDS are comparable to CPU-GPU; 4K random read IOPS are higher with GDS.
• Analyses are on pre-conditioned SSDs; out-of-the-box SSD analysis achieved ~800K IOPS (closer to the theoretical reported 900K IOPS for PM9A3 SSD [5]) and decreases with increasing IO sizes.
• From 32K–256K read sizes, 18% lower CPU utilization with GDS vs. CPU-GPU (2 CPU cores advantage with GDS).
• In Fig. 4(d), GDS performs better (throughput) than CPU-GPU in all cases.

IV.A.2. GDS performance for ResNet50 training (Fig. 4 (e)–(h)):

• Analyses of DALI optimizations with GDS integration in the ResNet50 v1.5 model training pipeline; measurements for varying mini-batch sizes (4 to 256) with all 8 V100S GPUs (multi-GPU training) on a single client DSS8440 and ImageNet 2012 on PM9A3 direct-attached storage.
• IOPS vary (decrease) with batch size; overall CPU utilization is reduced with GDS.
• ResNet training with GDS uses 6%–23% (about 5–10 cores) fewer CPU cores for batch sizes 4–32, and 42% (about 15 cores) fewer for batch size 64 and beyond, vs. CPU-GPU.
• GPU utilization shows no noticeable improvement with GDS.
• Training throughput improved ~1–4% for lower batch sizes (4–16) and ~12–15% for batch size 32 and above.
• Read latencies show no significant differences for lower batch sizes; from 16K block size and beyond, with GDS the latency is over ~1000 µsec lower vs. traditional CPU-GPU.

IV.B. Performance on Weka Remote Storage

• With a distributed file system like WekaFS, better performance comes from using multiple GPU IO threads; relevant parameters can be tuned with the GDSIO benchmarking utility.
• Fig. 5 (a)–(h) compare synthetic random read and ResNet50 training rates with the GDS and traditional CPU-GPU paths.
• These experiments use 3 GPUs for the synthetic read workload (optimized configuration) and all 8 GPUs for ResNet50 training; Weka file system mounted to a Dell DSS8440 client with 4 cores. The 4-core mount is observed to perform at the theoretical maximum of the PM9A3 SSDs.

IV.B.1. GDS performance for Random Read workload (Fig. 5 (a)–(d)):

• GDSIO provides the baseline measure of IO capability of the Weka-GDS reference architecture (Fig. 3); Fig. 5 (a)–(d) shows random read performance to Weka with varying block size.
• CPU utilization is about 2–3x lower with GDS for 32K read sizes and higher.
• The Weka-GDS test was conducted with a single DSS8440 GPU system to establish the performance WekaFS could deliver with the minimum hardware configuration of a single 100-Gbit InfiniBand link to the host.
• WekaFS is capable of nearly saturating a 100-Gbit link, delivering a peak read performance of ~11 Gib/second to a single client irrespective of cores (2, 4, 6) on the Weka client mount.
• IOPS measurement: WekaFS delivered over 740,000 IOPs to a single DSS8440-GDS system on one 100-Gbit network link with the 4-cores configuration.
• Maximum throughput at 2, 4, 6 cores is ~11 Gib/s. A significant IOPS increase appears with GDS when mounted with 4 cores: increase from ~400K (with 2 cores) to ~740K (with 4 cores), occurring with small IO read at lower block sizes (4K–32K). No such improvement at higher cores (6); with 6 cores the IOPS are ~675K at 4K block size.
• Latency performance is impressive with Weka (lowest 256 µs at 4K) because the kernel is bypassed (Fig. 4(c) and Fig. 5(c)). For both GDS and CPU-GPU paths, latency increases with IO transfer size.
• With Weka, GDS latency is 88% and 49% lower vs. CPU-GPU when varying block size from 32K to 256K, respectively.
• CPU utilization does not show overall improvement, but it decreases for the GDS path from medium to higher block sizes (32K to 1M); from 32K block size, ~18%– 21% decrease in CPU utilization.
• Since Weka core software runs inside a Linux container, it avoids introducing overhead latency and extra CPU cycles. The Weka testbed additionally uses performance-optimized networking DPDK (does not use standard kernel-based services); GDS is configured with --upstream-libs and --dpdk. With DPDK, the Weka cluster CPUs share a large burden vs. RDMA that completely offloads it; thus fewer CPU cores with GDS.
• Increasing processes or spawning threads across multiple GPUs does not improve performance; the performance boost is observed when spawning multiple threads in a single device.
• Key point: metrics here were delivered via a single client mount to a single Dell DSS8440 system — unprecedented for WekaFS. The benefit of GDS with a distributed file system is much more apparent with multiple GPUs. Fig. 5 uses 3 GPUs achieving maximum throughput; further increase in GPUs (e.g., all 8) did not show improvement for the random-read workload.

IV.B.2. GDS performance for ResNet50 training (Fig. 5 (e)–(h)):

• Analyses of DALI optimizations with GDS in the ResNet50 v1.5 pipeline; varying mini-batch sizes (4 to 256) with 8 V100S GPUs (multi-GPU) on a single client DSS8440 and ImageNet 2012 on the Weka remote cluster (6 nodes, 24 PM9A3 drives).
• Overall faster IOs with GDS integration; overall CPU utilization reduced: ResNet training with GDS uses ~5–10 cores fewer for batch sizes 4–32, and ~15 cores fewer for batch size 64 and beyond, vs. CPU-GPU.
• A typical DL training process fetches a minibatch from storage, preprocesses it (decompression, random cropping, resizing, flipping), runs GPU prediction, then updates model weights with computed gradients. In TensorFlow here, data preprocessing and computation happen in a pipeline; minibatches are prefetched and preprocessed using multiple CPU cores while the GPU computes the current minibatch.
• GPU utilization shows no noticeable improvement with the GDS path; however, training throughput improved ~2–8% for lower batch sizes (4–16) and ~12–15% for batch size 32 and above.
• Similar latency observations as before with GDS: significant latency differences from smaller to medium block sizes (Fig. 4(c)); equivalent latency at higher transfer sizes vs. CPU-GPU. As with smaller block sizes, beyond 32K latency includes most of the time to transfer data (as opposed to data-access overhead).
• It is worth exploring GDS for GPU-accelerated applications, given no downsides observed vs. traditional data paths.

V. Discussion and System Implications

• The authors present high-level design guidelines for synthetic and real DL workloads with GDS integration; GDS enables data-movement speed-up between GPU and storage by lowering CPU utilization and latency and increasing throughput.
• Proposed guidelines for deciding operating application settings with GDS:
  1. As workloads shift from CPU- to GPU-centric systems and data paths move from storage to GPUs, bottlenecks are created; GDS offers performance advantage.
  2. Communication delay decreases in the CPU-GPU path with larger transfer sizes; as IO transfer size grows, GDS mitigates the limitations of the traditional CPU-GPU data path and improves overall performance.
  3. Multi-threaded IOs and client-mount CPU cores offer the possibility to reach theoretical GDS throughput and IOPS, especially with distributed file systems like Weka.
  4. While it is challenging to ensure all workloads use large IO sizes (from small to medium transfers), considerable throughput improvements and better CPU utilization are observed with GDS, more prominently for WekaFS. A parallel distributed file system like WekaFS is a unified namespace whose execution happens in a user-space Linux container, eliminating kernel-specific dependencies and time-sharing; thus, unlike traditional software-based designs, Weka uses only the resources allocated inside its container.

VI. Conclusions and Future Work

• The paper quantifies the performance of integrating GDS access into local storage and file-based Weka remote storage.
• Single NVMe SSD local storage: 14% increase and decrease in throughput and overall latency respectively; 12% decrease in CPU utilization with GDS, to and from local storage, specifically for medium to high data-transfer sizes.
• Distributed (Weka remote) file system: evaluated with PM9A3 SSDs and GDS access.
  • Synthetic analyses: SSDs can achieve 740K IOPS and IO read throughput of 11 GiB/s.
  • ~2.8x throughput advantage with GDS for 4K block size.
  • Overall latency improves by 45% with GDS.
  • As IO transfer size increases beyond 32K, 18%–21% (2–3x) decrease in CPU utilization.
  • For Weka with DL workload: training throughput improved ~2–8% for lower batch (up to 16) and ~12–15% for batch sizes above 32, respectively; CPU utilization reduced by ~39–45% for batch size 32 and above; GPU utilization comparable to traditional CPU-GPU path.

Future work:

• Analyze different MLPerf benchmark models optimized with GDS.
• Leverage results to develop a methodology to depict tuning parameters for application-specific optimal GDS configurations.
• Conduct performance analysis with other remote storage systems such as Lustre, NFS, and S3 protocol services on top of WekaFS to observe impact on GDSIO workloads.

Consolidated Headline Quantitative Results

Named Methods, Tools, and Systems

Data-Path Architecture (Fig. 1)

WITHOUT GDS (Fig. 1a, cuda_io=posix, path 1→2→3→4)
  NVMe SSD --1--> CPU --2--> RAM (bounce buffer / page cache)
                   |                  |
                   +--3--> CPU -------+
                              |
                              4
                              v
                            GPU            (cudaMemcpy HostToDevice)

WITH GDS (Fig. 1b, cuda_io=cufile, path 1→2)
  NVMe SSD --1--> PCIe Switch --2--> GPU memory
                  (direct DMA; "No bounce buffers needed")

GDS SOFTWARE STACK (Fig. 2)
  Applications on CPU
        |
  cuFile API / libcufile.so
        |
  CUDA
========== kernel boundary ==========
  nvidia-fs.ko kernel driver
        |
  Virtual File System (VFS)
        |
  Filesystem driver | Block IO driver | Storage driver   ==>  [Proprietary
        |                                                       Distributed FS]
  Storage (SSDs) DMA engine  <-------------------------------------+

Limitations (stated and implied)

• GPU utilization shows no noticeable improvement with GDS in any workload — the win is on CPU offload, latency, and throughput, not GPU efficiency.
• For small/lower IO sizes, GDS read rates can be reduced due to overhead of querying the CUDA runtime for the GPU memory-buffer pointer; the benefit is concentrated at medium-to-high transfer sizes (≥32K).
• Scaling is limited: increasing processes/threads across multiple GPUs does not improve performance for random read; the boost comes from threads on a single device, and adding GPUs beyond 3 did not help the random-read workload.
• Single-client testbed (one DSS8440); results are for a single 100-Gbit IB link; multi-client linear scaling is expected but not measured here.
• Only EXT4 (local) and WekaFS (remote) evaluated; other remote file systems (Lustre, NFS, S3) are deferred to future work.
• Only one real DL model (ResNet50 v1.5) and one synthetic pattern (4K 100% random read) are evaluated; broader MLPerf models are future work.
• Results reported via bar charts (Fig. 4, Fig. 5) without exhaustive numeric tables; many magnitudes are stated only in prose.

Open Problems / Future Directions

1. Develop a methodology to tune application-specific optimal GDS configurations (parameters such as IO threads, client-mount cores, block size, queue depth).
2. Extend evaluation to MLPerf benchmark models beyond ResNet50.
3. Quantify GDS on other HPC workloads beyond DL training and synthetic random read.
4. Evaluate GDS over additional remote storage backends (Lustre, NFS, S3 on WekaFS) to generalize the IO-path findings.