GeminiFS: A Companion File System for GPUs

Shi Qiu, Weinan Liu, Yifan Hu, Jianqin Yan, Zhirong Shen, Xin Yao, Renhai Chen, Gong Zhang, Yiming Zhang | NICE Lab, Xiamen University / Shanghai Jiao Tong University / Huawei Theory Lab | FAST '25 (23rd USENIX Conference on File and Storage Technologies), Feb 25-27 2025, Santa Clara, CA | USENIX, pp. 221-236 | Code: github.com/nicexlab/GeminiFS

Problem

GPU-accelerated ML applications (GNN, LLM) have datasets and weights up to tens of TB that exceed GPU memory, so they offload to NVMe storage. Existing storage access falls into two camps, each deficient. CPU-centric solutions (GPUfs, Dragon, GDS) keep the host CPU on the I/O path to initiate transfers; with hundreds or thousands of GPU threads issuing I/O, the low-parallelism CPU and CPU-GPU synchronization become the bottleneck. GPU-centric BaM lets GPU threads submit NVMe commands directly by placing NVMe queues in GPU memory, but it exposes the device as a raw block device with no file abstraction — losing data/metadata integrity, crash consistency, durability, and cross-process data sharing, and still requiring host-memory bounce copies when reading host file-system files. No prior system gives the GPU a file-system interface while bypassing the CPU on the I/O path.

Core Insight

GPU ML workloads are predictable and mostly read-only / append-only (Table 1), so the dynamic-metadata problem that makes a general GPU file system intractable can be sidestepped: pre-allocate a fixed-size file on the host and embed only the existing private per-file metadata — including a precomputed logical-to-physical block map — directly into the file. The GPU then consumes existing metadata rather than synchronizing mutating metadata with the CPU. Combined with an extended NVMe driver that lets the CPU and GPU establish NVMe I/O queues in parallel, this makes GeminiFS the first GPU-centric file system that unlocks the GPU's view of the host file system for on-demand direct disk access without CPU triggering.

Method

GeminiFS is a lightweight GPU-side file system that coexists with the host file system (the host FS manages file lifecycle and metadata; GeminiFS does the GPU I/O). Four components:

Level 1: GVDK file format (metadata embedding)
  - Embed only private per-file metadata in the file's first block
  - GVDK Helper (host kernel module) precomputes logical->physical block map
  - Two-level mapping (L1 table -> L2 table -> data block); offset m=(m1,m2,m3)
  - ~0.2% capacity overhead (8B NVMe offset per 4KB block)

Level 2: SNVMe (CPU/GPU Shared NVMe Driver)
  - GPU buffer-management module pins GPU I/O-queue pages
    (nvidia_p2p_get_pages_persistent / nvidia_p2p_dma_map_pages)
  - Revises NVMe init steps 1 & 3 so I/O queues live in GPU memory
  - GPU I/O queues use thread polling (reuses BaM's I/O-queue driver)

Level 3: GPU-specific page cache
  - Cross-process sharing via host page-cache module + cuIpcGetMemHandle
  - Warp-level (not thread-level) page acquisition cuts lock contention
    (Ampere: at most 4x108=432 control flows contend at once)
  - Constant-time container (hash table + doubly-linked list) for zero-ref pages
  - Tunable page size / cache size / prefetch count

Level 4: libGemini
  - POSIX-like subset: host G_open/G_close/Geminifs_init; device G_read/G_write/G_sync
  - Not fully POSIX; crash consistency left to apps via G_sync

On read/write the GPU locates metadata via dev_fd, bounds-checks, translates the file offset to an NVMe offset via the cached L1/L2 map, looks up the warp-level page cache, and on a miss the GPU I/O-queue driver builds an NVMe command in a GPU-memory submission queue; the NVMe device DMAs data directly to/from GPU HBM over PCIe while the GPU polls the completion queue. The CPU is entirely off the I/O path.

Experimental Setup

Motivation microbenchmarks also used a Zhiti TiPro 7000 SSD (15 µs R/W latency) and an Intel Optane P5800X (4 µs R/W latency). The page-cache microbenchmark substitutes memory replication for NVMe so the disk is not the bottleneck.

Headline Quantitative Results

4K read vs SOTA (Figs. 6-7):

• Bandwidth: GeminiFS averages 7.33x GPUfs; reaches the NVMe peak at 1,024 threads; 6.2x GDS at 128-512 threads (GDS only ~57% higher than GeminiFS at 1-16 threads); 4.6% below BaM.
• Latency: 79.6%-90.9% lower than GPUfs; vs GDS, 57.2% higher at 1-8 threads but only 17% of GDS latency at 1,024 threads; only ~4.8% above BaM.

Page cache (Figs. 8-10):

• Prefetch: without it, read/write reach only 30.2% / 28% of theoretical; with it, ~2.4x / ~2.34x improvement (near max).
• Warps: scales to ~650 GB/s peak (write 1.7->641.2 GB/s, read 2.3->658.1 GB/s) — would need ~100 NVMe drives to saturate.
• Page size 4 KB->1,024 KB: write 45.8->120.1 GB/s, read 48.4->121.4 GB/s.

GPT2-124M LLM training (Fig. 11), page cache 2 GB:

Storage footprint of one GPT2-124M step (Table 3): model weights 238 MB (read & write), checkpoint 713 MB/step (append-only seq. write), activation 57.96 GB (read & write).

Limitations

• libGemini is not fully POSIX-compliant: no crash consistency by default (left to applications via G_sync) and no comprehensive POSIX I/O suite — argued as costly and unnecessary for read-mostly GPU workloads.
• ~4.6% bandwidth and ~4.8% latency overhead vs raw-device BaM, from metadata parsing and address translation.
• EXT4 cannot handle block sizes exceeding the system page size, constraining the block size to 4K.
• Current design does not fully support multi-GPU.
• The approach leans on workload predictability and append-only/read-mostly access; unpredictable workloads need the future file-pre-allocation scheme.

Open Problems / Future Work

1. Full multi-GPU support — parallel reads/writes by logically splitting files.
2. Aggregating multiple NVMe devices via RAID to meet multi-GPU bandwidth (and, in theory, approach DRAM-only performance for offloaded activations).
3. File pre-allocation for unpredictable workloads — pre-allocate file slots, then batch-allocate files into them at runtime by actual usage.
4. Integrating GeminiFS into PyTorch so solutions such as vLLM benefit.