GeminiFS: A Companion File System for GPUs — Detailed Summary

Shi Qiu, Weinan Liu, Yifan Hu, Jianqin Yan, Zhirong Shen, Xin Yao, Renhai Chen, Gong Zhang, Yiming Zhang | NICE Lab, Xiamen University / Shanghai Jiao Tong University / Huawei Theory Lab | FAST '25 (23rd USENIX Conference on File and Storage Technologies), Feb 25-27 2025, Santa Clara, CA | USENIX, pp. 221-236 | Code: github.com/nicexlab/GeminiFS

Per-section summary organized by paper headings. Each section includes paragraph-level bullet points and exact quantitative results where the paper provides them.

Abstract

• GPU-centric storage solutions let the GPU access NVMe storage directly via NVMe queues, completely bypassing the CPU; this removes the problems of CPU-centric solutions (high CPU-GPU synchronization overhead, I/O traffic amplification, high CPU processing latency).
• But state-of-the-art GPU-centric solutions provide no file abstraction or management (fine-grained isolation, access control) of traditional host file systems, and cannot satisfy ML applications (GNN, LLM) that need fast file access and data sharing.
• GeminiFS is a companion file system that gives GPU programs a file-system interface for direct file-based access to NVMe storage managed by the host file system.
• Key mechanisms: metadata synchronization by embedding metadata directly into the files; an extended NVMe driver that lets CPU and GPU set up control planes in parallel; a GPU-friendly software-defined page cache that exploits the GPU's internal bandwidth; and a programmer library (libGemini).
• Evaluation shows GeminiFS significantly outperforms state-of-the-art storage solutions for large-scale ML workloads.

1. Introduction

Capacity-demand gap:

• GPU-accelerated ML apps (GNN, LLM) have massive datasets and weights spanning many files, up to tens of TB and growing. Despite a decade of GPU memory capacity growth, the gap between GPU memory and application demand is widening.
• Memory-based expansion (host DRAM, multi-GPU memory pooling) is expensive; storage-based expansion lets GPUs reach into NVMe storage and is more cost-effective with little performance loss given high-performance flash.

CPU-centric inefficiency and GPU-centric response:

• Most storage-based solutions (Dragon, GDS) are CPU-centric: the CPU initiates storage access explicitly (user/OS code manages transfer) or implicitly (CPU page-fault handler triggered by GPU page faults on mmap'd files). The CPU and CPU-GPU synchronization become the bottleneck when hundreds/thousands of GPU threads do I/O.
• BaM (Big accelerator Memory) is GPU-centric: it allocates NVMe queues in GPU memory so GPU threads submit NVMe I/O commands directly, no host CPU involved.
• Downside of GPU-centric approaches: no file abstraction/management, hence no data/metadata integrity, crash consistency, durability, or cross-device in-storage resource management.
• When accessing files of a traditional file system, GPU-centric approaches still need CPU-GPU memory copies, preventing efficient NVMe utilization; they cannot meet the high parallelism and data-sharing demands (input data, shared KV-cache). Building a general GPU file system is hard because GPUs are unsuitable for stateful storage software needing complex metadata maintenance.

The companion file system idea:

• Solution: a lightweight GPU file system (a Companion File System) that coexists with the host file system. On the host, the host FS manages files (create/move/delete) and metadata is integrated into the files so it can be managed on the CPU and shared with the GPU; on the GPU, metadata is retrieved into GPU memory to provide file-system abstractions.
• Four unique technical challenges: (i) metadata synchronization, (ii) NVMe device-driver limitations, (iii) GPU page-cache efficiency, (iv) GPU programming complexity (each is elaborated in §2.3).

Workload characteristics exploited:

• GPU-accelerated apps have two useful properties: (1) storage I/O is predictable — access info can be obtained beforehand from model settings; (2) most on-disk data is read-only for its lifetime and writes are append-only, so its metadata is also predictable and stable. These properties simplify the design (metadata and index-structure synchronization).

Contributions:

• GeminiFS is, to the authors' knowledge, the first GPU-centric file system that unlocks the GPU's view of the host file system and lets GPUs create on-demand file accesses directly to disk data, without CPU initiation.
• GVDK file format: integrates indispensable metadata into the file (file size/type/offset, and mapping of file logical blocks to NVMe physical blocks), enabling efficient CPU/GPU metadata synchronization and GPU-required file ops.
• Extended NVMe driver (SNVMe): lets CPU and GPU set up control planes in parallel, supporting I/O queues on both so host/GPU file systems concurrently submit NVMe requests.
• GPU-friendly software-defined page cache: flexible API exploiting locality and data placement for predictable access; shareable by multiple GPU processes to cut GPU memory footprint.
• libGemini: simple, powerful abstractions hiding metadata retrieval, synchronization, NVMe I/O-queue control, and host-side initialization.
• Implemented for recently released GPUs; key components open-sourced.

2. Background and Motivation

2.1 Storage Access of GPU Workloads

• Various GPU ML workloads — DNN, LLM, GNN — need efficient storage access, extensively studied in the literature.
• DNN/LLM training studies focus on offloading intermediate data (intermediate weights, activations from forward propagation). GNN training uses high-volume SSDs to store hundreds of TB of adjacency matrices, feature vectors, and intermediate ("short-term") data; short-term data lives only minutes and is invalidated once the next iteration starts. Training periodically writes model weights as checkpoints (checkpoint size ~= model-weight size).
• LLM inference uses SSDs to store model weights (hundreds of GB to TB) and reuse KV-cache across multi-turn conversations to cut repetitive computation. Model weights, KV-cache, and training input data are long-term data: an initial append-only sequential-write phase, then unchanged and read-only.
• Long-term data is often shared and read by multiple GPU processes (model weights shared across parallel training; KV-cache shared via prefix caching in inference).
• Most data access is predictable: e.g., the DNN training process and data features are fixed by model design and iteration count, so access pattern and data size can be statically analyzed.
• Table 1 summarizes data types, access modes, sizes, and retention, yielding four characteristics: short-term data needs no persistence; long-term data is append-only; most access is predictable; data is shared across GPU processes.

Table 1 — Storage access characteristics of GPU-accelerated ML workloads (selected rows):

2.2 Extending GPU Reach to Storage

• Fig. 1 classifies existing approaches as CPU-centric vs GPU-centric, by whether storage access is initiated by the CPU or the GPU.

2.2.1 CPU-Centric Storage Access

• CPU-centric approaches rely on the CPU to initiate requests: GPUfs and syscalls-for-GPUs let GPUs request file data through the host CPU; ActivePointers adds a memory-map abstraction over GPUfs; Dragon ties storage access to the UVM page-fault mechanism.
• These give a POSIX-like interface but rely on CPU user/OS code to orchestrate storage<->GPU transfer; using low-parallelism CPUs to serve high-parallelism GPU demand is inefficient and the CPU-GPU control logic lengthens the path.
• GDS builds a direct DMA data plane between GPU memory and storage, but offers only a non-POSIX interface (high programming complexity) and still relies on the CPU to initiate I/O — a bottleneck. Its cuFileBatchIOSubmit handles at most 128 operations per batch, far short of GPU demand.
• Microbenchmark: a server with 64 CPU cores and an 80 GB-memory NVIDIA GPU (L1 cache disabled), two NVMe devices — Zhiti TiPro 7000 (R/W latency 15 µs) and Intel Optane P5800X (R/W latency 4 µs), both ~7 GB/s bandwidth; results in Fig. 2.
• Fig. 2(a): for GPUfs at low thread counts, I/O latency exceeds 190 µs on both devices — software-stack overhead is over 90% of total I/O latency. When GPU threads exceed CPU cores, average and tail latencies surge (~250% increase at 1024 threads). Fig. 2(b): GDS at small batch sizes has even higher avg/tail latency than GPUfs; larger batches reduce latency but it stays ~160 µs — GDS software-stack overhead is non-negligible.

2.2.2 GPU-Centric Storage Access

• BaM lets GPUs orchestrate high-throughput, fine-grained storage access without CPU overhead; GMT extends BaM's two-tier hierarchy (GPU memory + storage) to three tiers by adding host memory in between.
• BaM allocates NVMe queues in GPU memory, maps them via the GPU driver to be visible on the PCIe bus, and integrates an NVMe driver into the GPU so GPU threads send NVMe I/O commands executed by SSD controllers.
• BaM is analogous to SPDK (both expose a user-level full block stack for direct storage access — GPU and CPU respectively) and inherits SPDK's problem: each process gets the storage as a raw block device, losing the OS file-system abstraction. It must implement a user-level file system (e.g., SPDK's BlobFS) for integrity, crash consistency, durability. Metadata isolation makes data sharing across GPU processes (and GPU<->CPU) hard. Loading host-FS files requires reading NVMe data to host memory then copying to GPU memory — inefficient for model loading/saving, checkpointing, and sharing.

2.3 Challenges

• Goal: a GPU-centric approach that submits I/O directly to storage, fully bypassing the CPU, while still supporting management and a set of POSIX-like file-system interfaces.
• A file system must minimally: (i) maintain file/directory metadata (inode info, transactions for consistency); (ii) map logical offset to physical data blocks; (iii) offer a unified upper-layer interface; (iv) cache recently accessed blocks. Building such a general GPU file system is prohibitively difficult because GPUs are unsuitable for stateful software needing complex metadata maintenance.
• Since all GPU-accessed files can be managed by the CPU, a lightweight GPU file system coexisting with the host FS lets metadata be managed on the CPU and shared with the GPU — but four challenges arise:
• (1) Metadata synchronization. Must be efficient and safe between host and GPU. Hard to use GPU parallelism in metadata sync; in EXT4-like host FSes metadata is exclusively kernel-managed, so when both host and GPU have file systems, metadata safety is hard because GPU file ops are intertwined with metadata ops — risking the CPU-bypass benefit via communication overhead.
• (2) NVMe driver limitations. The kernel NVMe driver does the host part of the protocol (Admin queue pair, I/O queue pairs for admin/I/O commands) and does not support simultaneously establishing NVMe queue pairs on both host and GPU, so the GPU cannot directly submit NVMe commands.
• (3) GPU page-cache inefficiency. A GPU page cache could exploit internal GPU bandwidth (far above PCIe), but modern GPUs have no documented privileged mode, so the cache lives in the GPU process and is hard to share across GPUs — causing redundancy and synchronization issues; and page-cache consistency logic, if naively ported, reduces GPU parallelism. A GPU-specific design is needed.
• (4) GPU programming complexity. A GPU file system that shares metadata with the host must coordinate with the host file system and NVMe driver; providing a POSIX-like interface means abstracting CPU/GPU differences, keeping compatibility with existing GPU programming models, and managing host<->GPU data movement — considerable effort.

3. GeminiFS

• GeminiFS gives GPU programs direct access to disk space managed by the host file system through file interfaces; architecture in Fig. 3.

3.1 CPU-Bypassing via Metadata Embedding

• File systems are built on metadata (inode, superblock, directories, index structure, journal, etc.). GeminiFS embeds the host file system's metadata so it can be shared CPU<->GPU and provide file-system functionality on the GPU.

3.1.1 Selective Embedding of Metadata

• Embedding all FS metadata in files and building a complete GPU file system would be prohibitively costly.
• First, host kernel file systems have exclusive metadata control for security/ integrity; GPU file ops are intertwined with metadata changes (append needs index-structure allocation; open/close updates timestamps), and syncing all of this CPU<->GPU defeats the CPU-bypass goal. Because GPU I/O is predictable, GeminiFS preemptively allocates a fixed-size file on the host and embeds only existing metadata (access mode, file size, index structure) — it need not allocate or synchronize new metadata.
• Second, FS-management metadata (e.g., directories) is neither suitable nor necessary on the GPU; implementing directories on the GPU adds complexity, and CUDA mixes host C++ and device code, so directory/host-FS info is easy to get in host C++. Hence GeminiFS embeds only private per-file metadata.
• Private per-file metadata = file type, I/O block size, data blocks, index structure, block bitmap. This enables offset translation (virtual->physical), offset management, and read/write-range checks — supporting read()/write() for GPU programs. It also includes a dirty bitmap for crash consistency: the dirty-bitmap NVMe offset points to a contiguous storage page where each bit records whether the corresponding file page was written; after a write the dirty flag is set and the bitmap is flushed later.

3.1.2 Embedded Block Map

• Modern file-system index structures are complex (EXT4 uses an extent tree for logical<->physical block mapping), needing intricate control logic and branches. GPU control units are simple (no real branch prediction / OoO), so doing address translation on the GPU is costly and slows GPU I/O.
• GeminiFS therefore uses a host kernel module, the GVDK helper, that at file creation obtains each logical block's physical block offset from the host kernel and embeds it in the file. This costs ~0.2% capacity overhead (a 4KB physical block needs 8B to store its NVMe offset) but improves GPU I/O.

3.1.3 File Organization

• GeminiFS proposes a GPU-tailored file format, the GPU Virtual Disk format (GVDK); organization in Fig. 4.
• GVDK is organized in blocks equal to the host FS block size (e.g., 4K in EXT4); a block is the allocation unit for both data and metadata. Per-file private metadata is embedded in the file's first block and read into GPU memory on open.
• A two-level mapping translates file offset to NVMe offset: the first-level table (L1 table) has variable size (recorded in the header), may span multiple blocks but must be contiguous, and each entry holds an NVMe offset pointing to an L2 table; the second-level table (L2 table) is exactly one block, each entry holding an NVMe offset pointing to a data block. A GVDK file offset m splits into three parts m = (m1, m2, m3): m1 indexes the L1 table (locates the L2-table entry), m2 indexes the L2 table (locates the data block's NVMe offset), m3 is the offset within the data cluster. On open() the mapping table is cached into GPU memory to accelerate lookup.
• For security, GeminiFS can ship only precompiled static/dynamic libraries and headers (like CUDA libs) to hide file details, and can integrity-check the metadata region to prevent malicious tampering and unauthorized access to other on-disk files.

3.2 CPU/GPU Shared NVMe Driver

• GeminiFS must establish a storage device's control and data planes on both CPU and GPU at once — which current OS kernels/drivers do not support.
• The NVMe protocol defines host<->SSD commands. Normally the CPU allocates submission queues (SQ) and completion queues (CQ) in host memory. Linux NVMe controller init: (1) create exactly one admin SQ+CQ pair (manage controller — create/delete I/O queues, abort commands); (2) submit admin commands to get controller capabilities and namespace-specific settings; (3) the controller allocates an appropriate number of I/O queue pairs and registers them via the admin command.
• After init, the NVMe driver registers the namespace with the block layer as a host-managed block device, on which the OS builds a file system.
• Insight: the GPU need not manage the whole NVMe device space — only read/write it on demand. So the GPU need not implement the full NVMe driver; it only needs to establish I/O queue pairs, via a relatively simple I/O-QP driver, to set up the NVMe I/O control plane.
• They propose the CPU/GPU Shared NVMe Driver (SNVMe) (Fig. 3, blue dashed box) with two major changes vs the standard NVMe host driver: (a) a GPU buffer management module that records GPU memory allocations used to build I/O queues, cooperates with the GPU driver via nvidia_p2p_get_pages_persistent to pin the GPU I/O-queue pages (making them accessible to a third-party device), and uses nvidia_p2p_dma_map_pages to convert GPU virtual memory to DMA addresses visible to the NVMe device; (b) revised init steps 1 and 3 of the standard NVMe subsystem.
• Stage 1: before the first standard step, allocate the I/O-queue memory in GPU memory; the GPU buffer module translates the GPU virtual address to a DMA address visible to the NVMe device; preset the number of I/O queues and their depth (recorded in the GPU buffer module), then begin step 1 of the standard process.
• Stage 2 (step 3): the controller also registers the GPU-memory-allocated I/O queues. These queues do not use host interrupts; they use GPU-thread polling, reusing BaM's high-throughput I/O-queue driver for efficient SQ submission and CQ polling at high GPU parallelism.

3.3 GPU-Specific Page Cache

• A GPU page cache differs from a CPU page cache in two main ways.
• First: because GeminiFS runs in non-privileged mode, the page cache lives in the GPU process's memory, causing redundancy/wasted GPU memory when multiple processes open the same file. Solution: a page-cache management module in SNVMe on the host — when a program opens a file and builds a GPU page cache, it checks whether the cache already exists; if not, it allocates GPU memory, has the host driver create a persistent mapping, retrieves an inter-process memory handle, and stores it; another process opening the same file gets a pointer to the existing cache via the handle. CUDA already supports this: cuIpcGetMemHandle exports device memory for another process, and nvidia_p2p_get_pages_persistent pins GPU memory persistently.
• Second: locks are needed for mutually exclusive cache access during page-mapping changes (page swapping); the GPU's higher parallelism makes lock contention more severe than on a CPU, degrading performance. GeminiFS uses two mitigations.
• Method 1 — warp-level page acquisition. Acquire pages at warp (not thread) granularity. Each Streaming Multiprocessor (SM) executes one instruction from a warp at a time; the number of SMs and warp schedulers caps concurrent warps and thus contention. In Ampere there are 108 SMs each with 4 warp schedulers, so at most 4 × 108 = 432 control flows contend for the page-cache lock at any moment — far more reasonable than thousands of threads contending (though still above typical CPU page-cache contention).
• Method 2 — constant-time container. A hash table tracks file-page->memory- page mappings for constant-time hit lookup; on a miss, a doubly linked list + hash table manage zero-reference pages so insert/delete/lookup are all constant-time. On a hit, the page is queried in and removed from the zero-reference set in constant time; when a page is released and becomes zero-reference again it is appended to the list tail, so long-unreferenced pages drift to the head and become the coldest — minimizing critical-section duration and lock contention.
• The result is a page cache with a large tuning space: page-cache size and page size adjust lock-contention intensity (with fixed warp locality, larger pages mean fewer pages per warp, fewer cache accesses, less contention). A prefetch feature lets a single warp issue many NVMe commands while acquiring pages at warp granularity, maximizing NVMe bandwidth. The large tuning space helps reach optimal performance across GPU models; since the page cache is user-space, this tunability introduces no new security vulnerability.

3.4 GPU Programming Model

• libGemini is GeminiFS's GPU-oriented programming model, abstracting both the underlying architecture and the file system. Developers initialize GeminiFS with a simple interface and then issue GPU file I/O indistinguishably from host I/O, via a subset of POSIX-like APIs (Table 2). It integrates into existing frameworks (e.g., PyTorch's DataLoader) by setting up GeminiFS once and replacing the host FS interface, eliminating the host bounce buffer.

Table 2 — CPU-side and GPU-side APIs of GeminiFS:

• libGemini deliberately omits full POSIX semantics (e.g., crash consistency), leaving it to applications via the sync interface, for two reasons: data is mostly read-only (offloaded training intermediates need no consistency guarantee), and all in-file metadata is application-managed so GPU read/write does not affect host-FS metadata.
• libGemini also does not implement a comprehensive POSIX I/O suite — full POSIX compliance is deemed costly and unnecessary; read/write suffice for GPU ML apps. The model is presented via system startup and the read/write process (Fig. 5).
• SNVMe init: the developer calls Geminifs_init (CPU-side) with (i) dev_path (e.g., /dev/nvme0n1), (ii) GPU_ids (array of GPU device IDs), and (iii) the number of I/O queue pairs per GPU; this creates GPU NVMe I/O queues and initializes SNVMe per §3.2.
• File open: the CPU-side G_open returns a POSIX-like file descriptor for GPU programs, abstracting GeminiFS's logic (creating specially formatted files, building page caches) — Fig. 5(2)(3).
• G_open takes path, flag, cache_capacity, page_size. It opens only GeminiFS files; flag must include an access mode (O_RDONLY/O_WRONLY/O_RDWR), must add O_CREAT if the path does not exist (creating the file in the §3.1 format), and should include O_DIRECT to indicate whether the page cache is used. On completion the file's metadata is stored in GPU memory and its address is returned, abstracted as dev_fd, used by subsequent read/write.
• File read/write: GPU-side G_read / G_write take dev_fd, buf, offset, nbyte (like pread/pwrite). They locate the file metadata via dev_fd, check the access mode and that offset/nbyte are in bounds, translate the virtual address to an NVMe offset (§3.1), invoke the GPU-side NVMe I/O-queue driver to build a request submitted to the CQ, poll the CQ for completion, and return the bytes transferred.
• File sync: since model-weight and KV-cache generation involve writes that modify both in-file metadata and data blocks, the GPU-side G_sync ensures modified FS metadata and cached file data are written to the file for durability.
• File close: a CPU-side G_close (paper text labels the paragraph "File Close" while referencing G_sync) locates the file's metadata and page cache via dev_fd, ensures modified data is on disk, then releases the GPU memory resources.

4. Evaluation

• Three questions: performance advantage vs SOTA (§4.1); how to maximize GeminiFS performance on GPU architectures (§4.2); benefit to real-world applications (§4.3).

System settings / implementation:

• ~2000 LoC for the shared NVMe kernel module; ~3000 LoC for libGemini.
• Server: 64-core Intel Xeon 5416S, 512 GB RAM, Ubuntu 20.04, Linux 5.15.0.
• GPU: 80 GB HBM, peak HBM bandwidth 1,935 GB/s; host link PCIe Gen4 x16 = 64 GB/s.
• NVMe: Intel Optane 5800X with EXT4; controller supports up to 135 I/O queue pairs; ~7 GB/s max bandwidth. They allocate 64 QPs on the host, 32 QPs on the GPU (32 suffices to maximize disk bandwidth). EXT4 and GeminiFS both use 4K block size (EXT4 cannot handle block size exceeding the system page size).

Baselines:

• (a) GPUfs — CPU-centric, accelerator-centric model; 4 CPU threads handle GPU requests (saturates the disk).
• (b) NVIDIA GDS — CPU-centric; CPU threads orchestrate transfer via the cufile API.
• (c) BaM — GPU-centric, no file system; non-FS interface moving data from NVMe raw device to GPU memory.

4.1 Comparison with SOTA Solutions

• They compare 4K-granularity read at various parallelism with GPUfs, GDS, BaM. To remove caching effects, GeminiFS caching was bypassed; GPUfs cache reduced to 256 MB and host files opened with O_direct.
• Bandwidth (Fig. 6): GeminiFS exceeds GPUfs across all thread counts, averaging 7.33× the bandwidth of GPUfs; at 1,024 threads GeminiFS reaches the NVMe bandwidth peak. GPUfs degrades from CPU-core contention; GeminiFS bypasses the CPU and reuses BaM's high-throughput GPU I/O-queue design to cut I/O-queue contention.
• GDS is ~57% higher bandwidth than GeminiFS only at 1-16 threads (CPU at 4 GHz beats the GPU core at 1 GHz for the software stack), but at 128-512 threads GeminiFS reaches 6.2× the bandwidth of GDS because GPU physical parallelism saturates the disk even at 4K, whereas GDS under-parallelizes at low counts and suffers thread contention at high counts. Versus BaM, GeminiFS is 4.6% lower in bandwidth (metadata-parsing and address-translation overhead from going through file read/write interfaces vs BaM's raw device).
• Latency (Fig. 7): vs GPUfs, GeminiFS gives a 79.6%-90.9% latency reduction across thread counts (GPUfs needs much GPU+CPU memory for request transmission, lengthening both planes). Vs GDS, GeminiFS latency is 57.2% higher at 1-8 threads, but rises far more slowly, so at 1,024 threads GeminiFS latency is only 17% of GDS. Vs BaM, GeminiFS latency is only ~4.8% higher.

4.2 Performance of Page Cache

• A configurable page cache exploits internal GPU bandwidth. Microbenchmark: sequentially read a single 20 GB file in the GPU kernel via GeminiFS; to avoid NVMe being the bottleneck, memory replication substitutes for NVMe access.
• Prefetch (Fig. 8): page size 64 KB, cache 1 GB; on each miss the cache prefetches multiple pages (suits sequential R/W). Without prefetch, read/write bandwidth are only 30.2% and 28% of theoretical; with prefetch (regardless of pages prefetched), read/write improve ~2.4× and ~2.34×, nearly reaching max bandwidth.
• Number of warps (Fig. 9): warp-level acquisition + constant-time container for low contention; 32 threads per warp, varying warp count. Write bandwidth rises ~1.7 GB/s -> ~641.2 GB/s, read ~2.3 GB/s -> ~658.1 GB/s, growing multiplicatively and peaking around 650 GB/s — exceeding 640 GBps. Since a single NVMe gives ~7 GB/s, ~100 NVMe drives would be needed to saturate the page cache, so it is unlikely to be a bottleneck.
• Page size (Fig. 10): warp count 128 (max kernel-memcopy bandwidth ~120 GB/s), 4,096 pages, prefetch on. As page size grows 4 KB -> 1,024 KB, write bandwidth rises **~45.8 -> 120.1 GB/s** and read ~48.4 -> 121.4 GB/s; at 1,024 KB both approach the theoretical bandwidth. Larger pages reach peak page- cache bandwidth more effectively.

4.3 Performance Benefit for LLM Training

• Compare native (memcopy + read/write), DLRover-RM (fast async checkpoint persistence), GDS, and GeminiFS when offloading data generated during LLM training; GeminiFS page cache = 2 GB. Model: GPT2-124M (described as a large transformer LM with 1.5 billion parameters), batch size 64, 3 steps (each = one forward + one backward, weight update, and a checkpoint), sufficient since per-step compute and storage-access patterns are nearly constant.

Table 3 — Storage access in GPT2-124M training:

• Activations kept in HBM (Fig. 11a): GeminiFS runtime decreases 25%, 12%, 10% vs native, DLRover-RM, GDS. Compute dominates total runtime, so the main gain is checkpoint-I/O optimization: checkpoint write time reduced 85%, 75%, 59% vs the three — from bypassing the CPU (less communication overhead) plus the page cache's ultra-high bandwidth raising throughput.
• Activations offloaded (Fig. 11b): GeminiFS reduces training time 94.5% and 91% vs native and GDS. Here activation offloading dominates and involves many small I/Os that prevent good bandwidth use; GeminiFS cuts frequent GPU-CPU communication and the high-bandwidth page cache fully uses system bandwidth. Versus keeping all activations in GPU memory, training time rises only ~4×; in theory, integrating multiple NVMe devices could approach DRAM-only performance.

5. Conclusion and Future Work

• GeminiFS is a companion file system for GPUs: it synchronizes metadata between host and GPU file systems by embedding metadata into files, and extends the NVMe driver so CPU and GPU set up control planes in parallel, giving GPU-centric solutions direct file-interface access to host-FS-managed disk space. It shortens the control and data planes of GPU storage, leverages GPU architectural advantages, and its coexistence with the host FS better satisfies ML storage-access demands.
• Future work: full multi-GPU support — enable parallel reads/writes by logically splitting files; aggregate multiple NVMe devices with RAID to meet multi-GPU bandwidth; for unpredictable workloads, use file pre-allocation (pre-allocate file slots, batch-allocate files into them at runtime by actual usage); integrate GeminiFS into PyTorch so solutions like vLLM benefit.
• Acknowledgments: shepherd Prof. Sudarsun Kannan and anonymous reviewers; funded by China's National Key R&D Program (2022YFB4500302) and NSFC (62441220); Yiming Zhang is corresponding author.

6. System Architecture (synthesized)

        HOST (CPU side)                         GPU side
+--------------------------------+   +---------------------------------+
| User: CPU Runtime              |   | GPU Applications                |
| Kernel:                        |   |   libGemini (POSIX-like API)    |
|   GVDK Helper  ----embeds----> |   |   Metadata/Mapping cache        |
|   Metadata / GVDK File         |   |   (Header + L1/L2 + clusters)   |
|   File System (EXT4)           |   |   GPU Storage Volume Layer      |
|   Block Layer                  |   |   GPU Page Cache (warp-level,   |
|   NVMe Control Block           |   |     constant-time container)    |
|   SNVMe: Admin QP + 64 I/O QPs |   |   I/O Queue Driver + 32 I/O QPs |
+--------------------------------+   +---------------------------------+
         |                                       |
   Mem -- CPU -------- PCIe Gen4 x16 ------- NVMe SSD (Optane 5800X)
                                                 ^
                                    Device DMA <-+-> GPU HBM (direct data plane)

Control plane: GPU I/O QPs set up in parallel with host via SNVMe.
Data plane:    NVMe DMAs directly to/from GPU HBM; CPU is off the I/O path.

• What it is: a lightweight GPU-side file system coexisting with the host FS; the host FS manages file lifecycle and metadata, while GeminiFS lets GPU threads read/write NVMe files directly without the CPU on the I/O path.
• Key novelty: exploit predictable, read-mostly/append-only ML I/O to embed only private per-file metadata + a precomputed logical->physical block map in the GVDK file (so the GPU consumes existing metadata rather than synchronizing dynamic metadata), plus SNVMe so both CPU and GPU set up NVMe I/O queues in parallel and the GPU submits NVMe commands directly. First GPU-centric file system to unlock the GPU's view of the host file system for on-demand direct disk access without CPU triggering.
• Read/write flow: G_read/G_write -> locate metadata via dev_fd -> bounds check -> translate file offset to NVMe offset via cached L1/L2 map -> warp-level page-cache lookup -> on miss, GPU I/O-queue driver builds NVMe command in a GPU-memory SQ -> NVMe DMAs to/from GPU HBM over PCIe -> GPU polls CQ -> return bytes.

7. Related Work and Positioning

8. Limitations

• libGemini is not fully POSIX-compliant: no crash consistency by default (left to applications via G_sync) and no comprehensive POSIX I/O suite — argued as costly and unnecessary for read-mostly GPU workloads.
• ~4.6% bandwidth and ~4.8% latency overhead vs BaM from metadata parsing and address translation.
• EXT4 cannot handle block sizes exceeding the system page size, constraining the block size to 4K.
• Current design does not fully support multi-GPU (deferred to future work).
• The companion-FS approach leans on workload predictability and append-only/ read-mostly access; unpredictable workloads need the future file-pre-allocation scheme.

9. Cross-Cutting Take-Aways