Also: Brief SummariesDetailed Summaries

Architecture & Measurement-Design Analysis

GeminiFS: A Companion File System for GPUs

Source: Qiu, S.; Liu, W.; Hu, Y.; Yan, J.; Shen, Z.; Yao, X.; Chen, R.; Zhang, G.; Zhang, Y. 23rd USENIX Conference on File and Storage Technologies (FAST '25), Feb 25-27 2025, Santa Clara, CA. ISBN 978-1-939133-45-8. Code: https://github.com/nicexlab/GeminiFS Authors: NICE Lab, Xiamen University; Huawei Theory Lab; Shanghai Jiao Tong University. (Yiming Zhang, corresponding.) Reader: gemini-reader quota exhausted; extracted via general-purpose subagent using the Read tool with the pages parameter (17-page PDF read as 1-10 + 11-17). Analyst: Vishwakarma Date: 2026-06-29

Table of Contents

  1. System Architecture (the "specimen") — GeminiFS as a companion FS
  2. System-Under-Test Architecture (the hardware testbed / harness)
  3. Design-Space Diagram (axes swept, axes held fixed)
  4. Algorithm / Control Flow Diagrams (init, open, read, write/sync)
  5. Quantitative Results — Empirical Findings by Regime
  6. Configuration-Regime Trade-off Tables
  7. Bottlenecks & Insights Surfaced by the Measurements
  8. Limitations of the Methodology
  9. Note on NCCL Tuning
  10. Analogy

1. System Architecture (the "specimen")

GeminiFS answers a single structural question: how do you give a GPU direct, file-based access to NVMe storage without surrendering the file abstraction that GPU-centric block systems (BaM, SPDK) throw away? Its answer is to run alongside the host file system rather than replacing it — hence "companion." The host EXT4 instance still creates, moves, and deletes files and owns the device's global metadata; GeminiFS only needs the GPU to be able to find and touch the data blocks of a specific file at full device bandwidth. The entire design follows from that division of labor.

Three contributions compose the system: (1) the GVDK on-disk file format that embeds per-file private metadata into the file itself so that the same bytes are authoritative for both host and GPU; (2) the CPU/GPU Shared NVMe driver (SNVMe) that lets host and device stand up their control planes in parallel over one physical NVMe controller; and (3) a GPU-resident, software-defined page cache that exploits HBM's internal bandwidth and is shared across GPU processes via IPC handles. libGemini wraps all of it behind a POSIX-subset API.

+-------------------------------------------------------------------+
|                          GeminiFS                                 |
|                                                                   |
|   GPU MEMORY SIDE                  |   CPU MEMORY SIDE             |
|  +----------------------------+    |  +------------------------+   |
|  | GPU Applications           |    |  | CPU Runtime (user)     |   |
|  +-------------+--------------+     |  +-----------+------------+   |
|                |                    |              |               |
|                v                    |              v               |
|  +----------------------------+     |  +------------------------+   |
|  | libGemini                  |     |  | GVDK Helper            |   |
|  | (POSIX-subset file API)    |     |  | (host kernel module:   |   |
|  +-------------+--------------+      |  |  embeds NVMe offsets)  |   |
|                |                     |  +-----------+------------+   |
|                v                     |              |               |
|  +----------------------------+      |             v               |
|  | GPU Storage Volume Layer   |      |  +------------------------+  |
|  |  Header / Info /           |      |  | File System (EXT4) +   |  |
|  |  Mapping (NVMe offset) /   |      |  | Metadata + GVDK File   |  |
|  |  Data Cluster              |      |  +-----------+------------+  |
|  +-------------+--------------+       |             |               |
|                |                      |             v               |
|                v                      |  +------------------------+ |
|  +----------------------------+       |  | Block Layer            | |
|  | Page Cache (SW-defined,    |       |  +-----------+------------+ |
|  |  warp-level, LRU, hashed)  |       |              |              |
|  +-------------+--------------+        |             v              |
|                |                       |  +------------------------+|
|                v                       |  | NVMe Control Block     ||
|  +----------------------------+        |  |  Admin QP + I/O QPs    ||
|  | I/O Queue Driver + I/O QPs |        |  +-----------+------------+|
|  | (NVMe SQ/CQ in GPU memory) |        |              |             |
|  +-------------+--------------+         |             |             |
|                |                        |             |             |
|     +..........|.. CPU/GPU Shared NVMe (SNVMe) .......|...........+ |
|     :          |   (blue dashed box in Fig. 3)        |          :  |
|     +..........|.....................................|..........+   |
+----------------|-------------------------------------|------------+
                 |                                      |
                 +================ PCIe ================+
                                   |
                            +------v-------+
                            | NVMe SSD(s)  |
                            | (Optane /    |
                            |  TiPro 7000) |
                            +--------------+
^ Fig 1: GeminiFS overview (redraw of paper Fig. 3). The host owns global
  FS metadata; the GPU owns a data-plane I/O queue driver and a page
  cache. SNVMe (dashed) is the one piece that straddles both, letting
  each side hold live NVMe queue pairs against the same controller.

The architectural keystone is that the GPU never manages the NVMe device — it only manages I/O queue pairs. Admin-queue work (controller capabilities, namespace registration, queue creation/deletion) stays on the host's exclusive control path; the GPU receives a preset bundle of I/O SQ/CQ pairs whose memory lives in GPU HBM. This is the consequence that makes everything else legal: because the GPU's footprint is reduced to "submit a command, poll for completion," GeminiFS does not need to displace the host driver, and host and device can coexist on one device.

+----------------- SNVMe control-plane split --------------------+
|                                                                |
|   HOST (full NVMe driver)         GPU (minimal I/O QP driver)  |
|  +---------------------------+   +---------------------------+ |
|  | Admin QP                  |   |  (no admin queue)         | |
|  |  - controller caps        |   |                           | |
|  |  - namespace register     |   |  I/O QPs only             | |
|  |  - create/delete I/O QPs  |   |   - submit NVMe cmd to SQ | |
|  |                           |   |   - poll CQ (no IRQ)      | |
|  |  64 I/O QPs on host       |   |   32 I/O QPs on GPU       | |
|  +-------------+-------------+    +-------------+-------------+ |
|                |                                |               |
|                |   GPU buffer-management module |               |
|                |   in the NVMe driver:          |               |
|                |   nvidia_p2p_get_pages_persistent()           |
|                |     -> pin GPU queue pages                     |
|                |   nvidia_p2p_dma_map_pages()                   |
|                |     -> GPU vaddr => DMA addr (NVMe-visible)    |
|                v                                v               |
|            +=================== PCIe ===================+       |
+----------------------------------------------------------------+
^ Fig 2: The control plane is asymmetric by design. Only the host runs an
  admin queue; the GPU gets a stripped I/O-QP driver. The pin + DMA-map
  step (NVIDIA p2p APIs) is what turns GPU-allocated queue memory into
  something the NVMe controller can DMA into.

This asymmetry is the difference between GeminiFS and a naive "give the GPU the whole driver" approach. A full GPU-side NVMe driver would have to arbitrate admin operations with the host and risk two masters issuing conflicting controller commands. By keeping admin single-homed on the host and replicating only the I/O queue plane, GeminiFS gets parallel control-plane setup without parallel control-plane conflict. The GPU's completion path also drops host interrupts entirely and uses thread polling on the CQ — the same high-throughput SQ/CQ driver BaM introduced, reused here so thousands of GPU threads can drive the queues.

1.1 GVDK — metadata embedded into the file

Coherence between host and GPU is not solved with a synchronization protocol; it is solved by construction. GeminiFS stores each file's private metadata in the first block of the file, so there is only one copy and it is wherever the file is. On open, that block (plus the mapping table) is pulled into GPU memory and abstracted behind a dev_fd.

+--------------------- GVDK file layout --------------------------+
|                                                                |
|  Block 0 (private metadata, 4 KB)                              |
|  +----------------------------------------------------------+  |
|  | File type | File size | Access mode | IO block size      |  |
|  | Blocks | Offset | NVMe offset | Dirty-Bitmap NVMe offset |  |
|  +----------------------------------------------------------+  |
|                                                                |
|  Two-level block map (file offset m -> NVMe offset):           |
|                                                                |
|   m = ( m1 , m2 , m3 )                                          |
|         |    |    +--> byte offset within data cluster        |
|         |    +-------> index into L2 table                     |
|         +------------> index into L1 table                     |
|                                                                |
|   L1 table (contiguous, may span blocks)                       |
|   +-------+-------+-------+ ...                                 |
|   | off-> | off-> | off-> |   each entry = NVMe offset of an   |
|   +---+---+-------+-------+   L2 table                          |
|       |                                                         |
|       v                                                         |
|   L2 table (exactly ONE block)                                  |
|   +-------+-------+-------+ ...                                 |
|   | off-> | off-> | off-> |   each entry = NVMe offset of a    |
|   +---+---+-------+-------+   data block                        |
|       |                                                         |
|       v                                                         |
|   +-----------------+                                           |
|   | Data Cluster    |   <- DMA target / source                 |
|   +-----------------+                                           |
|                                                                |
|  Dirty bitmap: 1 bit / file page, contiguous page, flushed     |
|  on sync -> optional crash consistency                         |
+----------------------------------------------------------------+
^ Fig 3: GVDK format (redraw of paper Fig. 4). The L1/L2 indirection is a
  classic two-level page table, but the "physical" addresses are NVMe
  byte offsets handed up by the host kernel at file-creation time. Cost:
  ~0.2% capacity (8 B per 4 KB block).

The cost of this scheme is deliberately tiny — an 8-byte NVMe offset per 4 KB block is roughly 0.2% capacity overhead — and the benefit is that the GPU's address translation is a pure in-HBM table walk with no host round trip. Only private metadata is embedded; directory and FS-management metadata stay with the host. That boundary is what keeps GeminiFS a "companion" rather than a competing file system: the GPU can resolve a file offset to an LBA on its own, but it never touches the structures the host needs to manage the namespace.

2. System-Under-Test Architecture (the hardware testbed)

The measurement harness is a single-node, single-GPU, single-NVMe rig. That is a narrow testbed, and it shapes how far the results generalize — but it is the regime where the latency claims are cleanest because there is no multi-device aggregation or cross-GPU traffic to confound the I/O path.

+----------------- Testbed (System settings) -------------------+
|                                                               |
|  +-----------------------------+                              |
|  | CPU: Intel Xeon 5416S       |   Ubuntu 20.04               |
|  |  64 cores, 512 GB DRAM      |   Linux kernel 5.15.0        |
|  +--------------+--------------+                              |
|                 |                                             |
|            PCIe Gen4 x16 (64 GB/s)                            |
|                 |                                             |
|        +--------+--------+--------+                            |
|        |                          |                           |
|  +-----v-----------+        +-----v-----------------+         |
|  | GPU             |        | NVMe SSD              |         |
|  |  80 GB HBM      |        |  Intel Optane 5800X   |         |
|  |  HBM BW 1935    |        |  (EXT4 mounted)       |         |
|  |    GB/s         |        |  ~7 GB/s, 4 us R/W    |         |
|  |  ~1 GHz core    |        |  up to 135 I/O QP     |         |
|  +-----------------+        |   pairs               |         |
|                             +-----------------------+         |
|                                                               |
|  Queue split:  64 I/O QPs on host  +  32 I/O QPs on GPU       |
|  Block size:   4 KB (EXT4 and GeminiFS both)                  |
|  Code size:    ~2000 LoC (SNVMe kmod) + ~3000 LoC (libGemini) |
|                                                               |
|  Motivation rig (Fig. 2) also used a second device:           |
|    Zhiti TiPro 7000  (15 us R/W latency)                       |
+---------------------------------------------------------------+
^ Fig 4: Single GPU (80 GB HBM, ~1935 GB/s internal) over PCIe Gen4 x16
  to one Optane 5800X. 32 GPU-side I/O queue pairs are enough to saturate
  one NVMe device (per BaM). EXT4's 4 KB block ceiling fixes the FS block
  size at 4 KB throughout.

Two facts about this rig drive the entire results story. First, the GPU core runs at roughly 1 GHz against the CPU's 4 GHz — so at low parallelism a CPU-orchestrated path (GDS) can out-issue the GPU, and GeminiFS only wins once enough warps are in flight to amortize the slow clock with sheer concurrency. Second, a single Optane caps at ~7 GB/s, which is two orders of magnitude below HBM's ~1935 GB/s and the page cache's measured 640+ GB/s. That gap is why the authors can claim the page cache will never be the bottleneck, and why the device — not the software — sets the ceiling for the 4 KB I/O experiments.

+------------------- Software stack under test -----------------+
|  GPT2-124M training (llm.c) / micro-benchmarks  | application |
+---------------------------------------------------------------+
|  libGemini  (G_open/G_read/G_write/G_sync/...)  | FS API      |
+---------------------------------------------------------------+
|  GPU page cache  +  GVDK volume layer           | caching     |
+---------------------------------------------------------------+
|  GPU I/O-QP driver (BaM-style poll)  |  SNVMe kmod (host)     |
+---------------------------------------------------------------+
|  NVMe controller  /  EXT4 (host-side namespace)               |
+---------------------------------------------------------------+
|  PCIe Gen4 x16  +  Optane 5800X                 | hardware    |
+---------------------------------------------------------------+
^ Fig 5: Software stack. GeminiFS inserts itself as a caching + FS-API
  layer over a GPU-resident NVMe queue driver, while EXT4 keeps the
  namespace on the host below the same device.

3. Design-Space Diagram (axes swept, axes held fixed)

The evaluation is a micro-benchmark sweep over four primary axes plus one application-level workload, against four storage baselines. The most important structural choice is that for the cross-system bandwidth/latency comparison the page cache is deliberately bypassed (shrunk to 256 MB, host files opened O_DIRECT) so the comparison is apples-to-apples on the raw I/O path; the page cache is then characterized separately with NVMe replaced by memory copy so the device never bottlenecks the cache.

+--------------------- DESIGN SPACE -----------------------------+
|                                                                |
|  Axis 1: GPU PARALLELISM (raw I/O path, 4 KB)                  |
|    threads: 1, 4, 8, 16, 32, 64, 128, 256, 512, 1024          |
|                                                                |
|  Axis 2: PREFETCH DEPTH (page cache; page=64 KB, cache=1 GB)   |
|    prefetched pages: 0, 2, 4, 6, 8                            |
|                                                                |
|  Axis 3: WARP COUNT (page cache; 32 threads/warp)             |
|    warps: 1, 4, 8, 16, 32, 64, 128, 256, 512, 1024           |
|                                                                |
|  Axis 4: PAGE SIZE (page cache; 4096 pages, prefetch on)     |
|    KB: 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096     |
|                                                                |
|  Axis 5: APPLICATION (GPT2-124M, llm.c, batch 64, 3 steps)   |
|    mode: {activations in HBM}  vs  {activations offloaded}   |
|                                                                |
|  Baselines compared:                                          |
|    GPUfs (CPU-centric, 4 CPU threads)                         |
|    GDS / GPUDirect Storage (cuFile, CPU 4 GHz)               |
|    BaM (GPU-centric, no file system)                          |
|    Native (memcpy+read/write) + DLRover-RM (LLM only)        |
|                                                                |
|  Held FIXED:                                                  |
|    - 1 GPU, 1 NVMe (Optane 5800X), PCIe Gen4 x16             |
|    - FS block size = 4 KB (EXT4 page-size ceiling)          |
|    - I/O granularity = 4 KB for the raw-path sweep          |
|    - 32 GPU I/O QPs / 64 host I/O QPs                        |
|    - access pattern: sequential reads from a 20 GB file      |
|      for page-cache micro-benchmarks                        |
|    - page cache bypassed (256 MB, O_DIRECT) for SOTA cmp.   |
+----------------------------------------------------------------+
^ Fig 6: Five sweep axes plus the baseline set. The held-fixed line is
  load-bearing: single-device, 4 KB blocks, sequential access. Random
  vs sequential is NOT swept; multi-NVMe and multi-GPU are NOT swept.

The design space cleanly separates two regimes the paper keeps from contaminating each other. The raw-path regime (Axis 1) measures how well GPU-issued NVMe commands saturate one device at the smallest useful granularity, and it is where GeminiFS is benchmarked against GPUfs/GDS/BaM with caching off. The cache regime (Axes 2-4) measures the HBM-side ceiling with the device factored out entirely. Because these are separated, a reader can attribute any given number to either "how fast can the GPU talk to the SSD" or "how fast can the GPU talk to its own cache," but never both at once — a clean experimental decomposition.

4. Algorithm / Control Flow Diagrams

The lifecycle is five staged calls (paper Fig. 5): init, pre-allocate, open, read/write, sync/close. Initialization and pre-allocation are host-side and one-time; the read/write path is the hot loop and runs entirely on the GPU with no host involvement.

  Geminifs_init(dev_path, GPU_ids, Q_num)        [host, once]
       |
       v
  (1) create NVMe I/O queues on CPU AND GPU
       |  - allocate I/O-QP memory in GPU HBM
       |  - p2p pin + DMA-map (GPU vaddr -> DMA addr)
       v
  (2) initialize NVMe controller (admin queue, host-only)
       |
       v
  GVDK pre-allocation                            [host, per file]
       |  2.1 create file + pre-allocate space
       |  2.2 host kernel returns physical block offsets
       |  2.3 GVDK Helper embeds NVMe offsets
       |  2.4 build L1/L2 mapping table
       |  2.5 embed mapping table into file
       |  2.6 flush metadata to GVDK file on NVMe
       v
  DONE -> device is ready, files are GVDK-formatted
^ Fig 7: One-time control plane + file provisioning. The pin/DMA-map in
  step (1) is the only place GPU memory is made NVMe-visible; everything
  afterward reuses those mapped queues.
  G_open(path, flag, cache_capacity, page_size)  [host]
       |
       +-- flag has access mode (RDONLY/WRONLY/RDWR)?  --no--> error
       |
       +-- path missing AND O_CREAT? --> generate GVDK file
       |
       +-- O_DIRECT set? --> bypass page cache for this fd
       |
       v
  3.1 read metadata + L1/L2 mapping table from file head
       |
       v
  3.2 fill Metadata Cache in GPU memory; return dev_fd
^ Fig 8: Open resolves a path to a GPU-resident dev_fd whose metadata and
  block map already live in HBM, so later translation needs no host trip.
  G_read / G_write(dev_fd, buf, offset, nbyte)   [GPU, hot path]
       |
       v
  4.1 locate metadata via dev_fd; round up to IO block size
       |  - check access mode
       |  - check offset+nbyte within file bounds
       v
  translate vaddr: m=(m1,m2,m3)
       |  m1 -> L1 entry -> L2 table NVMe offset
       |  m2 -> L2 entry -> data block NVMe offset
       |  m3 -> byte within cluster
       v
  page cache consult (unless O_DIRECT)
       |
       +-- HIT --> read/write in HBM, done (no NVMe I/O)
       |
       +-- MISS -+
       |         v
       |   4.2 build NVMe request, submit to SQ (GPU memory)
       |         (warp may issue many cmds at page granularity
       |          = prefetch)
       |         v
       |   4.3 device DMA: NVMe <===> GPU memory (P2P, no CPU)
       |         v
       |   poll CQ for completion
       v
  return bytes transferred
^ Fig 9: The hot path. Address translation is an in-HBM two-level table
  walk; on a cache miss the GPU itself fills the SQ, the NVMe DMAs
  straight to/from HBM, and a GPU thread polls the CQ. The CPU is never
  on this path.
  G_sync(dev_fd)  [GPU]            G_close(dev_fd)  [host]
       |                                |
       v                                v
  flush dirty file metadata        5.1 locate metadata + page cache
  + dirty cached pages to NVMe          via dev_fd
       |                                |
       v                                v
  update dirty bitmap              5.2/5.3 ensure dirty data flushed,
       |                                sync metadata on host
       v                                v
  persistence guaranteed           release GPU memory resources
^ Fig 10: Durability is opt-in. G_sync pushes dirty pages + metadata and
  stamps the dirty bitmap; G_close drains and frees. Crash consistency is
  left to the application precisely because the workloads are read-mostly
  / append-only.

4.1 The GPU page cache as a concurrent data structure

The page cache is where GeminiFS spends its cleverness, because a naive lock would serialize thousands of GPU threads. Two ideas keep the critical section short: lock at warp granularity, not thread granularity, and make every cache operation constant-time.

+------------------ GPU page cache internals -------------------+
|                                                               |
|  Lookup (hit test):                                           |
|    hash table: file page  -->  HBM page    (O(1))            |
|                                                               |
|  Miss / eviction (zero-reference set):                        |
|    doubly linked list  +  hash table                          |
|                                                               |
|     head (coldest) <-> ... <-> tail (just freed)             |
|       ^                          ^                            |
|       |                          |                            |
|    evict here             page released here                 |
|                                                               |
|    HIT:  if page in zero-ref set, remove it (O(1))           |
|    FREE: page becomes zero-ref -> append at tail (O(1))      |
|    MISS: reclaim from head (coldest) (O(1))                  |
|                                                               |
|  Concurrency cap (Ampere): 4 warp sched x 108 SM = 432       |
|    control flows can contend the lock at once                 |
|    (vs thousands at thread granularity)                       |
|                                                               |
|  Cross-process sharing:                                        |
|    cuIpcGetMemHandle  + nvidia_p2p_get_pages_persistent       |
|    -> one page cache per file, shared by IPC handle           |
+---------------------------------------------------------------+
^ Fig 11: An LRU realized as a hash table (hit test) plus a doubly linked
  zero-reference list (eviction). Warp-level locking bounds contenders to
  ~432 on Ampere; IPC handles let multiple GPU processes share one cache.

The eviction structure is a textbook LRU, but its implementation discipline is the point: every operation a contending warp performs inside the lock is O(1), so the lock is held for a near-constant, minimal window. The warp-level acquisition then bounds the number of simultaneous contenders to the hardware's actual concurrency (4 schedulers x 108 SMs on Ampere), turning a thousands-way stampede into a 432-way one. This is the design choice that lets the cache scale to 640+ GB/s rather than collapse under lock contention.

5. Quantitative Results — Empirical Findings by Regime

5.1 Raw 4 KB read bandwidth (cache bypassed) — Fig. 6

Regime / comparison Result
vs GPUfs (avg over threads) GeminiFS = 7.33x GPUfs bandwidth
at 1024 threads GeminiFS reaches NVMe peak (~7 GB/s)
vs GDS, 1-16 threads GDS +57% over GeminiFS (CPU 4 GHz)
vs GDS, 128-512 threads GeminiFS = 6.2x GDS
vs BaM (all parallelism) GeminiFS -4.6% (metadata/xlate tax)

The shape of this result is the whole thesis in one figure. At low parallelism the CPU-orchestrated GDS wins because a 4 GHz core out-issues a 1 GHz GPU; as parallelism climbs, the GPU's physical concurrency takes over and GeminiFS pulls to 6.2x GDS and saturates the device. The 4.6% deficit to BaM is the measured price of having a file system at all — metadata parsing and the L1/L2 translation that BaM's raw-block interface skips.

5.2 Raw 4 KB read latency — Fig. 7

Regime / comparison Result
vs GPUfs (across threads) 79.6% - 90.9% latency reduction
vs GDS, 1-8 threads GeminiFS +57.2% higher latency
vs GDS, 1024 threads GeminiFS = 17% of GDS latency
vs BaM GeminiFS +4.8% higher latency

The latency curves mirror the bandwidth curves: GeminiFS starts behind GDS at trivial parallelism and ends six-fold ahead at saturation, because GDS's latency climbs steeply (CPU-core contention) while GeminiFS's stays flat. Against BaM the gap is a near-constant 4.8% — again, the file-system tax, not a scaling problem.

5.3 Page cache — prefetch depth — Fig. 8 (page 64 KB, cache 1 GB)

State Read BW (% of peak) Write BW (% of peak)
Prefetch OFF 30.2% 28%
Prefetch ON (any N) ~2.4x improvement ~2.34x improvement

Prefetch is not a tuning luxury here — it is mandatory. Without it a single warp issues too little outstanding I/O to keep the device pipeline full and the cache stalls at ~30% of theoretical bandwidth. Issuing many NVMe commands per warp at page granularity (the prefetch mechanism) recovers ~2.3-2.4x, and crucially the number of prefetched pages barely matters past enabling it — the win is in having depth at all, not in its exact value.

5.4 Page cache — warp count — Fig. 9 (32 threads/warp)

Metric Min (1 warp) Max (1024 warps)
Write bandwidth ~1.7 GB/s ~641.2 GB/s
Read bandwidth ~2.3 GB/s ~658.1 GB/s
Peak observed - > 640 GB/s (both)

Bandwidth scales nearly multiplicatively with warp count to a ~650 GB/s plateau — roughly a third of HBM's 1935 GB/s ceiling — confirming that the warp-level concurrent cache actually delivers on its concurrency promise rather than serializing under the lock.

5.5 Page cache — page size — Fig. 10 (128 warps, 4096 pages, prefetch on)

Page size Write BW Read BW
4 KB ~45.8 GB/s ~48.4 GB/s
1024 KB ~120.1 GB/s ~121.4 GB/s

Larger pages climb toward the ~120 GB/s memcpy ceiling of this experiment because a bigger page means a warp touches fewer pages for the same data, which means fewer page-cache lock acquisitions and less contention. Page size is therefore a direct contention knob: trade more memory per page for fewer critical-section entries.

5.6 Application — GPT2-124M training — Fig. 11

Mode A — activations kept in HBM (offload disabled):

Comparison Total runtime reduction Checkpoint-write reduction
vs Native 25% 85%
vs DLRover-RM 12% 75%
vs GDS 10% 59%

Mode B — activations offloaded to storage (offload enabled):

Comparison Training-time reduction
vs Native 94.5%
vs GDS 91%
vs HBM-only GeminiFS is only ~4x slower than keeping all in HBM

Mode A is compute-bound, so the end-to-end win is modest (10-25%) even though the checkpoint I/O itself is cut by up to 85%. Mode B is the showcase: activation offloading is a flood of small I/Os, exactly where CPU-centric paths drown in synchronization, and GeminiFS's high-bandwidth cache turns a catastrophe into a ~4x slowdown versus the physically impossible all-in-HBM baseline (57.96 GB of activations against 80 GB HBM). Storage Table 3: model weights 238 MB (R/W), checkpoint 713 MB/step (append-only), activations 57.96 GB (R/W).

6. Configuration-Regime Trade-off Tables

6.1 I/O orchestration paradigm

Dimension CPU-centric (GPUfs/GDS) GPU-centric (GeminiFS) Winner
Low parallelism (1-16) Fast (4 GHz core) Slower (1 GHz core) CPU-centric
High parallelism (>=128) CPU-core contention Saturates NVMe GPU-centric
Tail latency at 1024 thr Surges (~250% in Fig.2) Flat GPU-centric
Small-I/O offload (Mode B) Collapses Full BW utilization GPU-centric
CPU-GPU sync overhead On every I/O None on data path GPU-centric
Batch ceiling GDS <=128 ops/batch None GPU-centric

The crossover at ~16-128 threads is the load-bearing fact: the right paradigm depends entirely on how many GPU threads demand I/O at once. Below the crossover, orchestrating from a fast CPU core wins; above it, the GPU's raw concurrency wins and the CPU becomes the bottleneck.

6.2 File abstraction vs raw block access

Dimension Raw block (BaM) Companion FS (GeminiFS) Winner
4 KB read bandwidth Baseline -4.6% BaM (margin)
4 KB read latency Baseline +4.8% BaM (margin)
File semantics / isolation None Yes GeminiFS
Host/GPU data sharing Manual copy via host Embedded metadata GeminiFS
Crash consistency option None Dirty bitmap + sync GeminiFS
Capacity overhead 0% ~0.2% BaM (margin)

This is the central design trade GeminiFS makes explicit: a flat ~4.5-5% performance tax buys the entire file abstraction BaM discards. For ML pipelines that must share weights and checkpoints across processes, that tax is trivial against the cost of round-tripping through host memory the way raw-block systems force.

6.3 Page-cache tuning knobs

Knob Low setting High setting Winner / effect
Prefetch depth 0 -> ~30% BW >=2 -> ~2.3-2.4x ON (any depth)
Warp count 1 -> ~2 GB/s 1024 -> ~650 GB/s High (more concurrency)
Page size 4 KB -> ~46 GB/s 1024 KB -> ~121 GB/s Large (less lock contend)
Lock granularity per-thread (thousands) per-warp (~432) Per-warp

For DynamICCL-style tuning intuition: prefetch is a binary "must enable," warp count is "more is better up to the plateau," and page size is the genuine continuous trade (memory footprint vs lock contention). None of these interact perversely — they are monotone within the measured range, which is what makes the cache predictable to configure.

7. Bottlenecks & Insights Surfaced by the Measurements

The headline insight (Fig. 2) is that for CPU-centric GPU storage, the software stack — not the device — is over 90% of I/O latency at the measured points. GPUfs sits above 190 us per access on both a 15 us TiPro and a 4 us Optane, which means the flash is idle more than 90% of the time waiting on host software. GeminiFS exists to delete that 90%, and the 4 KB latency reductions of 79.6-90.9% versus GPUfs are essentially that deletion made visible.

The second insight is a clock-versus-concurrency crossover. Because the GPU core is ~1 GHz against the CPU's 4 GHz, any single GPU thread is a worse I/O issuer than a CPU thread — so GDS legitimately wins at 1-16 threads. GeminiFS only overtakes once enough warps are in flight that the GPU's physical parallelism overwhelms the per-issue clock disadvantage. This is why the same system can be behind GDS by 57% in one regime and ahead by 6.2x in another; the regime, not the system, decides.

The third insight is that GPUfs's wall is CPU-core contention: once GPU threads exceed CPU core count, average and tail latency surge (~250% at 1024 threads) because a bounded pool of CPU helpers cannot service an unbounded fan-in of GPU requesters. GeminiFS removes the CPU from the data path entirely, so there is no pool to contend for.

The fourth insight is that prefetch is the difference between 30% and ~100% of cache bandwidth. A warp that issues one outstanding request at a time cannot keep the NVMe pipeline full; issuing many at page granularity does. This is the storage analog of keeping a deep enough in-flight window to cover latency — without depth, bandwidth is left on the table regardless of how fast the device or cache is.

The fifth insight is that the page cache is structurally un-bottleneckable on this rig: at 640+ GB/s it would take close to 100 Optane drives to saturate, so on any single-device (or even modest multi-device) system the device, not the cache, is always the limiter. That justifies spending HBM on the cache — its ceiling is so far above the device that it effectively disappears as a constraint.

8. Limitations of the Methodology

Limitation Consequence
Single GPU, single NVMe No multi-device aggregation / multi-GPU data
No RAID / file-splitting yet Multi-NVMe bandwidth scaling unmeasured
Random vs sequential not swept Cache benchmarks use sequential 20 GB reads only
FS block size fixed at 4 KB EXT4 page-size ceiling; larger blocks untested
Cache benchmarked with memcpy, not NVMe Cache numbers are an upper bound, device removed
Cache bypassed (256 MB, O_DIRECT) for SOTA Two regimes never measured jointly
One real workload (GPT2-124M) No GNN / vector-search / KV-store app numbers
No crash consistency by default Durability is opt-in; relies on workload being
read-mostly / append-only
Low-parallelism regime is a loss to GDS GPU clock disadvantage unaddressed below ~16 thr
Non-privileged-mode constraint Page cache must live inside the GPU process
Predictability assumption Design leans on access patterns being known
beforehand; unpredictable workloads underserved

The most consequential gap is the single-device testbed against a multi-device future. The authors openly position multi-GPU and RAID-style NVMe aggregation as future work, which means the page cache's 640 GB/s — the number that makes the "cache never bottlenecks" claim — has not actually been stressed against a storage tier fast enough to challenge it. The second is the two-regime separation: bypassing the cache for the SOTA comparison and removing the device for the cache characterization each yields a clean number, but the paper never reports the combined, production-realistic path where cache hits and device misses interleave.

9. Note on NCCL Tuning

GeminiFS's page-cache sweep is, structurally, the same pipelined-transfer tuning problem a collective library faces, just expressed in storage vocabulary. Its prefetch depth is a buffering window — too shallow and a single warp cannot keep the device pipeline full, leaving bandwidth at ~30% exactly the way too few in-flight chunks starve a transfer; its page size trades memory footprint against lock-contention the way a larger transfer chunk trades buffer occupancy against per-message overhead; and its warp count is a parallelism dial that climbs to a plateau much like splitting one transfer across more parallel lanes. The non-obvious takeaway is that all three knobs are monotone within the measured range — enable prefetch, prefer larger pages, add warps to the plateau — so the only genuine continuous trade is footprint-versus-contention. A tuner that recognizes which knobs are binary "always on" versus genuinely continuous can skip exploring the former and spend its budget on the latter.

10. Analogy

GeminiFS is a warehouse where the forklifts learned to read the shelf labels themselves. In the old arrangement (CPU-centric I/O), every forklift on the floor — thousands of them — had to radio a small crew of clerks in the front office (the CPU cores) to look up where each pallet lived, and the clerks read back the aisle and bin. With a handful of clerks and a flood of forklifts, the radio queue is the whole delay: the pallets are sitting right there, but 90% of the time is spent waiting on the front office. GeminiFS prints the bin location directly onto the first pallet of every shipment (embedded GVDK metadata) and gives every forklift a pocket index (the cached L1/L2 table), so a forklift resolves any location on its own and drives straight to the rack over a dedicated lane (P2P DMA) without ever calling the office. The front office still runs the warehouse — it assigns the bins, owns the master ledger, hires and fires (the host keeps admin control of the device) — but it is off the critical path for fetching. The page cache is a staging area near the loading dock stocked with the pallets most likely to be needed next, and the rule that forklifts grab pallets one team at a time rather than one worker at a time (warp-level locking) is what keeps the staging area from becoming a scrum. The single catch the analogy preserves faithfully: a lone clerk with a fast phone (the 4 GHz CPU at low parallelism) still beats one slow forklift radioing in — GeminiFS only wins once the floor is genuinely crowded, which, for modern ML workloads, it always is.