Architecture & Measurement-Design Analysis

LithOS: An Operating System for Efficient Machine Learning on GPUs

Source: Coppock, P. H.; Zhang, B.; Solomon, E. H.; Kypriotis, V.; Yang, L.; Sharma, B.; Schatzberg, D.; Mowry, T. C.; Skarlatos, D. Proc. ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP '25), Seoul, Oct 13-16 2025. DOI: 10.1145/3731569.3764818 Affiliations: Carnegie Mellon University; Meta (†). Reader: general-purpose subagent via native Read (gemini-reader quota exhausted; codex-reader unavailable on account). Analyst: Vishwakarma Date: 2026-06-29

Table of Contents

1. System Architecture (the "GPU OS" stack)
2. System-Under-Test Architecture (the hardware specimen)
3. Design-Space Diagram (multitenancy x workload x mechanism axes)
4. Layered Stack — Where LithOS Inserts Itself
5. Algorithm / Control-Flow Diagrams (the four mechanisms)
6. Kernel/Atom Lifecycle State Machine
7. Quantitative Results — Empirical Findings by Regime
8. Configuration-Regime Trade-off Tables
9. Bottlenecks & Insights Surfaced by the Measurements
10. Limitations of the Methodology
11. Analogy

1. System Architecture (the "GPU OS" stack)

LithOS is a userspace GPU operating-system layer that interposes at the CUDA Driver API and arbitrates a single physical GPU among multiple unmodified ML applications of differing priority. Its thesis is that the right unit of GPU resource management is the TPC (Texture Processing Cluster) — finer than MIG's GPC-aligned partitions, coarser than intra-SM scheduling, which is best left to hardware and compilers. Around that unit it composes four cooperating mechanisms: a TPC Scheduler, a Kernel Atomizer, Hardware Right-sizing, and Power Management. A shared Online Latency Prediction module feeds all four, so that no offline profiling is required.

+-------------------------------------------------------------------+
|                  Unmodified ML Applications                       |
|                                                                   |
|  +----------------+  +----------------+  +--------------------+   |
|  | TensorRT       |  | JAX            |  | PyTorch            |   |
|  | (High Prio A)  |  | (High Prio B)  |  | (Best Effort)      |   |
|  +-------+--------+  +-------+--------+  +---------+----------+   |
|          |                   |                    |               |
|          +-------------------+--------------------+               |
|                              |  (CUDA Driver API calls)           |
|                              v                                    |
|  +-------------------------------------------------------------+  |
|  | LibLithOS  (drop-in, mimics native CUDA library)            |  |
|  |   intercepts cuLaunchKernel / cuStreamCreate / ...          |  |
|  |   rest of Driver API auto-generated; semantics preserved    |  |
|  +-------------------------------+-----------------------------+  |
|                                  |  (intercepted launches)        |
|  ============================== userspace ====================    |
|                                  v                                |
|  +-------------------------------------------------------------+  |
|  |                     LithOS core  (~5000 lines Rust)         |  |
|  |    system-wide view of GPU state across all priorities      |  |
|  |                                                             |  |
|  | +-------------+ +-------------+ +----------+ +-----------+   |  |
|  | | TPC         | | Kernel      | | Hardware | | Power     |   |  |
|  | | Scheduler   | | Atomizer    | | Right-   | | Mgmt      |   |  |
|  | | quotas,     | | Prelude +   | | sizing   | | kernel-   |   |  |
|  | | TPC Steal,  | | QMD patch,  | | scaling  | | dependent |   |  |
|  | | launch/sync | | -> atoms    | | model+k  | | DVFS      |   |  |
|  | | queues,     | |             | |          | |           |   |  |
|  | | Tracker thr | |             | |          | |           |   |  |
|  | +------+------+ +------+------+ +----+-----+ +-----+-----+   |  |
|  |        |               |             |             |         |  |
|  |        +-------+-------+------+------+------+-------+         |  |
|  |                |              |             |                 |  |
|  |        +-------v--------------v-------------v-------+         |  |
|  |        | Online Latency Prediction module          |         |  |
|  |        | (operator ordinal index k; no profiling)  |         |  |
|  |        +-------------------------------------------+         |  |
|  |        libsmctrl-derived TPC mapping (QMD), per-TPC timers   |  |
|  +-------------------------------+-----------------------------+  |
|                                  |  (atoms -> device queues)      |
|                                  v                                |
|  +-------------------------------------------------------------+  |
|  | GPU Device Driver (CUDA Driver, MPS substrate)              |  |
|  +-------------------------------+-----------------------------+  |
|                                  v                                |
|  +-------------------------------------------------------------+  |
|  | GPU Hardware: GPC -> [TPC] -> SM -> cores ; L2 + HBM        |  |
|  |   LithOS maps each atom to specific TPCs (some idle)        |  |
|  +-------------------------------------------------------------+  |
+-------------------------------------------------------------------+
^ Fig 1: LithOS system architecture. Unmodified apps link LibLithOS
  instead of the native CUDA library; the four core mechanisms share a
  system-wide GPU-state view and a single latency predictor, then map
  atomized work onto individual TPCs below the userspace boundary.

Three architectural commitments define the design. First, transparency: the only thing an application changes is which .so it links — LibLithOS presents the native CUDA Driver surface, so no framework, runtime, PTX, or source change is needed. Second, a single system-wide scheduler rather than per-process heuristics: because LibLithOS sees every launch from every tenant, LithOS can make global packing and preemption decisions, which is exactly what MPS (per-context, no coordination) cannot do. Third, prediction-driven control: all four mechanisms consume the same online latency estimate, so the system is self-calibrating and avoids the offline-profiling fragility of prior GPU schedulers.

The data path is a deferred-dispatch pipeline. An application's cuLaunchKernel does not reach the driver directly; LibLithOS enqueues it into a per-stream launch queue and returns control immediately. The TPC Scheduler then decides when and on which TPCs the work runs, the Atomizer optionally splits it into atoms (thread-block subsets), right-sizing trims the TPC count, DVFS sets the clock, and only then do atoms enter the GPU's device queues. A Tracker thread watches completions, updates per-TPC timers, and refines the predictor — closing the loop.

2. System-Under-Test Architecture (the hardware specimen)

The evaluation runs on a single NVIDIA A100 (SXM4) with 108 SMs and 40 GB, hosted on a 30-core, 216 GB Lambda Labs instance. The interest of the SUT is not its scale but the hardware abstraction LithOS reverse-engineers: NVIDIA's GPU -> GPC -> TPC -> SM -> core hierarchy, and the undocumented mapping from a launched kernel to the TPCs it occupies.

+----------------- NVIDIA GPU hardware hierarchy -------------------+
|                                                                   |
|   GPU                                                             |
|    |                                                              |
|    +-- GPC 0 ---------------------------------------+             |
|    |    |                                           |             |
|    |    +-- TPC 0  <== LithOS scheduling unit ==>   |             |
|    |    |     +-- SM 0  (CUDA cores, Tensor cores,  |             |
|    |    |     |          L0 i-cache, regfile,       |             |
|    |    |     |          warp sched, L1/shared)     |             |
|    |    |     +-- SM 1                               |             |
|    |    +-- TPC 1 ... TPC 8                          |             |
|    +-- GPC 1 ... GPC 7                               |             |
|    |                                                 |             |
|    +-- L2 Cache  ---  HBM / Memory                   |             |
|    +-------------------------------------------------+             |
|                                                                   |
|   Example capacities (H100): 8 GPC x 9 TPC x 2 SM x 128 cores     |
|   Testbed (A100 SXM4):       108 SMs total, 40 GB                 |
+-------------------------------------------------------------------+
^ Fig 2: GPU hardware hierarchy. LithOS chooses the TPC as its unit:
  MIG partitions only along GPC boundaries (too coarse, >5 s to
  reconfigure); intra-SM scheduling is ceded to hardware/compilers.

The TPC -> kernel mapping is not exposed by any driver API. LithOS extends the prior reverse-engineering tool libsmctrl, adding Hopper support including its new TPC-masking layout, and reimplements TPC identification through the Queue MetaData (QMD) structure to enable dynamic TPC allocation at launch time. NVIDIA's Hopper Thread Block Clusters are also reverse-engineered, so atoms are always emitted as multiples of the cluster size. The technique is verified across Ampere (A30, A100), Hopper (H100), and Ada Lovelace (L4); the prototype supports Ampere and Hopper and runs on top of MPS to obtain concurrent execution across separate GPU contexts.

  Software/hardware versions (testbed):
  +-----------------------------------------------------------+
  | Ubuntu 22.04 | CUDA 12.6 | Rust 1.83.0-nightly            |
  | Python 3.10  | PyTorch 2.3 | TensorRT 10.1                |
  | TensorRT-LLM 0.11.0 | Triton 24.07                        |
  +-----------------------------------------------------------+

The choice of a single A100 is deliberate: every mechanism LithOS introduces is intra-GPU (spatial TPC packing, temporal atom switching, per-kernel right-sizing, per-stream DVFS). Multi-GPU collective scheduling and host-side networking are explicitly out of scope, which keeps the measured effects attributable to LithOS's TPC-level control rather than to interconnect behaviour.

3. Design-Space Diagram (multitenancy x workload x mechanism axes)

The experiments sweep a multitenancy environment, a pair of workload mixes, a mechanism under test, and a baseline system. Hardware (one A100), the latency-slip parameter (k = 1.1), and memory residency (every model fits in GPU DRAM) are held fixed.

                   DESIGN SPACE (4 swept axes + held-fixed)
  +---------------------------------------------------------------+
  |                                                               |
  |  Axis 1: MULTITENANCY ENVIRONMENT (2 levels)                  |
  |    [Inference-only : 2 High-Prio + 1 Best-Effort]            |
  |    [Hybrid         : 1 HP inference + 1 BE training]         |
  |                                                               |
  |  Axis 2: WORKLOAD (HP inference x BE)                         |
  |    HP inference: RetinaNet, YOLOv4, ResNet-50 v1.5,          |
  |                  Llama 3 8B, GPT-J 6B, BERT-Large            |
  |    BE training : ResNet-50, MobileNetV2, VGG-19, DLRM,       |
  |                  BERT-Large, Llama 3 Finetuning             |
  |    swept within: BE train batch size; BE inf seq length      |
  |                                                               |
  |  Axis 3: MECHANISM UNDER TEST (4 + composition)              |
  |    [TPC Scheduler] [Kernel Atomization]                       |
  |    [Hardware Right-sizing] [Kernel-dependent DVFS]            |
  |    swept within: TPC count 1..54 ; freq 750..1410 MHz        |
  |                                                               |
  |  Axis 4: BASELINE SYSTEM (9 levels)                           |
  |    Time-slicing, MPS, stream Priority, MIG, thread Limits,    |
  |    TGS, REEF, Orion, vanilla NVIDIA                          |
  |                                                               |
  |  Held FIXED:                                                  |
  |    - Hardware: single A100 (SXM4), 108 SMs, 40 GB            |
  |    - latency slip k = 1.1 (right-sizing AND DVFS)           |
  |    - all models resident in GPU memory (<= 1/2 DRAM each)    |
  |    - HP load: Poisson via Triton, ~80% util target (hybrid)  |
  |    - SLOs: MLPerf inference, 2.3x..7.4x baseline latency     |
  |    - right-sizing + power OFF during scheduling comparisons  |
  |                                                               |
  +---------------------------------------------------------------+
^ Fig 3: 4-axis design space. The mechanism axis is the contribution;
  the 9-baseline axis is the measuring stick. Right-sizing and power
  management are disabled during the apples-to-apples scheduling
  comparison so the scheduler's effect is isolated.

Two scoping decisions shape the empirical content. First, memory is held constant by construction — every model is sized to occupy at most half of DRAM so it stays resident, which lets the paper attribute all sharing gains to compute isolation rather than memory contention (the authors separately bound bandwidth isolation at a 4-13% effect versus compute isolation's >10x effect). Second, the mechanisms are evaluated both in isolation and in composition: the scheduling comparison disables right-sizing and DVFS, while the ablation (Fig. 9 here) re-enables atomization on top of the scheduler to show the marginal tail-latency gain.

4. Layered Stack — Where LithOS Inserts Itself

+------------------------------------------------+
| Unmodified ML app (TensorRT / JAX / PyTorch)   |  <- user-facing
+------------------------------------------------+
| ML framework + cuDNN / TensorRT-LLM / Triton   |  <- unchanged
+------------------------------------------------+
| LibLithOS  (drop-in CUDA Driver shim)          |  <- insertion point
+------------------------------------------------+
| LithOS core: TPC Sched | Atomizer | Right-size |  <- the OS layer
|              | Power | Latency Predictor       |
+------------------------------------------------+
| CUDA Driver + MPS substrate                    |  <- relied upon
+------------------------------------------------+
| GPU hardware: GPC / TPC / SM / cores + HBM     |  <- arbitrated
+------------------------------------------------+
^ Fig 4: Software stack. LithOS occupies the thin layer between the
  unmodified framework and the driver. Because the shim presents the
  native Driver API, everything above it is byte-for-byte unchanged;
  everything below it (driver, MPS, hardware) is relied upon as-is.

The insertion point is the lowest-common-denominator interface: every ML framework ultimately funnels through the CUDA Driver API, so interposing there captures TensorRT, JAX, PyTorch, TensorFlow, ONNX Runtime, and Triton with one shim. This is the same design logic as a syscall-level interposition layer in a classical OS — by choosing the narrowest universal interface, LithOS gains control over every tenant without per-framework engineering. The cost is that LithOS must reverse-engineer the driver-private TPC mapping that a kernel-resident scheduler would get for free.

5. Algorithm / Control-Flow Diagrams (the four mechanisms)

5.1 TPC Scheduler dispatch loop

  (1) Kernel Submission  (~ms)
        app -> LibLithOS -> per-stream LAUNCH QUEUE (buffered)
        |   dispatch deferred: eager dispatch cedes control to HW
        v
  (2) TPC Control / Compute Quotas
        apply quota xi -> guarantee app its TPC allotment
        TPC STEALING: lend idle TPCs to other tasks (work-conserving)
        |
        v
  (3) Kernel Atomization  (~us)   [only if predicted long-running]
        predict duration -> n_atoms = duration / atom_duration
        split grid blocks into n_atoms non-overlapping atoms
        |
        v
  (4) Submit atoms to GPU device queues on the chosen TPCs
        |
        v
  (5) Outstanding Work / SYNC QUEUES
        throttle until backlog < 100 us (covers host-device latency)
        Tracker thread monitors completion -> update scheduler state
        |
        v
  (6) Right-sizing: trim TPC count per kernel/atom (model + slip k)
        |
        v
  (7) DVFS: set GPU frequency from sequence-based model
        |
        v
  COMPLETE -> Tracker updates per-TPC timers + latency predictor
^ Fig 5: Scheduler dispatch loop. Steps map to Fig. 9 markers (1)-(7)
  in the paper. Launch queues decouple submission from execution;
  sync queues bound outstanding work; the Tracker thread closes the
  feedback loop into the latency predictor.

The load-bearing idea is deferred dispatch. Once a kernel reaches the GPU its priority and resource allocation are frozen — it cannot be rescheduled. So LithOS holds work in launch queues and dispatches the minimum needed to keep the GPU busy (backlog < 100 us), preserving the freedom to insert a newly-arrived high-priority request ahead of best-effort work. This is the GPU analogue of keeping a short run queue so the scheduler always has a preemption opportunity.

5.2 Kernel Atomization (Prelude + QMD patch)

  Original kernel: grid {8,8,1} = 64 thread blocks
        |
        |  split into 2 atoms:
        |     atom A: block_idx in [0, 32)
        |     atom B: block_idx in [32, 64)
        v
  +-------------------------------------------------------------+
  | Prelude kernel (Algorithm 1), launched with SAME config     |
  |                                                             |
  |   if (atom->block_idx_lo <= block_idx < block_idx_hi)       |
  |        call original_kernel_entrypoint(...)                 |
  |   else exit early                                           |
  +-------------------------------------------------------------+
        ^
        |  implemented by patching the QMD program address:
        |   1. launch original kernel -> driver configures env
        |   2. patch QMD -> entry now points at Prelude
        |   3. execution starts at Prelude, keeps kernel resources
        v
  atoms inherit the kernel's allocated TPCs; n_atoms = dur/atom_duration
  practical atom size: 250-500 us (smaller -> per-block overhead grows)
^ Fig 6: Atomization. A kernel is split into thread-block-range subsets
  with no compiler, runtime, or PTX change. The Prelude gates each
  block by index; the QMD program-address patch redirects execution
  while preserving the original launch environment.

Atomization converts a coarse, multi-millisecond kernel (DLRM kernels exceed 30 ms) into microsecond-scale schedulable units. This is what makes sub-millisecond inference SLOs survivable in the presence of long best-effort training kernels: a high-priority request need only wait for the current atom, not the whole kernel. The tunable atom_duration trades packing granularity against per-thread-block launch overhead, with a practical floor of 250-500 us.

5.3 Hardware Right-sizing (per-kernel)

  on kernel submission:
        |
        v
  (a) FILTERING HEURISTIC (upper bound)
        max_useful_TPC = ceil(total_blocks / occupancy_per_TPC)
        occupancy from CUDA_OCCUPANCY API; blocks from atomizer
        |
        +-- if max_useful_TPC < allocated -> launch with that bound
        |
        v
  (b) SCALING MODEL          l = m/t + b
        l = predicted latency, t = #TPCs
        b = single-block-on-one-SM time   (Amdahl serial part)
        m = parallel benefit              (Amdahl parallel part)
        fit from 2 points: latency @ all TPCs and @ 1 TPC
        |
        v
  choose min t such that latency increase <= factor k  (k = 1.1)
^ Fig 7: Right-sizing. A two-point Amdahl-style curve plus an occupancy
  upper bound yields the minimal TPC count that respects a 10% slip.
  Per-kernel, because no single TPC config fits all kernels in a model.

5.4 Kernel-dependent DVFS

  per stream, sequence-based first-order Taylor model:
        w   = kernel_runtime / sum(all kernel runtimes)   (weight)
        s   = k / (f_max/f_th - 1)                         (sensitivity)
        S   = sum( w * s )                                 (aggregate)
        f_final = f_max / (1 + k/S)                        (clamp slip)
  memory-bound kernels saturate BW early -> tolerate low freq
  compute-bound kernels -> keep high freq
  start at f_max (1410 MHz), refine; switching latency ~50 ms -> conservative
^ Fig 8: DVFS control. Frequency is virtualized per workload; the slip
  parameter k bounds slowdown. Slow (~50 ms) frequency switching forces
  a conservative, sequence-aggregated policy rather than per-kernel.

5.5 Composition ablation (scheduler -> +stealing -> +atomization)

  (a) TPC SCHEDULING only          (b) + TPC STEALING
   A| ####    idle....              A| ######## (stole idle)
   B| ####    idle....              B| ####  [blocked by C2!]
   C| ##  ##                        C| ##  ##
   wasted capacity                  less waste, risk HoL/priority-inv

  (c) + KERNEL ATOMIZATION
   A| ######## packed
   B| ######## (no longer blocked; stealing disabled for C2 atoms)
   C| a a a a  (atomized -> fast switch)
   optimal packing, minimal HoL, all finish sooner
^ Fig 9: Three-phase scheduling story (paper Fig. 10). Spatial packing
  (stealing) reclaims idle TPCs but can invert priorities; temporal
  atomization restores fast preemption so the high-priority tenant is
  no longer blocked. Both axes are required.

The central systems lesson is here: spatial and temporal partitioning are both necessary. TPC stealing is the spatial mechanism that fills idle capacity; atomization is the temporal mechanism that keeps that filling preemptible. Either alone is insufficient — stealing alone causes head-of-line blocking, atomization alone leaves capacity idle.

6. Kernel/Atom Lifecycle State Machine

        submit
  [SUBMITTED] -----> buffered in launch queue
       |             (cannot re-prioritize once on GPU -> hold here)
       v
  [PREDICTED] -----> latency module estimates duration via
       |             operator ordinal index k
       v
  [RIGHT-SIZED] ---> filtering heuristic or scaling model + slip k
       |             -> minimal TPC count
       v
  [ATOMIZED] ------> if long-running: split into atoms (Prelude/QMD)
       |             else: launch whole
       |             (stealing+atomization DISABLED for cross-block-sync
       |              / persistent kernels)
       v
  [DISPATCHED] ----> assigned to specific TPCs (quota + stealing);
       |             DVFS frequency set
       v
  [EXECUTING] -----> tracked by sync queues; throttle backlog < 100 us;
       |             per-TPC timers guard against stealing-induced HoL
       v
  [COMPLETED] -----> Tracker thread updates scheduler state, clears
                     sync queues, updates timers, refines predictions
^ Fig 10: Kernel/atom lifecycle. The freeze at SUBMITTED-onto-GPU is
  why LithOS keeps work in launch queues: all reordering authority
  must be exercised before the kernel crosses into the device queue.

The state machine exposes a single hard constraint that drives the whole design: once a kernel is on the GPU it is immutable. Every degree of freedom LithOS exploits — reprioritization, right-sizing, atom splitting, frequency choice — must be spent before the DISPATCHED transition. The launch and sync queues exist precisely to widen the window in which those decisions are still legal.

7. Quantitative Results — Empirical Findings by Regime

7.1 Headline numbers (abstract)

7.2 Inference-only multitenancy (2 HP + 1 BE)

MPS throughput bar = 1.0 but SLO attainment only 45%. TGS leads non-isolating systems at 84% SLO. LithOS reaches 100% SLO at throughput 1.0 — the only system achieving both.

HP A P99 tail latency: LithOS is 13x better than MPS, 4x better than Orion, 1.2x better than TGS. Partitioning systems (MIG, thread Limits) sustain SLO but cannot use idle capacity for best-effort work (BE goodput 0.15); LithOS sustains both isolation and BE throughput.

7.3 Hybrid inference/training multitenancy (~80% util target)

LithOS mean P99 is 2.34x over REEF and 1.18x over TGS; it cuts latency by up to 13.54x and 4.7x on average vs native MPS, improves training throughput by an average of 34x and aggregate throughput 1.35x vs TGS, with total aggregate-throughput improvement of 1.23x-1.57x (avg 1.38x). The TGS failure mode is instructive: its adaptive rate control assumes a constant arrival rate, which is invalid under Poisson inference load; REEF throttles best-effort work without accounting for kernel duration, so long-batch BE kernels still blow up the HP tail to 8.93x.

7.4 Hardware right-sizing (Fig. 18)

The savings come from the fact that whole-model right-sizing is provably suboptimal — kernels scale heterogeneously (GEMM and multihead attention show diminishing returns; a token-frequency-penalty kernel does not scale at all), so no single TPC config fits a model. Per-kernel sizing captures the difference.

7.5 Kernel-dependent DVFS (Fig. 19)

7.6 Atomization ablation and case studies

7.7 Predictor accuracy and overheads

8. Configuration-Regime Trade-off Tables

8.1 GPU-sharing substrate

Best choice (paper's data): TPC-level sharing, because it is the only point that simultaneously reclaims idle capacity for best-effort work and holds the high-priority SLO — MPS sacrifices the SLO, MIG sacrifices the reclamation.

8.2 Preemption granularity

Best choice (paper's data): atomize long-running kernels only. The Atomizer divides predicted duration by atom_duration precisely so short kernels are left whole (no overhead) while long kernels are split (bounded HoL). The 10% BE cost buys a high-priority tail within 19% of ideal.

8.3 Right-sizing scope

Best choice (paper's data): per-kernel right-sizing, because kernel scaling is heterogeneous (some kernels do not scale at all); a single model-wide TPC count either overprovisions the serial kernels or starves the parallel ones.

8.4 Energy vs latency (DVFS slip)

Best choice (paper's data): kernel-dependent DVFS with a bounded slip. The 7% latency cost is acceptable for 26% energy when per-GPU power exceeds 1,000 W, and the slip parameter caps the worst case; the only reason it stays conservative is the ~50 ms hardware switching latency.

9. Bottlenecks & Insights Surfaced by the Measurements

Long kernels are the head-of-line enemy. DLRM kernels exceed 30 ms and many training kernels run several milliseconds, while tight inference SLOs demand sub-millisecond responsiveness. Whole-kernel scheduling therefore cannot protect a high-priority tenant; the measurement that the scheduler alone reaches only 1.38x of ideal HP tail, improving to 1.19x once atomization is added, isolates exactly this bottleneck and its fix.

Compute isolation dominates bandwidth isolation by an order of magnitude. Because the workloads are sized to remain DRAM-resident, memory contention contributes only 4-13%, while compute isolation delivers >10x. This justifies LithOS's exclusive focus on TPC (compute) arbitration and tells us where the leverage is on a shared GPU: the SMs, not the memory bus.

No single resource configuration fits a model. Per-kernel right-sizing recovers 26% mean capacity (up to 51%) precisely because GEMM, attention, and penalty kernels scale differently. A static, whole-model TPC allocation leaves this on the table — the heterogeneity is the opportunity.

Adaptive policies that assume stationarity break under bursty load. TGS's constant-arrival-rate assumption inflates inference latency to 1.41x and REEF's duration-blind throttling lets the tail reach 8.93x under unpredictable Poisson load. LithOS's duration-aware, prediction-driven control keeps the tail within 20% of ideal. The insight: a GPU scheduler for inference must model kernel duration, not just request rate.

Prediction is cheap and accurate enough to drive everything. HP misprediction is 0.38-0.9% with a P99 error tail of 31-49 us, and the whole interposition layer adds only 4% overhead (atomization <1%). This validates the architectural bet that one online predictor can feed all four mechanisms without offline profiling.

10. Limitations of the Methodology

The most consequential limitation is the single-GPU scope. Every LithOS mechanism is intra-GPU, so the paper does not speak to how TPC-level arbitration would interact with multi-GPU collective communication, host-side networking, or NVLink/PCIe contention — the regimes where a datacenter actually lives. The second is the reverse-engineering dependency: LithOS works because it reconstructs a driver-private TPC mapping, which the authors argue vendors should simply expose, but which today must be re-derived (in "days") for each new architecture.

11. Analogy

LithOS is an air-traffic control tower retrofitted onto an airport that was built without one. The runways are TPCs; the aircraft are kernels; the airlines are tenants of differing priority. Before LithOS, the airport ran like MPS — everyone taxis whenever they like, so a heavy long-haul jet (a 30 ms DLRM kernel) can sit on the only runway while a priority medevac flight (a sub-millisecond inference request) circles and misses its window. MIG is the opposite overreaction: it paves separate dedicated runways per airline that cannot be shared even when empty, so capacity sits idle. LithOS's tower does three things. It holds aircraft at the gate (launch queues) instead of letting them onto the taxiway, so it can still reorder departures right up to the runway threshold — because once a plane is rolling, it cannot be recalled. It breaks a long-haul taxi into segments (atomization), clearing the medevac between segments rather than after the whole journey. And it right-sizes the runway and throttles the engines (TPC right-sizing and DVFS), giving each flight only the asphalt and thrust it actually needs so fuel (energy) and pavement (capacity) are not wasted. The single online predictor is the tower's radar: one shared track of where every aircraft is and how long each leg will take, feeding every controller's decision at once. The result — 100% of priority flights on time and the spare runway still carrying cargo — is the both-worlds outcome that neither the free-for-all nor the dedicated-runway airport could reach.