Architecture & Measurement-Design Analysis

Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling

Source: Subramanya, S. J.; Arfeen, D.; Lin, S.; Qiao, A.; Jia, Z.; Ganger, G. R. SOSP '23, Koblenz, Germany, October 23-26 2023, pp. 642-657. DOI: 10.1145/3600006.3613175 Affiliations: Carnegie Mellon University + Cornell University + Petuum Inc. Built on: AdaptDL (petuum/adaptdl), Kubernetes, PyTorch, CVXPY + GLPK_MI Reader: pdftotext -layout direct extraction (gemini-reader script pinned to a retired model) Analyst: Vishwakarma Date: 2026-09-01

Table of Contents

1. System Architecture (the scheduler itself)
2. System-Under-Test Architecture (testbed, simulator, baselines)
3. Design-Space Diagram (axes swept, axes held fixed)
4. Algorithm & Control Flow Diagrams
5. Quantitative Results — Empirical Findings by Regime
6. Configuration-Regime Trade-off Tables
7. Bottlenecks & Insights Surfaced by the Measurements
8. Limitations of the Methodology
9. Note on NCCL Tuning
10. Analogy

1. System Architecture (the scheduler itself)

Sia is a pre-emptive, round-based cluster scheduler that maximizes cluster-wide goodput. Its defining claim is that two degrees of freedom prior schedulers treated separately — which GPU type a job runs on (heterogeneity) and how many GPUs plus what batch size it runs with (adaptivity) — must be decided jointly, in one solve, every round. Prior work splits into heterogeneity-aware schedulers (Gavel) serving only rigid jobs, and adaptivity-aware schedulers (Pollux) assuming all GPUs are identical. Sia is the intersection.

1.1 Component graph and job life cycle

+------------------------------------------------------------------+
|                        Sia scheduler                             |
|  (1) submit (max_bsz, max_ngpus)                                 |
|       |             (2) profile 1 GPU of EACH type, ~10          |
|       v                 batchsizes, < 20 GPU-seconds per type    |
|  +--------------+ ------------------------------> +-----------+  |
|  |    Queue     |                                 | Profiler  |  |
|  +------+-------+                                 +-----+-----+  |
|         |                       (3) profiles            |        |
|         | (4) waits until allocated                     v        |
|         |                  +--------------------------------+    |
|         |                  |      Goodput Estimator         |    |
|         |                  | - throughput model per         |    |
|         |                  |   (job, GPU_type)              |    |
|         |  (8) goodput for | - statistical-efficiency model |    |
|         v  all configs     |   per job (borrowed: Pollux)   |    |
|  +--------------------+    +----+-----------------------^---+    |
|  |     Sia Policy     |<--------+                       |        |
|  | build G (|J|x|C|)  |          (7) gradient + throughput       |
|  | row-normalize      |              stats, 30 s default |       |
|  | restart factor r_i |                                  |       |
|  | fairness exponent p|                                  |       |
|  | solve binary ILP   |                                  |       |
|  +---------+----------+                                  |       |
|            | (5) ALLOCATION (n, m, X)                    |       |
|            v                                             |       |
|  +--------------------------------------+                |       |
|  |    Placer (three rules; see Fig 10)  |                |       |
|  +---------+----------------------------+                |       |
|            | (6) PLACEMENT: physical GPUs                |       |
|            v                                             |       |
|  +--------------------------------------+                |       |
|  |         Adaptive Executors           |----------------+       |
|  | transparent ckpt-restore | batchsize |                        |
|  | adaptivity | stats reporting         |                        |
|  +--------------------------------------+                        |
|   continuous loop: (5) -> (6) -> (7) -> (8) -> (5) -> ...        |
+------------------------------------------------------------------+
^ Fig 1: Sia component graph, redrawn from Figure 3 of the paper with
  the paper's own lifecycle numbering. Steps (1)-(4) happen once per
  job; (5)-(8) loop for the rest of the job's life in the cluster.

Three kinds of state sit in three components, and that separation is what makes the round cheap. The Profiler produces static per-GPU-type compute measurements once; the Goodput Estimator holds the slowly refined per-(job, GPU-type) throughput model; the Policy holds no state that survives a round. Learning is amortized across rounds while per-round work is one integer program.

1.2 Implementation stack

+---------------------------------------------------+
| DL job (ResNet18/BERT/DeepSpeech2/YOLOv3/         | workload
|         ResNet50/2.8B GPT)                        |
+---------------------------------------------------+
| AdaptDL data-loaders (variable batch size); LR    | adaptivity
| scaling: square-root (AdamW), AdaScale (SGD)      |
+---------------------------------------------------+
| Sia Adaptive Executor: minibatch + gradient stat  | agent
| profiling; goodput-model fitting every 30 s;      |
| batch-size selection; ckpt-restore on realloc     |
+---------------------------------------------------+
| PyTorch + gradient all-reduce                     | training
+---------------------------------------------------+
| Sia Policy (Kubernetes service)                   | control
|  Eq. (4) as a MILP, solved by GLPK_MI via CVXPY   |   plane
+---------------------------------------------------+
| Kubernetes-managed heterogeneous GPU cluster      | substrate
| (t4 / rtx / a100 / quad node types)               |
+---------------------------------------------------+
^ Fig 2: Implementation stack. Sia is a drop-in scheduler and
  data-loader replacement inside AdaptDL, which is what makes a
  head-to-head comparison with the public Pollux implementation
  possible on identical framework machinery.

Because Sia and Pollux share the executor, data-loader, framework and checkpoint-restore path, the difference between them is attributable to the policy, not to engineering quality below it. Gavel, Themis and Shockwave lack that property — they are compared only in simulation.

1.3 Decoupled allocation and placement

Sia splits the problem into an allocation stage (number and type of resources per job) and a placement stage (which physical GPUs satisfy it), then restricts the allocation space so a valid placement is guaranteed for anything the solver can emit. Pollux instead considers every way to place each (job, GPU_count) pair across all nodes, giving a solution count exponential in nodes and GPUs-per-node.

Most of Sia's measured advantages follow from this one decision. Shrinking the set lets the ILP solve in under a second at 2048 GPUs; guaranteed feasibility lets the Placer be three rules rather than a second optimizer; and forbidding two distributed jobs from sharing a node eliminates NIC contention, a slowdown source cited from prior work. The stated cost is optimality, and the homogeneous-cluster result (Table 4) is the evidence that the restriction does not materially hurt job runtimes.

2. System-Under-Test Architecture (testbed, simulator, baselines)

2.1 Node types and cluster settings

+------------- Heterogeneous setting (64 GPUs, simulated) ---------+
|  6x t4 node         3x rtx node          2x a100 node            |
|  +------------+     +--------------+     +-----------------+     |
|  | 4x T4 16GB |     | 8x 2080Ti    |     | 8x A100 40GB    |     |
|  | cloud VM   |     | 11GB         |     | DGX-A100        |     |
|  |            |     | 50 Gb/s Eth  |     | 1.6 Tb/s IB     |     |
|  +------------+     +--------------+     +-----------------+     |
|     x6 = 24            x3 = 24              x2 = 16              |
|  interconnect spread 50 -> 1600 Gb/s (32x); VRAM 11 -> 40 GB     |
+------------------------------------------------------------------+
+--- Physical setting (44 GPUs, real): 3x rtx + 2x a100 + 1x quad -+
+--- Homogeneous setting (64 GPUs, simulated): 16x t4 node --------+
^ Fig 3: The three cluster settings. The heterogeneous cluster mixes a
  50 Gb/s Ethernet node type with a 1.6 Tb/s InfiniBand node type, so
  "which GPU type" is not reducible to "how fast is the GPU".

The interconnect spread matters more than the compute spread. Because scaling depends on a job's compute-to-network ratio, a node type can be the best choice at 1 GPU and the worst at 8: the paper states that GPU types can have distinct compute-to-network bandwidth ratios, so a job may scale better on one type even when single-GPU throughputs are ordered the other way.

2.2 Evaluation harness

+------------------------------------------------------------------+
|  Trace samplers                 Model catalogue (Table 2)        |
|   Philly   10 x 8h              S  ResNet18/CIFAR-10             |
|   Helios   10 x 8h  ---------> M  BERT/SQuAD, DeepSpeech2/ARCTIC|
|   newTrace 10 x 48h             L  YOLOv3/PASCAL-VOC             |
|   20 jobs/hr average            XL ResNet50/ImageNet-1k          |
|                                 XXL 2.8B GPT/SQuAD               |
|  +---------------------------------------v------------------+    |
|  | Scheduler under test                                     |    |
|  |  Sia | Pollux | Gavel+TJ | Shockwave+TJ | Themis+TJ       |    |
|  +--------------------------+-------------------------------+    |
|            +----------------+----------------+                   |
|            v                                 v                   |
|  +-------------------+          +--------------------------+     |
|  | PHYSICAL          |          | SIMULATOR                |     |
|  | 44 GPUs, 3 types  |          | Pollux discrete-time sim |     |
|  | 30 jobs, 3-h trace|          |  + Gavel, + Shockwave,   |     |
|  | 4 repeat runs     |          |  + heterogeneous Pollux, |     |
|  |                   |          |  + model-specific ckpt-  |     |
|  |                   |          |    restore delays        |     |
|  |                   |          | 200 repeat runs          |     |
|  +---------+---------+          +------------+-------------+     |
|            +----------- cross-check ---------+                   |
|             < 5% error in avg JCT and makespan for Sia and       |
|             Gavel; Pollux does NOT cross-validate                |
|                             v                                    |
|  Metrics: avg JCT, p99 JCT, makespan, GPU-hours/job, contention  |
|  (avg/max), restarts/job, FTF rho, unfair job fraction, policy   |
|  runtime (median / p95)                                          |
+------------------------------------------------------------------+
^ Fig 4: Evaluation harness. The physical cluster is used once, at
  44 GPUs, to validate the simulator; all breadth (cluster size,
  arrival rate, parameter sweeps) comes from simulation.

2.3 Trace provenance and baseline conditioning

Jobs are binned by total GPU time — Small (0-1 h), Medium (1-10 h), Large (10-100 h), Extra-large (>100 h) — and each bin maps onto representative models. XXL (2.8B GPT) is hybrid-parallel only.

Defaults: p = -0.5, lambda = 1.1 for Sia; p = -1 for Pollux and (10, 1e-1) for Shockwave, both as in their original papers. Round duration 60 s for Sia and Pollux, 360 s for Gavel, Themis and Shockwave. Gavel runs max-sum-throughput, chosen because it gives Gavel the lowest average JCT on Philly among its published policies.

3. Design-Space Diagram (axes swept, axes held fixed)

+------------------------------------------------------------------+
|                       SIA DESIGN SPACE                           |
|  SWEPT AXES                                                      |
|  A1  Scheduler policy (5): Sia, Pollux, Gavel+TJ,                |
|      Shockwave+TJ, Themis+TJ                                     |
|  A2  Cluster composition (3): Physical 44 (3 types),             |
|      Homogeneous 64 t4, Heterogeneous 64 (3 types)               |
|  A3  Trace (3): Philly 8h, Helios 8h, newTrace 48h               |
|  A4  Cluster size up to 2048 GPUs (traces scaled proportionally) |
|  A5  Job arrival rate (Helios, 64 GPUs)                          |
|  A6  Adaptivity class (3), also swept as a population fraction:  |
|      Adaptive (bsz + count + type), Strong-scaling (fixed bsz),  |
|      Rigid (fixed bsz + count; type free)                        |
|  A7  Profiling regime (3): Oracle, No Prof, Bootstrap            |
|  A8  Fairness exponent p in [-1.0, 1.0], default -0.5            |
|  A9  Scheduling round duration {30 s, 60 s, 300 s}               |
|  A10 Parallelism form (2): pure data parallel; pipeline-MP +     |
|      data parallel (2.8B GPT)                                    |
|                                                                  |
|  HELD FIXED                                                      |
|   - lambda = 1.1; Executor reporting interval 30 s               |
|   - Scale-up: start at 1 GPU, at most 2x growth per round        |
|   - LR scaling (square-root for AdamW, AdaScale for SGD)         |
|   - Configuration-set restriction and the three placement rules; |
|     node-exclusive placement for distributed jobs                |
|   - Data parallelism with gradient all-reduce as scale-out       |
|   - Solver: GLPK_MI via CVXPY                                    |
|   - Non-data-parallel degrees of a hybrid job (pipeline depth,   |
|     tensor-parallel width) are NOT co-optimized                  |
|   - Interconnect / collective configuration: never a decision    |
|     variable; communication time is only ever OBSERVED           |
+------------------------------------------------------------------+
^ Fig 5: Ten swept axes and the fixed frame around them. Sia's own
  action space is exactly three of these (GPU type, GPU count, batch
  size); the rest are experimental conditions.

The last fixed item is the most consequential. Sia treats communication time as something to measure and predict, never to change: a configuration is a bundle of resources, not resources plus a communication strategy. Every result is a statement about resource assignment under a fixed communication regime.

4. Algorithm & Control Flow Diagrams

4.1 The configuration set

A configuration is a 3-tuple (n, r, t): n nodes holding r resources of type t in total. For N physical nodes with R GPUs of type X each:

   SINGLE-NODE set                    MULTI-NODE set
   (n = 1, powers of 2, <= R)         (whole nodes only)
   +----------------------+           +----------------------+
   | (1, 2^0, X)          |           | (2, 2R, X)           |
   | (1, 2^1, X)          |     U     | (3, 3R, X)           |
   | ...                  |           | ...                  |
   | (1, R,   X)          |           | (N, N*R, X)          |
   +----------------------+           +----------------------+
        log2(R) entries                     N entries

   |C| = N + log2(R)        vs.   Pollux search space O(N^R)

   If R is not a power of 2: decompose R into a sum of powers of 2
   and model one physical node as several virtual nodes.

   Worked example (paper): 1 node of 2 type-A GPUs + 1 node of 4
   type-B GPUs -> C = {(1,1,A),(1,2,A),(1,1,B),(1,2,B),(1,4,B)}
^ Fig 6: Configuration-set construction. Within a node, allocations
  are powers of 2; across nodes, allocations consume whole nodes.

The Submesh Shape Covering theorem guarantees every valid allocation from this set admits a placement in which no two distributed jobs share a node — which is why the Placer never searches.

4.2 The bootstrapping ladder

The naive approach profiles many multi-GPU allocations per GPU type, at a cost growing linearly in both the number of GPU types and the nodes of each type. Sia climbs a ladder instead, replacing crude estimates with measured ones as the job runs.

  (1) PROFILE J on ONE GPU of each type: min batchsize upward until
      the GPU memory limit (~10 batchsizes), < 20 GPU-s per type
      KEY: compute time is INDEPENDENT of GPU COUNT, because scaling
      is data-parallel with all-reduce -- so only COMMUNICATION time
      remains unknown
             |
  (2) INITIALIZE throughput models from the 1-GPU profiles
             |
  (3) Sia places J on 1 GPU of type A; online profiling learns a
      statistical-efficiency model vs batch size and a refined 1-GPU
      throughput model for A
             |
  (4) To estimate 2 GPUs of A: ONE-TIME SIMPLIFYING ASSUMPTION --
      two data-parallel replicas give twice the throughput of one
      (perfect scaling, zero communication time)
             |
  (5) Sia assigns 2 GPUs of A -> measured communication times ->
      refine xput_A to cover compute AND communication
             |
  (6) Type B has never run multi-GPU. BOOTSTRAP it:
         est-xput_B(N) = ( xput_B(1) / xput_A(1) ) * xput_A(N)
      Assumption: B's compute:communication scaling matches A's
             |
  (7) If J ever runs multi-GPU on B -> refine xput_B from measured
      communication times -> DISCARD the bootstrapped model
^ Fig 7: The bootstrapping ladder. Each rung replaces an assumption
  with a measurement; the assumptions exist only to make the NEXT
  exploratory allocation reasonable, not to be permanently trusted.

The load-bearing observation is at step (1). Because scale-out is pure data parallelism, per-GPU compute time does not change with GPU count, so the unknown collapses from a surface (count x batchsize) to a curve (communication vs count) — which is what makes one measured multi-GPU point enough to calibrate a whole type.

4.3 Goodput matrix to allocation

  per-(job, GPU_type) throughput models
  + per-job statistical-efficiency models
                  v
     Goodput estimators  (f_A, f_B) for J1 ; (g_A, g_B) for J2
                  v
   +---------------------------------------------+
   | Goodput matrix G, size |J| x |C|            |
   | G_ij = goodput of job i in config j         |
   | ROWS comparable; COLUMNS ARE NOT            |
   +----------------------+----------------------+
                  | row-normalize: G_ij <- N_i^min * G_ij / min_j G_ij
                  |   -> scale-free utility matrix, columns comparable
                  | restart discount: for J_i running c_k,
                  |   G_ij <- r_i * G_ij for all c_j != c_k
                  | fairness exponent: entries raised to power p
                  |   (default -0.5; if p < 0 flip the objective sign)
                  v
   +---------------------------------------------+
   | Binary ILP over A (same shape as G):        |
   |   max_A  sum_ij A_ij (r_i G_ij)^p           |
   |          + lambda (1 - ||A_i||_1)           |
   | s.t. ||A_i||_1 <= 1  (<=1 config per job)   |
   |      allocated GPUs <= available, per type  |
   +----------------------+----------------------+
                  v
   A_i all-zero -> no resources this round; A_ij = 1 -> config c_j
^ Fig 8: Data flow from throughput models to a binary allocation
  matrix. Row normalization is the step that converts "which config
  is best for this job" into "which job deserves this config".

Normalizing by the row minimum, scaled by the job's minimum required GPU count N_i^min, gives two properties the paper calls out. G reads as a utility matrix — the largest entry along a row is the best configuration for that job — and entries become comparable down a column, so a configuration can go to the job that values it most. Unnormalized, G_mj > G_nj does not imply job m deserves j.

Worked example (Table 1), 1 node of 2 type-A + 1 node of 4 type-B:

Maximizing the sum selects (1,4,B) for J1 and (1,2,A) for J2.

4.4 One scheduling round, end to end

  ROUND BOUNDARY (every 60 s by default)
    |
  (1) Collect latest goodput-model parameters from every Adaptive
      Executor (each refits every 30 s and pushes to Policy), add a
      row to G per new job, delete rows for completed jobs
    |
  (3) Recompute restart factors r_i = (T_i - N_i*S_i) / (T_i + S_i)
      T_i = job age, S_i = GPU-seconds per restart, N_i = prior
      restart count
    |
  (4) Apply constraints for special job classes: non-preemptive jobs
      and reservations force requested resources; rigid jobs use the
      separate objective Eq. (5) over (job, GPU type) only;
      strong-scaling jobs substitute throughput for goodput
    |
  (5) SOLVE the MILP with GLPK_MI (median 96 ms, p95 426 ms at 64 GPU)
    |
  (6) Diff new allocation A against current assignment
    |
    +-- unchanged --> job continues, no restart
    |
    +-- changed  --> (7) PLACER assigns physical GPUs, minimizing
                         unnecessary migrations
                     (8) PREEMPT after the current minibatch (no comm
                         in flight): checkpoint model weights, data-
                         loader state and optimizer state to shared
                         storage; release GPUs
                     (9) RESTORE: one Adaptive Executor per GPU on the
                         new resources, restore, resume
    |
  NEXT ROUND
^ Fig 9: One scheduling round. The restart factor at step (3) is what
  keeps step (8) from firing on every tiny change in G.

4.5 Placement rules and job states

 Allocation (n, m, X) at Placer      Job state machine
   |                                  [SUBMITTED]
   v                                       | submit(max_bsz,
 partial-node (m < R)?                     |        max_ngpus)
   |                                       v
   +- yes -> must fit in ONE node    [PROFILING] 1 GPU/type
   |          free node? -> PLACE          v
   +- no  -> take n WHOLE nodes       [QUEUED] <-------+
              free nodes? -> PLACE         | A_ij = 1  |
                    v                      v           |
        RESOURCE FRAGMENTATION         [RUNNING] ------+ alloc
                    |                      |            unchanged
                    v                      | alloc changed
        EVICT some jobs and retry          v
        (rare; usually fewer than   [CHECKPOINTING] after current
         3 evictions at once)              |         minibatch only
                                           v
                                     [RESTORING] --> [RUNNING]
                                           | target metric met
                                           v
                                      [COMPLETE]
   also: epoch-boundary checkpoints allow resuming on different
   resources after worker failure
^ Fig 10: Placement decision flow (left) and job state machine
  (right). There is no search in the Placer -- the configuration-set
  restriction already did that work. The RUNNING -> CHECKPOINTING
  edge is what the restart factor makes expensive, since restart
  costs are 25-250 seconds for the Table 2 models.

5. Quantitative Results — Empirical Findings by Regime

5.1 Physical cluster, 44 GPUs, 3 GPU types

Setting: 3 rtx + 1 quad + 2 a100 nodes; a single 3-hour trace of 30 jobs mixing all Table 2 models; each scheduler run four times.

The Pollux discrepancy is attributed to three causes: the simulator modifications gave Pollux an advantage on single jobs over heterogeneous resources; its schedules vary widely from randomness in the genetic optimization and misguided adaptivity from noisy throughput estimators; and real heterogeneity is mapped onto virtual nodes Pollux assumes homogeneous.

Behavioural trace over 45 minutes (Figure 5 of the paper): rising congestion makes Sia scale down and move the ImageNet job to rtx, freeing a100 for incoming CIFAR-10 jobs; when congestion falls, Sia shifts ImageNet back to a100 and scales DeepSpeech2 out on rtx.

5.2 Heterogeneous simulation, 64 GPUs (Table 3)

Headline summary as stated in Section 5.2: Sia reduces average JCT by 30-93% and p99 JCT by 28-95% versus Pollux and Gavel, and allocates 12-60% fewer GPU hours per job.

The variance column is as informative as the mean — Sia's makespan error bars are ±1.3 to ±1.9 hours against Pollux's ±4.6 to ±8.0 and Gavel's ±8.6 to ±21.5 — and Pollux restarts jobs roughly twice as often as Sia on the moderately congested traces, attributed to Pollux optimizing in steps of 1 GPU while Sia moves in whole-node steps.

newTrace separates the schedulers most sharply, with bursts of up to 100 jobs/hr in the busiest hour. The paper describes a positive feedback loop for Gavel: rising contention forces more frequent swap-in/swap-out, burning GPU capacity on checkpoint-restore exactly when GPUs are scarce — reaching contention average 96 and maximum 243 against Sia's 13 and 69. At peak load Sia and Pollux both scale jobs to 1 GPU each, narrowing their gap, but Sia's type matching still improves cluster goodput.

5.3 Homogeneous simulation, 64 t4 GPUs, Philly traces (Table 4)

Restarts per job: Sia 2.6, Pollux 5.1.

This is the control condition — heterogeneity removed, so only adaptivity is in play, and Pollux was designed for exactly this. Sia matches and slightly exceeds Pollux on every metric, attributed to the ILP guaranteeing a global optimum where Pollux's genetic algorithm does not, plus fewer wasted GPU hours on checkpoint-restore. Both show a 50-70% improvement over the inelastic baselines on all metrics; among those, Shockwave is best because its objective targets job progress and finish-time fairness while penalizing large makespan.

5.4 Rigid-job regime (heterogeneity without adaptivity)

With auto-scaling and batch-size tuning disabled for both Sia and Pollux — every job pinned to the trace's batch size and GPU count — Sia still outperforms Gavel by about 25%, in the regime Gavel was designed for. The cause is the objective, not the mechanism: Sia maximizes summed goodput while Gavel maximizes cluster throughput, so with inelastic jobs Sia always delivers higher per-GPU goodput. Pollux also optimizes summed goodput but produces worse JCTs because it cannot see the heterogeneity.

Figure 1 frames the argument in three panels: Pollux and Sia win on adaptive jobs on a homogeneous cluster; Gavel and Sia win on rigid jobs on a heterogeneous cluster; in the middle case Sia beats both, with the middle trio of bars showing 40-70% lower average JCTs.

5.5 Matching jobs to GPU types (Figure 6, Helios traces)

5.6 Workload intensity sweep (Figure 7, Helios, 64 GPUs)

5.7 Hybrid-parallel jobs (Section 5.3)

Sia is presented as the first cluster scheduler able to elastically scale hybrid-parallel jobs.

Scaling rule: a job with N data-parallel replicas over a P-GPU pipeline uses exactly N x P GPUs; with mini-batch M and micro-batch m, each replica computes M / (m*N) micro-batches across its P GPUs, then the N replicas synchronize by gradient all-reduce. The objective gains terms and constraints keeping allocations in units of P.

5.8 Finish-time fairness (Figure 8, Helios, heterogeneous)

Finish-time fairness ratio rho is a job's completion time in the shared cluster over its completion time in an isolated, fair-sized cluster of N_gpus / N_avg GPUs. Sia extends it as rho = sum_G P(G = g) * rho_g, the expectation of the per-GPU-type ratio weighted by the share of cluster GPUs of type g; with one GPU type this reduces to the original definition.

The abstract states the same result as 64% lower worst-case finish-time fairness and 99% lower unfair job fraction. Shockwave beats Gavel and Pollux on unfair fraction because it penalizes jobs with high FTF ratios, trading worst-case ratio for fraction. Sia's advantage is behavioural: Gavel and Shockwave prioritize either makespan or long jobs, hurting short jobs during congestion, whereas Sia scales long jobs down during congestion to admit incoming short jobs and scales them back out afterwards.

5.9 Policy overhead and scalability (Figure 9)

64-GPU heterogeneous setting, Helios traces:

Scaled to 2048 GPUs with proportionally sized Helios traces, Sia stays at a single-second runtime. Pollux's genetic algorithm runs roughly 100x slower and struggles to find optimal solutions at large cluster sizes from search-space explosion — without the extra complexity heterogeneity would add — and Section 2.1 adds that at 1000+ GPUs it takes tens of minutes. Gavel is faster than both because it does not consider job adaptation at all.

5.10 Parameter sensitivity (Figure 10)

No significant change in p99 JCT or makespan was observed across round durations, and policy optimization takes under a second even for moderate clusters, so 60 s is not a solver-imposed floor.

5.11 Adaptivity and profiling ablations (Figures 11, Helios/Philly)

Two supporting results: profiling two GPU counts per GPU type performed worse than Sia's minimized approach, and at 20 GPU types bootstrapping adds under 5% overhead to a job's execution.

6. Configuration-Regime Trade-off Tables

6.1 Scheduler posture toward the two degrees of freedom

Sia's position: the two degrees of freedom are not additive. Extending Gavel's (job_id, GPU_type) throughput matrix to (job_id, GPU_type, num_GPUs, minibatch_size) breaks twice — filling a non-trivial portion needs extensive per-job profiling, and the program becomes too large to solve quickly. Sia answers both: bootstrapping for the first, configuration-set restriction for the second.

6.2 Search-space formulation and configuration granularity

Sia's position: shrink the space until an exact solver fits rather than keeping it large and accepting a heuristic. Fine granularity is a liability once restarts cost 25-250 seconds — coarse steps change the solver's output less often, so more allocated GPU-hours are spent training.

6.3 Throughput-model acquisition

Sia's position: profile only what cannot be inferred — the single-GPU point per type — and let the allocation policy supply the rest as a side effect of running the job. Profiling two GPU counts per type was tested and performed worse, so the minimum is also the best.

6.4 Fairness mechanism

Sia's position: fairness is nearly free because the row-normalized matrix is already scale-free — a power reshapes how sharply the objective favours high-utility matches without a second objective. Elasticity is the primary congestion valve and type matching the secondary one, which is why the Sia-Pollux gap narrows but does not close at peak load.

7. Bottlenecks & Insights Surfaced by the Measurements

The search space, not the model, was the blocking problem. Pollux's per-round solve takes 2.2 s median at 64 GPUs and tens of minutes at 1000+ GPUs, before heterogeneity multiplies the space, and Gavel's fast LP could not extend to adaptivity because the extended throughput matrix becomes both unfillable and unsolvable. Sia's contribution is less a better model of job behaviour than a better-shaped decision variable.

Compute time is GPU-count-invariant, and that is the whole profiling trick. Because scale-out is pure data parallelism with all-reduce, per-GPU compute time depends on batch size and GPU type but not count. The unknown collapses from a surface to a curve, letting one measured multi-GPU point per type complete a model and letting Equation (1) transfer that curve to an unmeasured type by a single ratio.

Restart cost is the hidden budget every adaptive scheduler spends. Checkpoint-restore costs 25-250 seconds for the Table 2 models, and Pollux restarts about twice as often as Sia, tied to its 1-GPU allocation granularity. Gavel's newTrace collapse is the same mechanism at an extreme: time-sharing under contention pays restart costs when the cluster can least afford them, driving contention to an average of 96.

Elasticity in GPU count carries more value than batch-size co-adaptation. Adding GPU-count optimization on top of GPU-type optimization improves average JCT by 56%; adding batch-size optimization adds a further 13%. Since batch-size adaptation carries the model-quality risk — the paper acknowledges a generalization gap at large batch sizes — a cluster forbidding batch-size changes still captures most of the benefit.

Specialization has a small cost. Sia sends BERT almost exclusively to a100 and DeepSpeech2 to rtx, so YOLOv3 and DeepSpeech2 consume roughly 5% more GPU hours than under heterogeneity-unaware Pollux — a per-job evaluation on DeepSpeech2 alone would report a loss.

Fairness improved as a by-product of goodput optimization. Sia's worst FTF ratio of 1.2 and unfair fraction below 0.3% beat Shockwave's 3.3 and 14%, despite Shockwave being purpose-built for fairness. Gavel and Shockwave prioritize makespan or long jobs and hurt short jobs during congestion, while Sia's congestion response happens to be the short-job-friendly policy — fairness emergent from elasticity rather than legislated.

The simulator validated for two of three schedulers. Under 5% error for Sia and Gavel, but Pollux performed significantly worse in reality than in simulation. Since Pollux is the strongest simulated baseline, the asymmetry cuts in Sia's favour.

8. Limitations of the Methodology

Breadth comes from simulation. Only one experiment runs on real hardware: 44 GPUs, a single 3-hour trace of 30 jobs, four repeats, three schedulers. Every claim about cluster size, arrival rate, parameter sensitivity, fairness, hybrid parallelism and profiling regime is simulated. The paper explains why — dedicated time on the on-premise nodes required blocking out all other users — and cross-validates the simulator, but that validation covers Sia and Gavel and explicitly fails for Pollux.

Baselines are hand-tuned, and Pollux runs outside its envelope. Gavel, Themis and Shockwave use TunedJobs, in which the authors chose each job's batch size and GPU count by searching for a configuration whose runtime lands at 50-80% of ideal speedup — a heuristic band, not an optimum, so neither a floor nor a strict upper bound on those schedulers. Making Pollux run on heterogeneous clusters required virtual 4-GPU nodes plus a majority-type heuristic that the paper itself calls imperfect; those numbers are "Pollux plus an adapter".

Reported ranges are inconsistent across the paper's own sections. The abstract gives 12-55% for GPU-hours reduction while the introduction and Section 5.2 give 12-60%; the abstract folds makespan into the 28-95% p99 range while the introduction gives it separately as 38-65%. A reader quoting a single range should say which section it came from.

Statistical efficiency modelling is inherited, not evaluated. The Goodput Estimator combines Sia's throughput model with a Gradient Noise Scale-based statistical efficiency model borrowed from Pollux, never validated here; for the 2.8B GPT experiment the profile is borrowed from BERT. Model-quality outcomes — accuracy, F1, mAP — are not reported anywhere; jobs are simulated as running to a target metric rather than measured as reaching it.

Hybrid-parallel support is partial by construction. Sia scales such jobs only by replicating a fixed pipeline configuration data-parallel-wise; pipeline depth and tensor-parallel width are fixed at submission, and the paper defers elastic scaling without fixing those degrees to future work because existing hybrid-parallel optimizers are time-consuming. The GPT result is also simulated.

The bootstrap transfer assumption is validated only in aggregate. Equation (1) assumes an unmeasured GPU type's compute-to-communication scaling ratio matches a measured one's, over a testbed spanning 50 Gb/s Ethernet to 1.6 Tb/s InfiniBand. The defence is end-to-end — Bootstrap lands within 8% of Oracle — not a direct measurement of Equation (1)'s prediction error; the model is discarded as soon as a real multi-GPU measurement arrives, limiting how long error persists.

No memory-capacity or failure experiments, and a single solver. Gradient accumulation (for batch sizes exceeding GPU memory) and checkpoint-based recovery from worker failures are described but never measured. All results use GLPK_MI through CVXPY on one ILP formulation, with no comparison against other solvers or a well-tuned greedy heuristic over the same restricted set, so reported runtimes characterize this implementation, not the formulation's difficulty.

9. Note on NCCL Tuning

Sia's throughput model rests on a factorization that is directly useful when reasoning about collective-library configuration: because scale-out is data-parallel, per-GPU compute time is invariant to GPU count, so the only quantity that must be learned as a function of scale is communication time. That reduction is what makes a single measured multi-GPU point per hardware type sufficient, and Equation (1) then transfers the measured scaling curve to an unmeasured type by one ratio of single-GPU throughputs — an explicit bet that the compute-to-communication balance behaves similarly across hardware, made on a testbed spanning a 32x interconnect range — the same bet made whenever a configuration learned on one interconnect is reused on another. Sia also removes cross-job NIC interference by construction rather than modelling it, so its communication measurements are clean by design, a property any measurement-driven tuner must reproduce deliberately for its observations to mean much.

10. Analogy

Sia is a hospital operating-theatre scheduler for a hospital with both a brand-new robotic surgical suite and several older conventional theatres, where every operation can also be performed by a variable-sized surgical team.

The theatres are GPU types and the team sizes are GPU counts. Gavel knows which theatre suits which operation but insists the surgeon name the team size at booking; Pollux sizes teams beautifully but believes all theatres are identical. Figure 1's outer panels show each winning in its own hospital. The middle panel is the real hospital, where a cardiac case that runs three times faster in the robotic suite is sent to a conventional theatre by one scheduler and given a six-person team by the other.

The profiling story maps precisely. You cannot rehearse every operation in every theatre with every team size — the Oracle regime, 1 to 10 GPU-hours per job. So Sia times a single surgeon performing the operation in each theatre, under 20 seconds of theatre time apiece. Because the surgical work per surgeon does not change when you add surgeons — only the coordination overhead does — one timed two-surgeon run in one theatre reveals the whole coordination curve, which then transfers to the other theatres by the ratio of their single-surgeon times. That is Equation (1), discarded the moment a real multi-surgeon run happens there.

The configuration set is the hospital's booking rule: within a theatre you may book 1, 2, 4 or 8 surgeons; across theatres you must book whole theatres. It looks arbitrarily coarse until you notice what it buys — every bookable request is guaranteed a physical arrangement, so the booking desk needs three rules instead of a second optimizer, and no two multi-theatre operations share a corridor, which is the elimination of NIC contention. That Sia matches Pollux in the single-theatre hospital while offering fewer booking options is the evidence surgeons were not using the finer options well.

The restart factor is the rule that you do not move a patient mid-procedure for a marginal improvement, because scrubbing out and back in costs 25 to 250 seconds. Pollux, booking in single-surgeon increments, reschedules and pays that cost twice as often. Gavel under a surge rotates every patient through every theatre in turn, so at the busiest hour the corridors hold 243 waiting cases against Sia's 69. The fairness result falls out of the same behaviour rather than a separate rule — when the hospital fills up, Sia shrinks the long operations to make room for the short ones and grows them again when the pressure lifts, which is exactly the policy that keeps a twenty-minute procedure from waiting all day behind a twelve-hour one.