Architecture & Design Analysis

Synergy: Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters

Source: Mohan, J.; Phanishayee, A.; Kulkarni, J.; Chidambaram, V. 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI '22), July 11-13, 2022, Carlsbad, CA. Pages 579-596. ISBN 978-1-939133-28-1. Affiliations: Microsoft Research (Mohan, Phanishayee, Kulkarni); The University of Texas at Austin and VMware Research (Chidambaram). Mohan's work done as an MSR intern in Project Fiddle. Code: https://github.com/msr-fiddle/synergy Extended version: referenced by the paper as [38] — carries the full Synergy-OPT formulation, the fragmentation proof, and the heterogeneous- cluster extension. Reader: Direct PDF text extraction (pdftotext -layout), cross-checked page-by-page against the USENIX proceedings pagination (579-592 body, 592-596 references). Analyst: Vishwakarma Date: 2026-09-01

Conventions. Fig N with a ^ caption is a diagram drawn in this document; Figure N / Table N refers to the Synergy paper's own numbering. Paper Figures 2a, 3, 5, 6c, 7-13 are plots whose numeric axis values are not recoverable from the PDF text layer — this document reports only what the caption or body text states, and marks such diagrams shapes-only. No axis value has been invented. Anything that is my own reading rather than a paper claim is marked [derived inference].

Table of Contents

1. System Architecture (profiler + policy + mechanism + iterator)
2. System-Under-Test Architecture (physical cluster, simulator, traces)
3. Design-Space Diagram (axes swept, axes held fixed)
4. Algorithm & Control-Flow Diagrams (key procedures)
5. Quantitative Results — Empirical Findings by Regime
6. Configuration-Regime Trade-off Tables
7. Bottlenecks & Insights Surfaced by the Measurements
8. Limitations of the Methodology
9. Analogy

1. System Architecture

Synergy is a round-based, resource-sensitive scheduler for homogeneous multi-tenant GPU clusters. Its structural claim is a separation that prior DNN schedulers collapsed: the GPU demand of a job is fixed and user-supplied, while CPU and memory demands are fungible and should be decided by the scheduler. Everything in the architecture follows from that one split. The paper states it plainly: "Note that Synergy only alters the auxiliary resource allocations; GPU demands are left unaltered for the lifetime of a job and are provided as inputs by the user."

The system decomposes into four parts, each owning one question: the optimistic profiler (what does this job want? — runs once per job lifetime, on arrival); the scheduling policy (which jobs run this round? — unmodified; FIFO, SRTF, LAS, FTF all plug in); the scheduling mechanism (where do they go, and with how much CPU/memory? — Synergy-GREEDY / TUNE / OPT); and the Synergy data iterator (how does the decision reach the job? — a thin wrapper around the PyTorch and DALI iterators, over gRPC).

+---------------------------------------------------------------------+
|                        SYNERGY SCHEDULER                            |
|                                                                     |
|  job arrival                                                        |
|      v                                                              |
|  +-------------------------------+                                  |
|  | Optimistic Profiler (3.1)     |   once per job lifetime          |
|  |  empirical CPU sweep          |   (binary search, ~8 points)     |
|  |  + analytic memory model      |   (MinIO fixed hit rate)         |
|  +---------------+---------------+                                  |
|                  | resource sensitivity matrix W_j[c, m]            |
|  +---------------v---------------+                                  |
|  | Priority Job Queue            |   sorted by the policy metric    |
|  |  (FIFO / SRTF / LAS / FTF)    |   e.g. SRTF -> remaining time    |
|  +---------------+---------------+                                  |
|                  | runnable set J_t  (sum of GPU demand <= cluster) |
|  +---------------v-------------------------------------------+      |
|  | Scheduling MECHANISM  (one of three)                       |      |
|  |   +----------------+ +---------------+ +----------------+  |      |
|  |   | GREEDY (3.3)   | | TUNE (4.2)    | | OPT (4.1)      |  |      |
|  |   | first-fit,     | | sort + best-  | | 2 x LP/ILP,    |  |      |
|  |   | skip on miss   | | fit + revert  | | upper bound    |  |      |
|  |   | -> fragments   | | ladder        | | only           |  |      |
|  |   +----------------+ +---------------+ +----------------+  |      |
|  +---------------------------+-------------------------------+      |
|                              | placement + (g_j, c*_j, m*_j)        |
|  +---------------------------v---+                                  |
|  | Event queue (arrival /        |   deploy event fires here        |
|  |  schedule / deploy)           |                                  |
|  +---------------+---------------+                                  |
+------------------|--------------------------------------------------+
                   | gRPC: allocation, lease grant / lease terminate
                   v
+---------------------------------------------------------------------+
|                        CLUSTER (homogeneous)                        |
|   +------------------------+   +------------------------+           |
|   | Server 1               |   | Server s        ...    |           |
|   |  G_i GPU / C_i CPU     |   |  G_i GPU / C_i CPU     |           |
|   |  M_i memory            |   |  M_i memory            |           |
|   |  [DNN job + Synergy    |   |  [DNN job + Synergy    |           |
|   |   iterator (wraps      |   |   iterator (wraps      |           |
|   |   PyTorch / DALI)]     |   |   PyTorch / DALI)]     |           |
|   |  MinIO cache           |   |  MinIO cache           |           |
|   +------------------------+   +------------------------+           |
+---------------------------------------------------------------------+
^ Fig 1: Synergy system architecture. The profiler runs once per job;
  the policy is untouched; the mechanism is the replaceable component
  that turns a runnable set into a multi-dimensional placement.

Synergy is policy-preserving by construction: it proposes a new mechanism underneath existing policies, not a new policy. "Synergy is not constrained to one particular scheduling policy, but is instead general enough to improve a wide range of scheduling policies (e.g., LAS, FIFO, SRTF, FTF, etc), creating Synergy-augmented variants for all of them." The consequence is that every evaluation number is a delta against the same policy under GPU-proportional allocation — which makes the comparisons unusually clean.

1.1 The iterator as the only application-side change

The scheduler cannot enforce CPU/memory allocations without a hook inside the job; Synergy's hook is deliberately thin.

   DNN job script    |   Synergy iterator        |   Scheduler
  -------------------+---------------------------+-------------------
   (1) construct     |                           |
       iterator ---->| register job -- gRPC ---->| enqueue after
                     |                           | profiling
   (2) for batch in  | fetch + preprocess        |
       iterator: --->| (CPU cores as allocated)  |
   (3) multi-GPU:    | sync across GPU procs so  |
       ranks step    | each makes identical      |
       together      | progress                  |
   (4) round ends    |<---- lease update? -------| grant OR terminate
                     | if terminated: checkpoint |
                     | to shared storage, exit --| back to job queue
^ Fig 2: Swim lane — the Synergy data iterator is the entire application-
  side contract. It registers the job, sends lease updates, checkpoints
  on lease termination, and synchronizes ranks of a multi-GPU job.

The stated requirement is minimal intrusion: "Synergy transparently communicates with the DNN job using a thin iterator API, that is a wrapper around the existing data iterator, thereby requiring minimal code changes to the DNN job script." The lease mechanism is borrowed, not invented — "By default, every job requests for a lease update to continue running on the same server [42]," where [42] is Gavel.

1.2 Where the resource-sensitivity signal comes from

The whole system is downstream of one observation (§2.1): DNN jobs differ in how much CPU and memory they need before throughput saturates. The profiler turns that into a matrix.

                 memory allocation (discrete, e.g. units of 50 GB)
                 +-------+-------+-------+-----+--------+
                 |  m_1  |  m_2  |  m_3  | ... |  m_max |
        +--------+-------+-------+-------+-----+--------+
   CPU  |  c_1   |  est  |  est  |  est  | ... | EMPIR  |
 alloc  |  ...   |  ...  |  ...  |  ...  | ... |  ...   |
        |  c_max |  est  |  est  |  est  | ... | EMPIR  |
        +--------+-------+-------+-------+-----+--------+
                                                   ^ ONLY this column
                                                     is measured, by
                                                     running the job at
                                                     full memory.

   Every cell to its left is computed analytically from the
   CPU-determined preprocessing rate, the known storage bandwidth,
   and MinIO's fixed cache hit rate.
   W_j[c, m] = progress job j makes per round given c CPU, m memory
^ Fig 3: The resource sensitivity matrix W_j and optimistic profiling's
  shape. Structure follows paper Figure 4; the specific c/m axis values
  are shapes-only — the paper gives the profiling *cost* numbers, not a
  labelled matrix.

The word "optimistic" names the assumption: profile at maximum memory, then extrapolate down. The enabling property depends entirely on MinIO — "MinIO ensures that a job gets a fixed number of cache hits per epoch" and "the use of MinIO in Synergy makes cache performance predictable and hence reduces Synergy's profiling costs — allowing optimistic profiling." Synergy rejects the OS page cache for that reason plus isolation: "Synergy makes a conscious decision to use application-level MinIO cache instead of Page Cache because MinIO provides memory isolation across independent jobs sharing the machine." This is a layer-violation trade made on purpose: the profiling cost model is cheap only because a specific application-level cache with an analytic hit-rate guarantee sits underneath it.

2. System-Under-Test Architecture

Two environments: a 32-GPU physical cluster, and an event-driven simulator scaled to 512 GPUs and validated against that cluster on identical replayed traces. Both are homogeneous.

+-------------------- PHYSICAL CLUSTER (32 V100) ---------------------+
|  Internal Microsoft servers, part of a larger multi-tenant cluster  |
|   Server 1        Server 2        Server 3        Server 4          |
|  +-----------+   +-----------+   +-----------+   +-----------+      |
|  | 8 x V100  |   | 8 x V100  |   | 8 x V100  |   | 8 x V100  |      |
|  | 24 CPU    |   | 24 CPU    |   | 24 CPU    |   | 24 CPU    |      |
|  | 500 GB    |   | 500 GB    |   | 500 GB    |   | 500 GB    |      |
|  +-----------+   +-----------+   +-----------+   +-----------+      |
|  Default CPU:GPU ratio  = 3                                         |
|  Fair-share memory      = 62.5 GB per GPU                           |
|  Framework              = PyTorch 1.1.0                             |
+---------------------------------------------------------------------+

+-------------------- SIMULATED CLUSTERS -----------------------------+
|   128 GPUs across 16 servers      512 GPUs across 64 servers        |
|   (load sweeps, policy sweeps)    (Philly trace subrange, 8000 jobs)|
|   Each simulated machine mirrors the physical server config above.  |
|   Simulator: event-driven, Python. Synergy-OPT via cvxpy.           |
|   Fidelity: metrics differ from the physical cluster by < 5%.       |
+---------------------------------------------------------------------+
^ Fig 4: System under test. Note the CPU:GPU ratio of 3 — this is not an
  artifact of the testbed but the paper's central scarcity claim: real
  GPU SKUs ship with 3 to 6 CPUs per GPU while some models want 9-24.

2.1 The scarcity table that motivates everything

Paper Table 2b lists the CPU:GPU ratios of production GPU SKUs — the hardware constraint the whole paper pushes against.

Against that supply, §2.1 reports the demand: "smaller models like ShuffleNet and ResNet18 require 9-24 CPU cores per GPU to pre-process data items," while the paper Figure 2 caption adds "Some jobs such as Transformers need as few as 1 CPU core per GPU to achieve maximum training speed." The gap between "3 to 6 available" and "1 to 24 wanted" is the entire opportunity. Synergy does not add CPUs; it moves them between co-located jobs.

2.2 Workload

Paper Table 4 — 10 DNNs across three task categories:

Traces are drawn as a workload split — the percentage triple (image, language, speech): "if the split for a given trace is (30,40,30), then the percentage of image, language, and speech models in the job trace is 30%, 40% and 30% respectively." Splits used: (60,30,10), (30,60,10), (20,70,10), (33,33,33), (50,0,50). The split is the knob controlling how much resource-sensitive demand exists in the trace, since image and speech models are the hungry ones; (50,0,50) is therefore the adversarial case.

2.3 Traces

  +-----------------------------------------------------------------+
  | ACTUAL PHILLY TRACE (Microsoft, public)                         |
  |   used for: 512-GPU simulation, 8000-job subrange               |
  |   preserves: job GPU demand, arrival time, duration             |
  |   workload split assigned: (20,70,10)                           |
  +-----------------------------------------------------------------+
  | PRODUCTION-DERIVED TRACE (all other experiments)                |
  |   GPU demand : extracted from the Philly trace                  |
  |   model      : assigned from the chosen workload split          |
  |   duration   : 10^x minutes, x ~ U[1.5,3] w.p. 0.8;             |
  |                x ~ U[3,4] w.p. 0.2 (baseline = GPU-proportional)|
  |   arrival    : static (all at t=0) -> makespan; OR              |
  |                dynamic (Poisson lambda) -> avg / 99p JCT        |
  |   measured   : 1000 jobs in cluster steady state                |
  +-----------------------------------------------------------------+
^ Fig 5: Trace construction. Only the 512-GPU run uses the unmodified
  Philly trace; every load-sweep result uses the derived trace so that
  arrival rate can be varied as an independent variable.

Note the subtlety in the duration definition: job durations are defined relative to the GPU-proportional baseline, so the baseline defines the clock. [derived inference] That is the right construction for a speedup claim, but it makes "load in jobs/hr" a baseline-relative unit.

3. Design-Space Diagram

+---------------------------------------------------------------------+
|                       SYNERGY DESIGN SPACE                          |
|                                                                     |
|  Axis 1: SCHEDULING POLICY -- [FIFO] [SRTF] [LAS] [FTF]             |
|          plus big-data baselines [DRF] [Tetris] in 5.7              |
|  Axis 2: ALLOCATION MECHANISM -- [GPU-proportional]                 |
|          [Synergy-GREEDY] [Synergy-TUNE] [Synergy-OPT (bound only)] |
|  Axis 3: WORKLOAD SPLIT (image, language, speech) --                |
|          [60,30,10] [30,60,10] [20,70,10] [33,33,33] [50,0,50]      |
|  Axis 4: CLUSTER LOAD (jobs/hr, Poisson lambda) -- swept;           |
|          points named in text: 4, 5.5, 9 jobs/hr                    |
|  Axis 5: CLUSTER SIZE -- [32 GPU phys] [128 GPU sim] [512 GPU sim]  |
|  Axis 6: CPU:GPU RATIO -- [3] [4] [5] [6]                           |
|  Axis 7: JOB GPU-DEMAND CLASS --                                    |
|          [single-GPU trace] [multi-GPU trace, up to 16 GPUs]        |
|                                                                     |
|  ---------------- HELD FIXED (no sweep) -----------------------     |
|     - Cluster homogeneity (one GPU type)                            |
|     - GPU demand per job: fixed for job lifetime, user-supplied     |
|     - No GPU time-sharing or space-sharing / multiplexing           |
|     - No GPU elasticity (batch size and LR untouched)               |
|     - Caching layer: MinIO, always                                  |
|     - Data augmentation pipeline: as published per model            |
|     - Parallelism: data-parallel only                               |
|     - Network and storage bandwidth: NOT allocated or scheduled     |
|     - Framework: PyTorch 1.1.0                                      |
|     - Round duration: fixed (paper example: 5 minutes)              |
+---------------------------------------------------------------------+
^ Fig 6: Seven swept axes and the held-fixed set. The held-fixed list is
  the more informative half: Synergy schedules exactly two fungible
  resources (CPU, memory) and explicitly leaves network and storage
  bandwidth to future work.

Two exclusions bound the result.

Network and storage bandwidth are not scheduled. §6: "We leave it to future work to explore how ideas in Synergy can also be extended to reason about demands that individual jobs place on storage and network bandwidths." The resource vector is 3-dimensional (GPU, CPU, memory) and the fungible sub-vector is 2-dimensional.

GPU sharing is out of scope, for a workload reason rather than an implementation one: "Synergy works by improving the throughput of jobs that are bottlenecked on data stalls. For jobs that have data stalls, GPU efficiency cannot be improved by multiplexing (spatial sharing) because they are waiting for input data." A data-stalled GPU is idle-but-committed; a second tenant does not help, because the first will want it back the instant its data arrives.

4. Algorithm & Control-Flow Diagrams

4.1 One scheduling round, end to end

  START of round
 (1) [job arrivals] --> Optimistic Profiler (once per job lifetime),
      |                 emits W_j[c, m]
 (2) insert into priority job queue, ordered by the policy metric
 (3) POLICY selects runnable set J_t: top n jobs whose GPU demands
      |   can be EXACTLY satisfied -- CPU/memory ignored here
 (4) build job demand vector (g_j, c*_j, m*_j), where c*_j and m*_j
      |   are the MINIMUM values that SATURATE throughput in W_j
 (5) MECHANISM packs J_t onto servers
      |   GREEDY -> first-fit, skip unschedulable jobs
      |   TUNE   -> sort desc by (GPU, CPU, mem); best-fit;
      |             revert-to-proportional ladder on failure
      |   OPT    -> ILP(1-5) for ideal alloc, then 2nd LP for placement
 (6) DEPLOY event: gRPC allocations to the Synergy iterators
 (7) run for the round duration (paper example: 5 min)
 (8) per job: grant lease update OR terminate
      |   terminate -> iterator checkpoints to shared storage,
      |                job returns to the priority queue
  END of round -> recompute placement next round
^ Fig 7: Control flow of one scheduling round. Step (3) is the hinge:
  the runnable set is chosen on GPU demand alone, which is what keeps
  the mechanism from distorting the policy's fairness ordering.

Step (3) is where TUNE differs structurally from GREEDY: "unlike Synergy-GREEDY, we do not skip over any jobs unless it cannot be scheduled (GPU demand cannot be met). Therefore, we never underutilize the GPUs when the cluster is at full load."

4.2 Optimistic profiling

  Job arrives
 (1) assume FULL memory allocation; sweep the CPU axis only
 (2) prune that sweep by binary search:
      |   +--------------------------------------------------+
      |   | start at MAX CPU; profile a point                 |
      |   |   (~50 iterations, ~1 minute)                     |
      |   | improvement < threshold (say 10%) ?               |
      |   |    YES -> recurse on the LOWER half of CPU values |
      |   |    NO  -> profile MORE points on the UPPER half   |
      |   +--------------------------------------------------+
 (3) analytically fill every remaining (c, m) cell from the
      |   CPU-determined preprocessing speed, known storage
      |   bandwidth, and MinIO's fixed cache hit rate
 (4) emit W_j[c, m]; enqueue the job
^ Fig 8: Optimistic profiling. Two independent savings compose: the
  memory axis is never measured (analytic model, 10x), and the CPU axis
  is subsampled by binary search (8 points instead of 24).

Cost accounting and validation, from §3.1:

Cost is amortized against job length: "We believe that this is a reasonable overhead as it is incurred only once per lifetime of the job, which typically runs for hours."

4.3 Synergy-OPT — two linear programs

Synergy-OPT produces an upper bound, not a deployable schedule: "We do not deploy the optimal allocations due to the challenges enumerated in §4.1.3."

  Runnable set J_t + matrices W_j
      |
 +----v----------------------------------------------------------+
 | LP #1 -- IDEAL ALLOCATION (solved as an ILP)                   |
 |  Fiction: all cluster CPU and memory live on ONE super-machine |
 |  with G units GPU, C units CPU, M units memory.                |
 |  variable y_{c,m,j} = 1 iff job j gets c CPU and m memory      |
 |                                                                |
 |   maximize  sum_j sum_{c,m}  W_j[c,m] * y_{c,m,j}        (1)   |
 |   s.t.      sum_j sum_{c,m}  c * y_{c,m,j}  <=  C        (2)   |
 |             sum_j sum_{c,m}  m * y_{c,m,j}  <=  M        (3)   |
 |             all j:  sum_{c,m} y_{c,m,j}  =  1            (4)   |
 |             all j:  sum_{c,m} W_j[c,m] * y_{c,m,j}             |
 |                            >=  W_j[C_g, M_g]             (5)   |
 |   where C_g = C_i / G_i * g_j , M_g = M_i / G_i * g_j          |
 |         (the GPU-proportional share for job j)                 |
 |   outputs c*_j := c , m*_j := m  if y_{c,m,j} == 1     (6)(7)  |
 +----+-----------------------------------------------------------+
      | (c*_j, m*_j) per job
 +----v-----------------------------------------------------------+
 | LP #2 -- FEASIBLE PLACEMENT ON s REAL MACHINES                  |
 |  variable x_{i,j} = 1 iff job j's (g_j, c*_j, m*_j) lands on    |
 |           machine i;  x_{i,j} MAY BE FRACTIONAL -> job split    |
 |  objective: MINIMIZE the number of fragmented jobs (splitting   |
 |             costs network communication); bound: AT MOST 3s     |
 +-----------------------------------------------------------------+
^ Fig 9: Synergy-OPT's two-stage relaxation. Stage 1 optimizes what to
  allocate under a single-machine fiction; stage 2 repairs that fiction
  by placing it on s real machines while minimizing splits.

Constraint (5) is the fairness floor and the load-bearing line of the paper: no job may end up with less progress than it would have had under GPU-proportional allocation. Every "never worse than proportional" claim traces back to it; Synergy-TUNE re-implements it heuristically rather than as an LP constraint. Theorem 4.1 states "Throughput achieved by LP(1-5) is at least the throughput achieved by an optimal solution to our problem," proved by mapping any optimal assignment (c*, m*) to the feasible LP point y_{c*,m*,j} = 1, which satisfies (1-4).

Why it is not deployable (§4.1.3): (1) "Solving two LPs per scheduling round is computationally expensive... the time to find an optimal allocation increases exponentially." (2) Fractional GPU allocations — "a valid allocation might assign 3.3 GPUs on server 1 and 2.7 GPUs on server 2 for a 6 GPU job" — and GPU sharing is out of scope.

4.4 Synergy-TUNE — the deployable heuristic

  Runnable set J_t (GPU demands already satisfiable)
 (1) SORT jobs by GPU demand, then CPU demand, then memory demand
 (2) for each job j in that order:
      |   pick the server with the LEAST free resources that is
      |   STILL just enough to fit j's demand vector (best-fit)
      |   multi-GPU job -> find the MINIMUM set of servers with
      |   sufficient GPU availability that fits j entirely
 (3) DOES IT FIT?  --YES--> place; next job.  --NO--> ladder:
 (4) LADDER STEP A: is j's demand ABOVE GPU-proportional share?
      |   YES -> revert j to GPU-proportional share, RETRY (2)
      |   NO  -> fall through
 (5) LADDER STEP B: find a server satisfying ONLY j's GPU requirement
      |   by construction at least one job there holds MORE than
      |   GPU-proportional share -> call that set J_s
      |   switch J_s down to GPU-proportional, releasing EXACTLY as
      |   much CPU/memory as j needs -> by design, j now fits
 (6) recurse for all remaining runnable jobs
  WORST CASE: every running job holds exactly GPU-proportional share
              -> Synergy degenerates gracefully to the baseline,
                 never below it
^ Fig 10: Synergy-TUNE's allocation ladder. The critical property is the
  bottom line: the worst case is *equality* with GPU-proportional, not
  degradation. That is what distinguishes TUNE from GREEDY.

Stated as both requirement and result: "Synergy ensures that its allocations results in job throughputs that are never worse than GPU-proportional allocation."

4.5 Synergy-GREEDY and why it fails

Synergy-GREEDY is a first-fit approximation of multi-dimensional bin packing: take the next runnable job in policy order, place it on the first server satisfying all dimensions, and skip it for the round if no such server exists. The paper documents two failure modes that both follow from the skip: (1) "It can result in auxiliary resources being exhausted by jobs, while leaving GPUs underutilized, and fragmented"; (2) "It also hurts the fairness of the scheduling policy as some jobs can be skipped over for a long time if their resource demands cannot be satisfied in the cluster."

Skipping is the whole difference: GREEDY lets the mechanism override the policy's ordering, and TUNE never does. The empirical consequence is in §5.4 — under the (50,0,50) split "Synergy-GREEDY breaks down, and ends up degrading JCTs significantly compared to a GPU-proportional allocation."

4.6 The multi-GPU split constraint

  Job demand vector: (2 GPU, 12 CPU, 300 GB)

  CONSOLIDATED (preferred)      SPLIT (allowed, constrained)
  +----------------------+      +----------------+ +----------------+
  | Server A             |      | Server A       | | Server B       |
  |  2 GPU / 12 CPU      |      |  1 GPU / 6 CPU | |  1 GPU / 6 CPU |
  |  300 GB              |      |  150 GB        | |  150 GB        |
  +----------------------+      +----------------+ +----------------+
                                  ^ CPU and memory MUST be proportional
                                    to the GPU split across servers

  WHY: each GPU runs a separate process; the processes synchronize at
       regular intervals. If the ratios differ, the job "will eventually
       proceed at the speed of the process with the lowest allocation of
       CPU and memory."
^ Fig 11: Multi-GPU allocation rule. An asymmetric split creates a
  straggler rank, so Synergy forbids it — resource symmetry across the
  ranks of one job is a hard constraint, not an optimization.

Single-GPU jobs get a stronger rule: "The GPU, CPU, and memory resources requested by a single-GPU job must all be allocated on the same server."

5. Quantitative Results — Empirical Findings by Regime

5.1 The motivating example (paper Tables 1-3, Figure 3)

Two physical servers, each 8 GPUs / 24 CPUs / 500 GB DRAM. Four jobs, each requesting 4 GPUs.

Paper Tables 1-3 collapsed into one view (every job requests 4 GPUs; cells are server / CPU cores / GB DRAM):

Result: "The increased resource allocation to CPU and memory sensitive jobs in Schedule 2 speeds up J1 and J2 significantly, while leaving the runtime of J3 and J4 unaffected. The average JCT in the cluster thus drops by 1.5x."

Two separable things happened at once: re-pairing (J1 stops sharing S1 with J2 and shares it with J3 instead, so a sensitive job now sits beside an insensitive one) and re-allocation (12/250 becomes 23/400). Neither alone would do it, and both schedules use exactly 24 CPU and 500 GB per server — nothing was added.

5.2 Sensitivity magnitudes (paper §2.1)

The asymmetry is the point: language models are "GPU compute bound" so "fetching data items from storage if they are not available in memory does not affect training throughput," while "image and speech models benefit from larger DRAM caches."

5.3 Physical cluster, 32 V100 GPUs (paper Table 5)

Headline claims stated by the paper for this table:

The Deploy/Simulate column pair is the paper's fidelity argument, and it does real work: every subsequent result at 128 and 512 GPUs is simulated only, so this table is the sole calibration against hardware.

5.4 Philly trace, 512 GPUs / 64 servers, split (20,70,10)

Paper Figure 6a — average JCT in hours, by policy:

Paper Figure 6b — SRTF cluster metrics, 1000 monitored jobs split into short (JCT < 4 hrs) and long:

Stated interpretations: "Synergy reduces the tail of the distribution by 2.2x for short jobs and the average JCT of both long and short jobs by 15%." Paper Figure 6c reports per-job speedups: "Synergy speeds up jobs by upto 9x using better resource allocations."

Note the shape: the 99th percentile for short jobs improves 2.2x, while for long jobs it barely moves (660 -> 641). [derived inference] Long jobs are dominated by their own run time, so a data-stall fix shortens the queueing prefix rather than the body.

5.5 Simulation with varying load, 128 GPUs / 16 servers, split (20,70,10)

The FIFO row is the strongest single data point in the paper: 81 hours to 22 hours against a floor of 20, meaning Synergy-TUNE captures nearly all the achievable headroom at that operating point.

Gains grow with load, via two composing mechanisms: "Synergy improves cluster metrics by both reducing qeuing delays and speeding up individual jobs." At low load the gap is smaller because "at low load the cluster is not at full capacity." The non-obvious part is what the baseline does at high load: "in GPU-proportional allocation, at higher loads, all CPUs and memory in the system are allocated to the running jobs but they can still be underutilized by individual jobs." Allocation is not utilization, and that gap is what Synergy harvests. (The load-sweep figures — paper Figures 7-9 and 11-12 — are plots whose axis values are not in the PDF text layer, so only the operating points named in the body text are reported here.)

5.6 Workload split sensitivity (paper §5.4, Figure 11)

Stated: "even in the worst case workload split shown in Figure 11c, where all the jobs are CPU- and memory-sensitive, Synergy-TUNE performs as good as GPU-proportional allocation." For GREEDY, the opposite: "As the percentage of resource-sensitive jobs increase... Synergy-GREEDY breaks down, and ends up degrading JCTs significantly compared to a GPU-proportional allocation. This is because, the naive greedy technique results in resource fragmentation when the demand along CPU and memory dimensions are high, leaving several GPUs underutilized."

5.7 Resource utilization (paper §5.4, Figure 10)

Synergy-GREEDY "severely under-utilizes GPU resources throughout the workload, trading it off for higher CPU and memory allocation" — the cleanest statement of the failure: greedy trades the expensive resource for the cheap one.

5.8 CPU:GPU ratio sweep (paper §5.5, Figure 12)

FIFO scheduler, single-GPU trace, at load 9 jobs/hr:

Mechanism, stated: "As the CPU:GPU ratio in a server increases, the baseline GPU-proportional scheduler gets more CPU cores per GPU, thereby reducing data stalls in the baseline. This in turn, reduces the gap between GPU-proportional and Synergy-TUNE." Synergy's gain is a function of how scarce CPU is relative to demand and decays monotonically as hardware gets more generous — but it is still 1.8x at ratio 6, the most generous SKU in Table 2b.

5.9 Synergy-TUNE vs Synergy-OPT (paper §5.6)

At 128 GPUs, Synergy-TUNE "converges at allocations that are within 10% of the optimal value, 200x faster than Synergy-OPT." TUNE's per-round allocation time "is hardly a second," while OPT's "increases exponentially with increasing cluster sizes." On the physical-cluster trace of Table 5, TUNE lands within 4% of OPT.

5.10 Comparison to big-data multi-resource schedulers (paper §5.7)

Cluster of 128 GPUs. DRF and Tetris are fed Synergy's profiled best-case CPU/memory demands, so the comparison isolates the mechanism, not the profiler.

W1 = (20,70,10) is described as "a good mix of resource-sensitive as well as resource-insensitive jobs"; W2 is "dominated by resource-sensitive jobs, which is one of the worst-case scenarios for multi-dimensional scheduling as it could lead to GPU fragmentation."

Attribution, stated: "Synergy is able to allocate auxiliary resources in a fungible-manner every round, whereas the big-data scheduler's static allocations performs similar to greedy techniques, resulting in GPU fragmentation, and thereby degrading the overall cluster metrics." The structural difference named in §7 is that for big-data jobs "resource demands are prior known," whereas "the primary resource in a DNN job is the accelerator (GPU), whose requirement is specified by the job; other resources are fungible."

6. Configuration-Regime Trade-off Tables

6.1 Mechanism choice

The GREEDY column makes one argument: multi-dimensional bin packing with a skip rule is not merely suboptimal, it is worse than not trying, because it converts a cheap-resource shortage into an expensive-resource idle.

6.2 Workload-composition regimes

Synergy's gain is a disparity harvest: it needs insensitive jobs to donate. A homogeneously sensitive trace has nothing to redistribute, and the design's honest response is to fall back rather than to thrash.

6.3 Hardware-provisioning regimes

6.4 Job-shape regimes

The single-vs-multi gap (3.4x vs 1.6x) is the price of the symmetry constraint in Fig 11: a multi-GPU job cannot accept an opportunistic surplus on one server unless every server holding its ranks can match it.

7. Bottlenecks & Insights Surfaced by the Measurements

7.1 The bottleneck is not the GPU

Every DNN scheduler surveyed in §7 — Gandiva, Tiresias, Themis, Gandiva-Fair, Gavel, AFS, Pollux — treats the GPU as dominant and derives CPU and memory from it. Synergy shows that for image and speech models the binding constraint is the CPU preprocessing pipeline and the DRAM cache, so a GPU-proportional derivation systematically under-serves them while over-serving language models. The 3.1x AlexNet figure at ratio 12 and the 1-CPU-per-GPU Transformer figure are two ends of one measurement.

7.2 Allocation is not utilization

The most portable observation in the paper: "in GPU-proportional allocation, at higher loads, all CPUs and memory in the system are allocated to the running jobs but they can still be underutilized by individual jobs." A fully-allocated cluster can be a 60%-utilized cluster (paper Figure 10b). Synergy's gain at high load comes from re-pointing allocated-but-idle CPU at a job that will use it.

7.3 Fungibility is what separates DNN scheduling from big data

Big-data schedulers assume the job declares its full resource vector. Synergy assumes exactly one dimension is declared (GPU) and the rest are the scheduler's to choose. That single change is why DRF's and Tetris's static allocations behave "similar to greedy techniques" here, and why their mechanisms cannot be lifted unmodified.

7.4 Predictability is what makes profiling cheap

Two predictability claims compose. Iteration-level: "Since DNN training has a highly predictable structure, empirically evaluating training throughput for a few iterations gives a fair estimate of the actual job throughput" — 50 iterations per profile point. Cache-level: MinIO's fixed hit rate makes throughput-vs-memory analytically modellable. Without the first, profiling would need epochs; without the second, the full grid. Together they turn 240 minutes into 8.

7.5 The fairness floor makes the heuristic safe; fragmentation is the

tax it pays

Synergy-TUNE has no optimality proof. It has a floor: the revert-to-proportional ladder guarantees that in the worst case every job runs at exactly its GPU-proportional throughput — downside zero, upside measured at 10% from optimal. [derived inference] This is the safety pattern of an admission-control fallback: the heuristic need not be good if it provably never loses. Both OPT and TUNE then spend their remaining design effort on fragmentation rather than throughput maximization. OPT's second LP has minimizing fragmented jobs as its objective and bounds them at 3s; TUNE uses best-fit — "the server with the least amount of free resources just enough to fit the demand vector" — for the same reason. GREEDY does neither, which is the whole distance between "1.4x better" and "significantly worse than baseline."

7.6 The gain decays with hardware generosity; consolidation stays open

The CPU:GPU sweep (3.4x, 3x, 2.2x, 1.8x) is a self-limiting result the paper publishes rather than hides. Its counterweight is in §6: "recent schemes such as RandAugment [14], AutoAugment [13] which consider more computationally-intensive augmentation schemes... Such a rising trend in extreme preprocessing, makes a strong case for a system like Synergy." The trade Synergy declines to quantify is consolidation: "some jobs may benefit from giving up consolidation if the throughput gain due to increased CPU or memory allocation is higher than the penalty due to splitting." The current design resolves this by fiat in favour of consolidation.

8. Limitations of the Methodology

Two of these dominate. First: the largest headline number (3.4x) comes from a simulated 128-GPU cluster on a single-GPU trace, while the only hardware-validated number is 1.5x on 32 GPUs. The paper reports both and publishes the simulator's < 5% divergence, but a reader should carry 1.4x-1.5x as the measured result and 3.4x as the modelled one. Second, the MinIO dependency: optimistic profiling is a technique plus a precondition. "If we do not use MinIO, we will have to profile the model at discrete memory allocations which will increase the profiling costs, and also potentially change the trends in profiling matrix" — the second clause is the sharper admission, since without MinIO the shape of W_j may change, not just the cost of obtaining it.

9. Analogy

Synergy is a restaurant that stops seating parties by headcount alone and starts seating them by appetite.

The GPUs are the tables — expensive, fixed in number, and the thing every party books in advance. A party of four asks for a table for four and gets it; Synergy never touches that. What the conventional scheduler also does, automatically, is hand every table for four the same allotment of waitstaff and the same slice of the kitchen's prep counter: four seats, therefore three waiters and a quarter of the counter. That is GPU-proportional allocation, defensible only because nobody measured what each party actually consumes.

Synergy's measurement is that appetite has nothing to do with headcount. A table of four food photographers (ResNet18, ShuffleNet, the speech models) will keep three waiters sprinting and want half the counter, and if they do not get it they sit staring at empty plates while their table — the expensive part — is fully occupied and producing nothing. That is a data stall: the table is booked, the table is idle. A table of four sharing one dish (Transformer, GNMT) needs one waiter and essentially no counter, and more of either changes nothing. The spread is paper Figure 2a: one CPU per GPU at one end, twenty-four at the other, on hardware that ships with three.

The maitre d's move is not to hire waiters. It is to seat one demanding party next to one undemanding party, then hand the demanding one the undemanding one's untouched share. Paper Table 3 is exactly this: J1 and J3 are re-paired onto one server so J1 can take 23 of the 24 waiters and 400 of the 500 units of counter, leaving J3 the single waiter it was always going to use. Nobody was hired, nothing was built, and the same 24 and 500 produced a 1.5x better average wait.

The rest of the architecture makes that move safe at scale. The optimistic profiler is the maitre d' who has learned to predict a party's appetite from a two-minute look at how fast they order — provided the pantry behaves predictably, which is why Synergy insists on MinIO rather than the house pantry. The fairness floor is the house rule that no party is ever served worse than the standard three-waiters allotment, which is what lets the maitre d' improvise without ever needing to apologize. Synergy-GREEDY is the maitre d' who improvises without that rule: he gives the first demanding party everything, then turns away a party of eight because he has no waiters left, leaving the eight-top empty all evening. He optimized the cheap resource and wasted the expensive one.

The honest ending is in the CPU:GPU sweep. The gain is 3.4x in a restaurant with three waiters per table and 1.8x in one with six, because the whole trick is arbitrage against scarcity and scarcity is what the hardware vendor decides. Synergy's bet — stated in the discussion, not the results — is that appetites are growing faster than staffing: every new augmentation scheme adds another course to prepare, and the parties keep getting hungrier.