Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters (Synergy) — Detailed Summary

Jayashree Mohan, Amar Phanishayee, Janardhan Kulkarni (Microsoft Research); Vijay Chidambaram (UT Austin / VMware Research) | 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI '22), July 11–13, 2022, Carlsbad, CA | Open source: https://github.com/msr-fiddle/synergy

Per-section summary organized by the paper's own headings. Every paragraph of the paper produces at least one bullet; all quantitative tables, equations, and named methods are preserved.

Abstract

• DNN training is a popular workload in enterprises and cloud data centers, run on shared multi-tenant clusters.
• Existing DNN training schedulers treat the GPU as the dominant resource and allocate CPU and memory proportional to the number of GPUs requested ("GPU-proportional allocation").
• Those schedulers ignore a job's sensitivity to CPU and memory: some jobs benefit from more than a GPU-proportional share, others are unaffected by receiving less.
• The paper proposes Synergy, a resource-sensitive scheduler for shared GPU clusters that infers per-DNN sensitivity via optimistic profiling.
• Synergy performs multi-resource, workload-aware assignment across co-scheduled jobs using a new near-optimal online algorithm.
• Headline claim: workload-aware CPU and memory allocation improves average job completion time (JCT) by up to 3.4× over GPU-proportional scheduling, purely by better utilizing existing cluster resources — no hardware upgrade.

1. Introduction

Setting. DNN training is resource-intensive and long-running (hours to days). Enterprises build large multi-tenant clusters with expensive accelerators shared across users and production groups. Jobs specify a GPU demand and run once that many GPUs are free. They are managed either by traditional big-data schedulers (Kubernetes, YARN) or by modern DNN-aware schedulers, which decide GPU allocation while implementing cluster-wide policies optimizing average JCT, makespan, or user-level fairness.

The gap.

• Current DNN cluster schedulers assume GPU is the dominant resource; the user requests a fixed GPU count and CPU/memory are handed out GPU-proportionally.
• DNNs exhibit varied sensitivity to auxiliary resources (CPU, memory).
• Prior work shows data ingestion — reading from storage into memory and pre-processing at the CPU — is computationally expensive and produces data stalls, both in research and at industry scale (Google, Facebook).
• Some image and video recognition models achieve up to 3× speedup by overcoming data stalls when CPU allocation exceeds their GPU-proportional share; other models such as GNMT are unaffected by less than GPU-proportional CPU.

Main insight.

Allocating these auxiliary resources in a workload-aware fashion, rather than the traditional GPU-proportional allocation can significantly improve performance by effectively utilizing cluster-wide resources.

• Synergy targets homogeneous, multi-tenant GPU clusters. Figure 1 (128-GPU cluster, Philly-derived trace, load swept) shows Synergy reduces average JCT and sustains higher load for multiple scheduling policies.
• Synergy allocates auxiliary resources disproportionately, with a guardrail: a job gets less than GPU-proportional auxiliary resources only if that does not degrade its throughput versus GPU-proportional. This mitigates data stalls in several models and raises overall cluster throughput.

Two open problems Synergy must solve:

1. What is the ideal resource requirement for each job (with fixed GPU demand), and how can it be determined with low overhead?
2. How should such jobs be packed onto servers along multiple resource dimensions, especially when the job's demand for those resources is itself tunable?

Optimistic profiling (answer to 1).

• Synergy exploits the predictability of DNN computation to measure job throughput as CPU and memory vary. Profiling runs offline in the Synergy scheduler, before job execution on the cluster.
• Profiling all CPU × memory combinations is computationally expensive. Optimistic profiling instead empirically profiles throughput across CPU allocations assuming maximum memory, then analytically estimates throughput for all remaining CPU × memory combinations.
• The enabling insight: job performance versus memory allocation is predictable under DNN-aware caching such as MinIO, which guarantees a certain cache hit rate.
• Optimistically profiled performance closely resembles true empirical values while reducing profiling time by up to 30× (§3.1). From the profiles, Synergy identifies the best allocation beyond which throughput shows diminishing returns.

Scheduling mechanism (answer to 2).

• Synergy makes round-based scheduling decisions (e.g., 5-minute rounds), like prior DNN schedulers. Each round a policy — FIFO, SRTF, LAS, FTF — identifies the runnable job set; Synergy then packs those jobs across servers along all resource dimensions identified during profiling.
• This is multi-dimensional bin packing, which is NP-hard, so approximate solutions are required.
• Unlike prior big-data multi-dimensional packing (Tetris, DRF) with fixed demands, Synergy contends with fungible demands. Two challenges must be solved in tandem: (a) find an optimal partition of CPU and memory across jobs maximizing throughput while ensuring fairness (every job's throughput at least GPU-proportional), and (b) find a feasible packing of those resources onto servers.

Two algorithms.

• Synergy-OPT: formulated as a linear program; determines an upper bound on achievable throughput for a given workload trace. Impractical because (1) it is computationally expensive as cluster size scales, and (2) it produces fractional GPU allocations unrealizable in real deployments. It nonetheless serves as an aspirational goal against which practical solutions are measured.
• Synergy-TUNE: fast and near-optimal — within 10% of Synergy-OPT in evaluation. If a job does not fit along all dimensions, its demands revert to GPU-proportional if they were above it; if already at or below GPU-proportional, Synergy-TUNE finds another job in the cluster holding more than GPU-proportional and reverts that one instead.
• Synergy-TUNE also outperforms the simpler greedy baseline Synergy-GREEDY, which recursively packs jobs along multiple dimensions using first-fit.

Implementation and results preview. Prototype plus an event-driven simulator, both in Python. Synergy talks to the DNN job through a thin iterator API wrapping the existing data iterator, requiring minimal job-script changes. Across policies and traces, average JCT improves up to 1.5× on a physical 32-GPU cluster and up to 3.4× on a simulated cluster of up to 512 GPUs.

Stated contributions:

1. Identify the importance and need for resource-sensitive scheduling of DNN jobs in multi-tenant GPU clusters (§2).
2. Present Synergy — optimistic profiling plus disproportionate allocation such that no job achieves lower than GPU-proportional throughput (§3).
3. Present Synergy-TUNE, a heuristic mechanism mapping profiler-computed allocations onto the cluster with better utilization than GPU-proportional (§4).
4. Extensive physical and simulated experiments showing up to 3.4× average JCT improvement and support for higher input load (§5).

2. Background and Motivation

Scheduling ML training jobs in a cluster. Training is resource-intensive and long-running; collocating training workloads in a shared multi-tenant cluster is common at large organizations. The work targets clusters like those published by Microsoft and Alibaba, using on-premise servers or cloud VMs with pre-defined GPU/CPU/memory. Each server can host more than one job with varying usage — some heavy on CPU-side pre-processing, others heavy on GPU computation; a server with 8 GPUs can host 8 single-GPU jobs from different users.

Policy vs. mechanism. A policy (FIFO, SRTF, LAS, FTF) decides the set of jobs J to run; a mechanism decides where J runs and how much resource it gets. GPU demand is fixed (user-requested); CPU and memory allocation is fungible.

GPU-proportional allocation defined. During training a mini-batch is fetched from storage into memory, cached for subsequent accesses, pre-processed at the CPU, then copied to the GPU. Existing DNN schedulers, real-world GPU cluster schedulers, and even recent GPU-elasticity schedulers all allocate CPU/memory GPU-proportionally. Concrete example: on a server with 4 GPUs, 16 CPUs, 200 GB memory, a 1-GPU job receives 4 CPUs and 50 GB memory.

2.1 Motivation: Resource sensitivity

Insight. DNNs co-scheduled on a cluster exhibit different levels of sensitivity to CPU and memory allocation during training. Prior characterization of Microsoft's Philly cluster shows CPU cycles are under-utilized in multi-tenant clusters — the disparity in requirements across jobs can be exploited without any hardware upgrade (storage, CPU, or memory).

CPU sensitivity (Figure 2a — per-epoch time vs. CPU count, single-GPU training, dataset fully cached).

• Most image and speech models are sensitive to CPU allocation. Smaller models such as ShuffleNet and ResNet18 require 9–24 CPU cores per GPU to pre-process data items.
• Some jobs (Transformers) need as few as 1 CPU core per GPU for maximum training speed.
• State-of-the-art ML-optimized servers and cloud GPU VMs have a CPU:GPU ratio as low as 3.

• Most language models are insensitive to CPU allocation (Figure 2a(ii)) — they have modest input pre-processing requirements. Transformer models, unlike image classification models, do not perform several unique data augmentation operations per data item per epoch.

GPU VM SKUs and their CPU:GPU ratios (Figure 2b):

Memory sensitivity. Two models are trained with varying memory on a server whose GPU-proportional memory share is 62 GB per GPU: ResNet18 on OpenImages and GNMT on WMT.

• GNMT is insensitive to memory: even at 20 GB (the required process memory for training), throughput is unaffected.
• ResNet18: increasing memory from 62 GB (GPU-proportional) to 500 GB (max) speeds up training by almost 2×.
• Reason: language models such as GNMT and transformers are GPU-compute-bound, so fetching uncached items from storage does not hurt throughput. Image and speech models benefit from larger DRAM caches; an uncached item incurs a storage fetch introducing fetch stalls.

Takeaway. When two jobs must share a server, it is possible to co-locate a CPU-sensitive job with a CPU-insensitive one, enabling resource-sensitive rather than GPU-proportional CPU allocation. Symmetrically, it is always beneficial to pack a memory-sensitive job with an insensitive one, allowing disproportionate memory sharing to improve aggregate cluster throughput.

Worked example (2 physical servers, each 8 GPUs, 24 CPUs, 500 GB DRAM; internal servers at large cloud provider X). Table 1 — four jobs, each requesting 4 GPUs: J1 = ResNet18, J2 = Audio-M5, J3 = Transformer, J4 = GNMT.

Table 2 — GPU-proportional allocation:

Table 3 — resource-sensitive allocation:

• Figure 3 compares epoch times under the two schedules: increased allocation to CPU- and memory-sensitive jobs speeds up J1 and J2 significantly while leaving J3 and J4 unaffected. Average cluster JCT drops by 1.5×.
• Note the co-location change: sensitive J1 is paired with insensitive J3, sensitive J2 with insensitive J4 — the pairing itself creates the slack.

2.2 Synergy Scheduling Policies

• Synergy is not tied to one scheduling policy; it is general enough to improve a wide range (LAS, FIFO, SRTF, FTF), creating "Synergy-augmented" variants of each.
• The main challenge: find an efficient partition of available cluster CPU and memory among jobs that maximizes throughput while ensuring every job's throughput is at least that of GPU-proportional allocation. Synergy's innovation lies in exploiting differences in resource sensitivity across jobs to improve cluster-wide metrics.

2.3 Assumptions and Limitations

Three explicit assumptions, derived from analysis of large multi-tenant clusters: homogeneous clusters, fixed GPU allocation for the lifetime of a job, and use of the MinIO cache.

• Synergy's design is not tied to these, but they aid focused profiling by reducing the dimensionality of the search space.
• In a large-scale production cluster it is practical to assume tens of thousands of accelerators per homogeneous cluster and constant per-job GPU allocation.
• Recent work explores heterogeneous-cluster scheduling and GPU elasticity, but several practical challenges remain — e.g., with elastic training the impact of changing batch sizes and hyperparameters on training accuracy is unclear across tasks. §6 revisits what relaxing each assumption would mean.

3. Synergy: Design

Overview. Synergy is a round-based scheduler arbitrating multi-dimensional resources (GPU, CPU, memory) in a homogeneous cluster. It augments existing policies with resource sensitivity in two steps: (1) identify the job's best-case CPU and memory requirements via optimistic profiling (§3.1); (2) identify a runnable job set for the round using a policy (SRTF, FTF, LAS, …) whose collective GPU demand ≤ available cluster GPUs. Using profiled demands, Synergy packs jobs onto servers along multiple dimensions with a near-optimal heuristic (§4). At round end, the runnable set is refreshed by the policy and placements recomputed. Synergy only alters auxiliary resource allocations; GPU demand is user-supplied and unchanged for the job's lifetime.

                      +---------------------------+
   job arrives  --->  |   Optimistic Profiling    |   once per job lifetime
                      |  (§3.1)  CPU x Mem matrix |
                      +-------------+-------------+
                                    |  resource sensitivity matrix W_j
                                    v
                      +---------------------------+
                      |   Priority Job Queue      |   sorted by policy metric
                      |   (FIFO/SRTF/LAS/FTF)     |   (e.g. SRTF -> remaining)
                      +-------------+-------------+
                                    |  runnable set J_t for this round
                                    v
                      +---------------------------+
                      |  Scheduling Mechanism     |
                      |  GREEDY | TUNE | OPT (§4) |
                      +-------------+-------------+
                                    |  (GPU, CPU, Mem) placement per job
                                    v
                      +---------------------------+
                      |  Deploy on cluster        |  lease grant / terminate
                      |  Synergy iterator + gRPC  |  checkpoint on lease end
                      +---------------------------+

3.1 Optimistic Profiling

• A DNN job is profiled for resource sensitivity once per job lifetime, on arrival, by varying CPU and memory; a resource sensitivity matrix is constructed over discrete CPU × memory combinations (Figure 4).
• Because DNN training has a highly predictable structure, empirically evaluating throughput for a few iterations gives a fair estimate of actual job throughput.

Cost of naive profiling. If profiling one (CPU, memory) combination costs 1 minute, profiling all discrete combinations (memory in units of 50 GB) on a server with 24 CPUs and 500 GB DRAM takes 24 × 10 = 240 minutes (4 hours).

The optimistic-profiling trick.

• With DNN-specific application-level caches like MinIO, throughput as memory varies at fixed CPU allocation is easy to model — MinIO ensures a job gets a fixed number of cache hits per epoch.
• Synergy deliberately uses application-level MinIO rather than the OS page cache because MinIO provides memory isolation across independent jobs sharing a machine. Without MinIO, the model would have to be profiled at discrete memory allocations, increasing cost and potentially changing the trends in the profiling matrix.
• For a given CPU allocation (which determines pre-processing speed) and known storage bandwidth, throughput at varying memory can be modeled analytically. Therefore only the CPU axis at full memory is profiled empirically (Figure 4); remaining entries are estimated.
• This yields a 10× reduction in profiling time, bringing it to 24 minutes.

Memory-model validation (Figure 5a). For an 8-GPU ResNet18 job, modeled throughput is compared to empirical results from training 2 epochs at varying memory. Synergy's estimates are within 3% of empirical results, without actually running the model.

CPU profiling — binary search over CPU counts.

• Exact throughput values are not needed, only the shape of the empirical throughput curve.
• Algorithm: start at maximum CPU allocation and binary-search over CPU values. If a profiled point improves throughput by less than a fixed threshold (say 10%), continue the binary search on the lower half; otherwise profile more points on the upper half.
• Rationale: empirically profile CPU regions showing significant throughput differences, skip regions with little to no improvement.

CPU-profiling validation (Figure 5b). For a 1-GPU ResNet18 job, normalized runtime (w.r.t. 1 CPU) is compared between empirical results averaged over 2 epochs and optimistic profiling averaged over 50 iterations (~1 minute per profile). Synergy mimics empirical performance closely in under 8 minutes, using just 8 CPU profile points instead of 24. This overhead is reasonable since it is incurred once per job lifetime and jobs typically run for hours. Together with the 240-minute naive baseline this is the up to 30× reduction claimed in §1.

After profiling, the job plus its sensitivity matrix is enqueued into the main scheduling queue, from which the policy picks runnable jobs each round.

3.2 Scheduling mechanism

• At the start of each round Synergy identifies runnable jobs from the scheduling queue that can be packed for the round's duration, using FIFO, SRTF, LAS, or FTF. Using the sensitivity matrix it packs them onto available servers satisfying multi-dimensional resource constraints rather than doing GPU-proportional allocation.
• Job demand vector: encodes GPU demand plus best-case CPU and memory requirements. Best-case values come from the sensitivity matrix as the minimum value of CPU and memory that saturates the job throughput.
• Packing jobs with multi-dimensional demands is analogous to multi-dimensional bin packing, which is NP-hard — hence a naive greedy approximation is evaluated first.

3.3 Synergy-GREEDY: Greedy Scheduling

• A naive greedy multi-resource packing algorithm is a first-fit approximation to multi-dimensional bin packing. Given a job demand vector, it takes the next runnable job chosen by the policy and places it on a server satisfying the job's demands in all dimensions. If no such server exists, the job is skipped for this round and the next runnable job is checked.
• Two major problems result: (1) auxiliary resources can be exhausted while GPUs remain underutilized and fragmented — shown in §5.4 to severely degrade cluster objectives; (2) it hurts fairness, since jobs can be skipped over for a long time if their resource demands cannot be met.

Bridge to §4. The challenge is a mechanism that eliminates GPU under-utilization from fragmentation and upholds the fairness properties of the policy while still doing multi-dimensional allocation. To judge a heuristic, the paper first formulates a theoretical upper bound on optimal cluster throughput given a set of jobs and their sensitivity profiles, then discusses the challenges of materializing that optimum, then introduces the close-to-optimal Synergy-TUNE.

4. Scheduling Algorithms

4.1 Synergy-OPT

• Goal: allocate CPU and memory to each job maximizing overall throughput while guaranteeing each job makes at least as much progress as under its GPU-proportional share. The problem is NP-hard, so approximate solutions are found via LP formulation.
• To find an upper bound on achievable throughput, two LPs are solved. The first is described in the paper; the complete formulation and proof (plus the heterogeneous-cluster extension) are in the extended version (arXiv:2110.06073).

4.1.1 Finding ideal allocation

• Idealized setting: all CPU and memory across all machines is present in one (super) machine with G units of GPU, C units of CPU, M units of memory. With s homogeneous machines, per-machine G_i = G/s, C_i = C/s, M_i = M/s.
• Find ideal CPU c*_j and memory m*_j for every job j (GPU demand g_j) in the runnable set J_t for the round.
• LP variables y_{c,m,j}: if y_{c,m,j} = 1, then c units of CPU and m units of memory are allocated to job j. There is one variable per job per possible (CPU, memory) allocation, drawn from the discrete space of the sensitivity matrix W_j.
• W_j[c,m] = the progress job j makes per round given c CPU and m memory.
• GPU-proportional reference allocation for machine i: C_g = C_i / G_i * g_j and M_g = M_i / G_i * g_j. Under the GPU-proportional baseline the progress per round is W[C_g, M_g].

(1)  Maximize   SUM_{j in J_t} SUM_{[c,m]}  W_j[c,m] * y_{c,m,j}

(2)  SUM_{j in J_t} SUM_{[c,m]}  c * y_{c,m,j}  <=  C           (CPU capacity)

(3)  SUM_{j in J_t} SUM_{[c,m]}  m * y_{c,m,j}  <=  M           (memory capacity)

(4)  for all j in J_t:  SUM_{[c,m]} y_{c,m,j} = 1               (one config per job)

(5)  for all j in J_t:
        SUM_{[c,m]} W_j[c,m] * y_{c,m,j}  >=  W_j[C_g, M_g]     (no worse than fair share)

Theorem 4.1. Throughput achieved by LP(1–5) is at least the throughput achieved by an optimal solution to the problem.

Proof. Consider an optimal solution O to our problem. Suppose job j receives c* units of CPU and m* units of memory in O. Then we define the following feasible solution to our LP (1-5): Set y_{c*,m*,j} = 1. Clearly, this is a valid solution and satisfies constraints (1-4).

• In experiments this is solved as an Integer Linear Program (ILP) with boolean y_{c,m,j}. Definitions (6) and (7): for each job j, c*_j := c if y_{c,m,j} == 1 and m*_j := m if y_{c,m,j} == 1.

4.1.2 Feasible Allocation on Multiple Machines

• LP(1–5) assumed all resources sit on one machine; in reality they are spread across machines, so a second LP finds a feasible multi-machine allocation. Its objective is to minimize the number of jobs that get fragmented, accounting for the communication overhead when a job is split across machines.
• Variables x_{i,j} (machine i, job j): x_{i,j} = 1 means job j's resources (g_j GPU, c*_j CPU, m*_j memory) are allocated on machine i. x_{i,j} can be fractional, in which case job j is split across machines.
• The paper proves the second LP's solution ensures the total number of fragmented jobs is at most 3s (s = number of machines). Detailed formulation is in the extended version.

4.1.3 Challenges with operationalizing Synergy-OPT

1. Computational cost. Solving two LPs per scheduling round is expensive; as cluster size and jobs-per-round increase, the time to find an optimal allocation grows exponentially (§5.6).
2. Fractional GPU allocations. The second LP can assign fractional GPUs when jobs are split — e.g., 3.3 GPUs on server 1 and 2.7 GPUs on server 2 for a 6-GPU job. Realizing this requires a rounding heuristic, since GPU time/space sharing and its performance impact are out of scope.

4.2 Synergy-TUNE

• Goal: a mechanism performing multi-dimensional allocation where GPU demand is fixed but auxiliary allocations are fungible, ensuring (1) the fairness properties of the policy are not affected, and (2) expensive GPU resources are not underutilized.

Allocation requirements.

• All GPU, CPU, and memory requested by a single-GPU job must be allocated on the same server.
• A multi-GPU distributed-training job may be consolidated on one machine or split across machines. If split, CPU and memory must be proportional to the GPU allocation across servers — e.g., a job needing (2 GPU, 12 CPU, 300 GB DRAM) split across two servers must get (1 GPU, 6 CPU, 150 GB) on each.
• Rationale: multi-GPU jobs train a separate process per GPU and synchronize at regular intervals (after one or many iterations). If each GPU does not get the same resource ratio, per-process performance varies and the job proceeds at the speed of the process with the lowest CPU/memory allocation.

Fairness requirement. No job may run at a throughput lower than under a GPU-proportional share of CPU and memory. The priority order of jobs identified by the policy must be respected — e.g., FIFO is a priority queue sorted by arrival time.

Runnable-set selection. Synergy-TUNE picks the top n jobs from the scheduling queue whose GPU demands can be exactly satisfied by available servers, irrespective of their other (fungible) demands. Unlike Synergy-GREEDY, no job is skipped unless its GPU demand cannot be met — so GPUs are never underutilized when the cluster is at full load.

Packing. Synergy-TUNE greedily packs each runnable job along multiple dimensions onto one of the available servers, with the objective of minimizing fragmentation. Runnable jobs are sorted by GPU demand, then CPU, then memory demand. For each job j in order, pick the server with the least amount of free resources that is just enough to fit j's demand vector. For a multi-GPU job, find a minimum set of servers with sufficient GPU availability that can fit the job's demands in entirety.

Fallback when the job does not fit along all dimensions:

1. If the job's demand vector is greater than proportional share, switch its demand to GPU-proportional share and retry.
2. If it still does not fit, or if its demand was already ≤ GPU-proportional:
   • (a) Repeat step 1 ignoring the job's CPU and memory requirements — find a server that just satisfies the job's GPU requirement. By construction there is at least one job on that server allocated more than GPU-proportional. Identify that job or set J_s, switch them to GPU-proportional share, and release just as much resource as job j requires. By design, j will then fit.
   • (b) Continue recursively for all runnable jobs.

• In the worst case all running jobs in a round get GPU-proportional share — hence Synergy's allocations produce job throughputs never worse than GPU-proportional. §5.6 empirically confirms Synergy-TUNE is practical and near-optimal.

4.3 Implementation

• Synergy and its simulator are implemented in Python; the scheduler is event-driven. A global event queue holds job arrivals, schedule events, and deploy events, handled in arrival-time order.
• A priority job queue holds all jobs arriving into the cluster after profiling, sorted by the policy's priority metric (e.g., SRTF sorts by remaining time).
• On a schedule event the scheduler collects runnable jobs and computes placement for the next round using Synergy-GREEDY, Synergy-TUNE, or Synergy-OPT. On a deploy event, allocations are deployed onto the cluster.
• By default every job requests a lease update to keep running on the same server; the scheduler either grants it or terminates the lease and returns the job to the queue.
• Scheduler and jobs interact via a thin API in the Synergy data iterator, a wrapper around the default PyTorch and NVIDIA DALI iterators. The iterator registers the job with the scheduler, sends lease updates, checkpoints the job to shared storage if its lease is terminated, and synchronizes across GPU processes in a multi-GPU job so each process makes identical progress. gRPC carries scheduler↔︎job communication.
• Synergy-OPT is implemented in cvxpy for use in the simulator. The optimistic profiling module is also Python; it profiles incoming jobs hooked to the Synergy iterator before their initial addition to the scheduling queue (one-time overhead per job).

5. Evaluation

Evaluation uses trace-driven simulation from production cluster traces plus physical cluster deployment, answering: does resource-sensitive scheduling improve makespan and average JCT on a physical cluster (§5.2) and in large-scale simulation (§5.3); how do Synergy-TUNE and Synergy-GREEDY perform across workload splits and utilize resources (§5.4); how does Synergy perform across CPU:GPU ratios (§5.5); Synergy-TUNE vs. Synergy-OPT (§5.6); Synergy vs. big-data schedulers (§5.7).

5.1 Experimental setup

• Physical experiments run on state-of-the-art internal servers at Microsoft, part of a larger multi-tenant cluster. Default assumption unless stated: CPU:GPU ratio = 3 and fair-share memory of 62.5 GB per GPU, matching the server configuration. Simulated machines resemble the physical server configuration.

Models (Table 4) — 10 DNNs (CNNs, RNNs, LSTMs) across 3 tasks:

• Models are categorized by task and assigned weights in traces — the workload split. A split of (30,40,30) means 30% image, 40% language, 30% speech models in the job trace. All experiments use PyTorch 1.1.0.

Traces.

• Publicly available production traces from Microsoft's Philly cluster. The actual Philly trace (preserving job GPU demand, arrival time, duration) is used on the 512-GPU cluster in §5.3.1, over a subrange containing 8000 jobs.
• For all other experiments a production-derived trace is constructed: GPU demand is extracted from the Philly trace, a model is assigned per the chosen split, and runtime/arrival time are scaled for the target cluster size while keeping the duration distribution similar to Philly:
  • Duration: for the GPU-proportional baseline, job duration is sampled from an exponential distribution as 10^x minutes, where x is drawn uniformly from [1.5, 3] with 80% probability and from [3, 4] with 20% probability.
  • Arrival: two kinds — a static trace where all jobs arrive at the start, and a dynamic trace where arrival time is governed by load, a Poisson distribution at rate λ.
• Derived traces with varying arrival rates use the 128-GPU cluster; metrics such as JCT are averaged across 1000 jobs in steady state.
• The physical experiment uses a fixed arrival rate keeping the cluster at full load (GPU demand of all runnable jobs > available GPUs). Simulated experiments sweep load λ and cover two trace categories: all jobs request a single GPU, or multi-GPU distributed jobs requesting up to 16 GPUs.

Policies and metrics. Synergy is evaluated against GPU-proportional scheduling for four policies: FIFO, SRTF, LAS, FTF. Static trace → makespan (time to complete all jobs submitted at trace start). Dynamic trace → average JCT of a steady-state subset, plus the CDF.

5.2 End-to-End Physical Cluster Experiments

Two workload traces run under Synergy-TUNE (tune) and GPU-proportional (proportional): (1) a static production-derived trace of 100 jobs, split (60,30,10), FIFO, evaluated for makespan; (2) a dynamic production-derived trace with continuous arrivals, split (30,60,10), SRTF, evaluated for average and 99th-percentile JCT. Both are sized to keep the cluster fully loaded. Results are compared to the same trace replayed in the simulator and to the Synergy-OPT upper bound.

Table 5 — Physical cluster experiments (time in hours):

• Synergy-TUNE reduces makespan by 1.4× on the static trace vs. GPU-proportional.
• On the dynamic trace, Synergy-TUNE reduces average JCT of steady-state jobs by 1.5× and 99th-percentile JCT by 2×.
• Simulator fidelity: the difference between real and simulated metrics is less than 5%.
• Synergy-TUNE's cluster objectives are within 4% of the optimal solution in this case. Optimal allocations are not deployed on the physical cluster due to the challenges in §4.1.3.

5.3 End-to-end results in simulation

5.3.1 Simulation with production traces

512 GPUs across 64 servers, subrange of the public Philly trace, workload split (20,70,10).

Figure 6a — Average JCT (hrs) on the Philly trace:

• Across all policies Synergy reduces average JCT versus GPU-proportional scheduling due to a better split of resources between jobs. Gains are attributed to reallocating underutilized resources from one job to a different, resource-sensitive job whose throughput improves with the increased allocation.

Figure 6b — Cluster metrics under SRTF, 1000 monitored jobs split into short (JCT < 4 hrs) and long jobs (JCT in hrs):

• Synergy reduces the tail of the distribution by 2.2× for short jobs and the average JCT of both long and short jobs by 15%.
• Figure 6c plots per-job speedup vs. GPU-proportional for each of the 1000 monitored jobs: Synergy speeds up jobs by up to 9×.

5.3.2 Simulation with varying load

• 128 GPUs across 16 servers, production-derived traces, four policies (FIFO, SRTF, LAS, FTF), dynamic workload with Poisson arrivals. Both single-GPU traces (all jobs request 1 GPU) and multi-GPU traces (up to 16 GPUs) are used; metric is average JCT over 1000 steady-state jobs.
• Three scenarios shown: LAS (multi-GPU) in Figure 7, SRTF (multi-GPU) in Figure 8, FIFO (single-GPU) in Figure 9. Workload split (20,70,10) in all cases. CDFs for multi-GPU traces are split into short and long jobs to differentiate the tail.

Three key observations:

1. Synergy-TUNE improves average JCT by up to 3.4× on the single-GPU trace and up to 1.6× on the multi-GPU trace, by speeding up resource-sensitive jobs with disproportionate allocation. The improvement grows as load increases: at low load the cluster is not at full capacity; as load rises jobs queue and incur queueing delay. Since Synergy speeds up individual jobs, pending jobs get scheduled sooner — Synergy improves cluster metrics by both reducing queueing delays and speeding up individual jobs. Under GPU-proportional allocation at high load, all CPU and memory are allocated to running jobs but remain underutilized by individual jobs. At low load, jobs are spread across the cluster and the unallocated CPU and memory is assigned to jobs that benefit from extra auxiliary resources.
2. Synergy-TUNE sustains a larger cluster load than GPU-proportional allocation. For multi-GPU scheduling with LAS, Synergy-TUNE reduced the 95th-percentile JCT of long jobs by 2×.
3. The average JCT achieved with Synergy-TUNE is within 10% of the optimal solution in all cases.

• FTF policy: Synergy-TUNE observed 2.3× (single-GPU trace) and 2× (multi-GPU trace) improvement in average JCT.
• Figure 9 data point (FIFO, single-GPU): at a load of 9 jobs/hr, Synergy reduces average JCT from 81 hrs to 22 hrs, close to the upper bound of 20 hrs predicted by Synergy-OPT.

5.4 Impact of workload split

• The workload split determines the percentage of resource-sensitive jobs. As the share of speech and image models rises, there may not be enough spare CPU/memory for disproportionate allocation, since those models are CPU- and memory-hungry.
• Figure 11 plots average JCT vs. load for three splits under FIFO with multi-GPU jobs: (20,70,10), (33,33,33), (50,0,50).
• As the percentage of resource-sensitive jobs increases, Synergy-GREEDY breaks down and significantly degrades JCTs relative to GPU-proportional allocation — the naive greedy technique fragments resources when CPU and memory demands are high, leaving several GPUs underutilized.
• By design, Synergy-TUNE allocates at least the resources needed to match GPU-proportional throughput. Therefore even in the worst-case split (50,0,50) (Figure 11c), where all jobs are CPU- and memory-sensitive, Synergy-TUNE performs as well as GPU-proportional allocation.

Resource utilization (Figure 10).

• Figure 10a plots GPU allocation over time for the (50,0,50) workload at a load of 5.5 jobs/hr, where cluster GPU demand exceeds 100%. Synergy-TUNE sustains a higher load by finishing jobs faster, while Synergy-GREEDY severely under-utilizes GPU resources throughout, trading GPUs for higher CPU and memory allocation.
• Figure 10b: at low load, GPU-proportional allocation utilizes only 60% of available CPU resources, while Synergy-TUNE utilizes up to 90%, resulting in 1.5× lower average JCT.

5.5 Impact of CPU:GPU ratio

• Prior experiments assumed CPU:GPU = 3 (similar to NVIDIA DGX-2). Figure 12 sweeps the ratio from 4 to 6 (other SKUs from Table 2b) for a FIFO scheduler on a single-GPU trace as cluster load increases.
• As the ratio rises, the baseline GPU-proportional scheduler gets more CPU cores per GPU, reducing baseline data stalls and narrowing the gap to Synergy-TUNE.

Average-JCT reduction at load 9 jobs/hr:

5.6 Comparison to Synergy-OPT

• Computing optimal allocations every round is expensive, especially at large cluster sizes. Experimentally validated: per-round allocation time for Synergy-OPT increases exponentially with cluster size, while Synergy-TUNE is hardly a second.
• Synergy-TUNE's allocations are close to Synergy-OPT's estimates (§5.2, §5.3.2). For a cluster of 128 GPUs, Synergy-TUNE converges to allocations within 10% of optimal, 200× faster than Synergy-OPT.

5.7 Comparison to DRF and Tetris

• Big-data schedulers DRF (Dominant Resource Fairness) and Tetris explored multi-dimensional resource allocation for map-reduce jobs. DNN jobs differ: they are gang-scheduled (they run only when all requested GPUs are simultaneously available); their auxiliary CPU/memory requirements are fungible unlike GPU demand; DRF and Tetris assume resources are statically allocated for the job's lifetime, whereas Synergy treats them as fungible and may vary allocation across the job's lifetime; and profiling the DNN job's resource demands is unique to Synergy — big-data schedulers assume the job request already encodes demands across all dimensions.
• To make the comparison, the best-case CPU and memory requirement (from Synergy's profiler) is fed as input to the big-data policies.
• On a 128-GPU cluster, two workload compositions are evaluated: W1 = (20,70,10) — a good mix of sensitive and insensitive jobs — and W2 = (50,0,50) — dominated by resource-sensitive jobs, a worst case for multi-dimensional scheduling because it can cause GPU fragmentation.

Figure 13 result, workload split W2:

• Reason: Synergy allocates auxiliary resources fungibly every round, whereas the big-data schedulers' static allocations behave like the greedy technique, causing GPU fragmentation and degrading cluster metrics. Synergy performs best in each scenario because it uses best-case resource demands to perform fungible, disproportionate allocation.

6. Discussion and Future Work

Homogeneous clusters. The assumption is grounded in the observation that production clusters have thousands of accelerators per homogeneous cluster. Hardware heterogeneity exists across clusters, but users typically select one homogeneous cluster for a production job — a production cluster may have two homogeneous virtual clusters (VCs), each a specific GPU generation, each managed separately and assigned to training or inference for predictable performance. Recent work explores blurring these boundaries, but co-scheduling poses practical challenges: low-latency inference is business-critical, user-facing, and needs specific hardware and data isolation; other tasks have specific GPU memory requirements or need advanced features like NVLink. Users in the authors' production settings therefore specify an instance type per job. Synergy's ideas can extend to heterogeneous clusters by profiling CPU and memory along an additional dimension — GPU type — at extra profiling cost; the optimal algorithm then maximizes throughput over a 3-dimensional resource-sensitivity matrix W_j (formulation in the extended version).

Use of MinIO. Assumed because it is DNN-aware, outperforms OS page caching, allows performance predictability, provides resource isolation, and reduces storage fetch stalls. Without MinIO, the model would need profiling at discrete memory allocations, increasing profiling cost and potentially changing the trends in the profiling matrix.

Preprocessing overhead. Vision pre-processing includes random cropping and transformations in the critical path. Reusing the same transformed images across epochs hurts accuracy, while pre-processing offline is practically infeasible due to prohibitive storage cost (dataset size × epochs). CPU intensiveness could be altered by varying the number of augmentations, but the paper keeps the augmentations specified by the published models so as not to affect accuracy. Emerging schemes (RandAugment, AutoAugment) use more computationally-intensive augmentation with associated accuracy gains — this rising trend in extreme preprocessing "makes a strong case for a system like Synergy."

Sharing storage and network. The paper reallocates only CPU and memory across jobs resident on the same server (e.g., co-locating a CPU-intensive task with a non-CPU-intensive one), assuming the dataset is downloaded locally and loaded into server memory at job start, constrained by memory allocation limits. Prior work has co-located network-intensive with non-network-intensive jobs, but unlike Synergy those schedulers do not explicitly handle reallocation of shared network bandwidth. Extending Synergy to reason about per-job storage and network bandwidth demands is left to future work.

GPU elasticity and sharing. Some works transparently change GPU allocation during a job's life, but the impact of changing batch sizes and hyperparameters on training accuracy is unclear across tasks — so a constant per-job GPU demand is a practical assumption matching the authors' production clusters. Synergy improves throughput of jobs bottlenecked on data stalls; for such jobs GPU efficiency cannot be improved by multiplexing (spatial sharing) because they are waiting for input data. For the subset of jobs insensitive to auxiliary resource allocation, GPUs could be multiplexed; combining resource-sensitivity awareness with GPU spatial sharing is left to future work.

Trade-off between consolidation and allocation. When multi-GPU jobs are split across servers they may incur a network communication penalty; DNN jobs therefore prefer consolidation. Synergy assumes no more than a server's worth of CPU or memory can be allocated to a job whose GPU demand fits on one server. However, some jobs may benefit from giving up consolidation if the throughput gain from extra CPU/memory exceeds the splitting penalty. Exploring this trade-off, accounting for network overhead, is future work.

Leveraging model and pipeline parallelism. Evaluation assumes distributed data-parallel jobs. Model- and pipeline-parallel schemes also have an input stage that ingests and pre-processes data, but each pipeline stage may have a different CPU:GPU and memory:GPU requirement. Such jobs would have to be profiled per stage, but Synergy's contributions "directly carry forward to such settings."

7. Related Work

DNN cluster schedulers. Recent schedulers each target one objective: cluster utilization (Gandiva), JCT (Tiresias), fairness (Themis, Gandiva-Fair); others exploit performance heterogeneity among accelerators. All assume GPU is the dominant resource — the user requests a fixed GPU count and the job runs when those GPUs are free. Building on GPU elasticity for a single job, AFS and Pollux use throughput metrics to provide GPU elasticity in multi-tenant clusters (also tuning batch size and learning rate). In all these cases CPU and memory are allocated proportional to GPUs; existing schedulers thus ignore resource sensitivity. Synergy shows that irrespective of the number of GPUs allocated, auxiliary resource-sensitive allocation is crucial for better cluster utilization.

Big-data schedulers. Synergy builds on insights from big-data scheduling literature. Tetris and DRF address multi-dimensional allocation for big-data jobs, proposing policies for a specific cluster objective where resource demands are known a priori. In contrast, for a DNN job the primary resource is the accelerator, whose requirement is job-specified; other resources are fungible. Synergy exploits this to perform disproportionate allocation by profiling sensitivity and then packing onto servers.

Data stalls. Recent deep characterization studies explored the impact of CPU and memory on individual DNN jobs. Unlike that prior work, Synergy focuses on "the tricks we can play when we schedule multiple jobs together in a cluster."

Disaggregated data prep. Orthogonal efforts reduce data-preprocessing cost and CPU load using disaggregated data prep, but one must pay the network cost of shuffling preprocessed tensors, which can quickly become the bottleneck, especially for vision models with rich datasets. Synergy instead assumes standard pre-processing pipelines at the training servers and reduces preprocessing cost through better resource allocation.

8. Conclusion

• Synergy is a resource-sensitive scheduler for DNN training jobs, based on the insight that not all jobs exhibit the same sensitivity to CPU and memory allocation during training.
• "Breaking the shackles of GPU-proportional allocation" improves utilization of existing cluster resources and improves both job- and cluster-level objectives.
• Experiments on physical and large simulated clusters show Synergy reduces average JCT by up to 3.4× over GPU-proportional allocation.

Cross-Cutting Summary of Headline Numbers

Limitations

• Homogeneous clusters only. The heterogeneous extension needs a third profiling dimension (GPU type) and appears only in the extended version.
• Fixed GPU allocation per job lifetime. GPU elasticity and GPU spatial sharing are explicitly out of scope; Synergy-OPT's fractional GPU allocations are unrealizable for this reason.
• Depends on MinIO. Without DNN-aware caching the analytical memory model breaks and profiling cost rises.
• Data-parallel jobs only in the evaluation; model- and pipeline-parallel jobs would require per-stage profiling.
• Storage and network bandwidth are not allocated. Only CPU and memory are reallocated; network reallocation is left to future work.
• Consolidation is enforced when a job's GPU demand fits on one server, even though giving up consolidation could sometimes pay off.
• Synergy-OPT is not deployable — exponential per-round solve time and fractional GPU allocations; it is used only as a simulated upper bound.
• Augmentation pipeline held fixed to avoid affecting accuracy, so the CPU-intensiveness axis is not explored.
• Profiling is a one-time per-job overhead (up to ~8 minutes for the validated case), justified only because jobs run for hours.
• Gains shrink as CPU:GPU ratio rises — 3.4× at ratio 3 down to 1.8× at ratio 6 — so the benefit is partly a function of today's server SKUs.
• Worst-case workload splits yield no gain: at split (50,0,50) where all jobs are sensitive, Synergy-TUNE merely matches GPU-proportional allocation.

Open Problems Called Out by the Paper

1. Extending resource-sensitivity awareness to heterogeneous clusters via a 3-dimensional sensitivity matrix over (CPU, memory, GPU type).
2. Combining resource-sensitivity awareness with GPU spatial sharing for the subset of jobs insensitive to auxiliary resources.
3. Reasoning about storage and network bandwidth demands per job, in addition to CPU and memory.
4. Exploring the consolidation-vs-allocation trade-off for multi-GPU jobs, accounting for the network penalty of splitting a job across servers.
5. Profiling and scheduling model- and pipeline-parallel jobs, where each pipeline stage has a distinct CPU:GPU and memory:GPU requirement.