Also: Brief Summaries Detailed Summaries

Architecture & Measurement-Design Analysis

BytePS: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters

Source: Jiang, Y.; Zhu, Y.; Lan, C.; Yi, B.; Cui, Y.; Guo, C. 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI '20), November 4-6, 2020. URL: https://www.usenix.org/conference/osdi20/presentation/jiang Open source: https://github.com/bytedance/byteps (referenced as [4] in the paper). Authors: Tsinghua University + ByteDance + Google. Yimin Jiang did the work as a Tsinghua PhD intern at ByteDance; Chuanxiong Guo is the senior author at ByteDance. Reader: Direct PDF read via PyMuPDF (gemini-reader free-tier quota exhausted; full text extracted to /tmp/byteps_full.txt). Analyst: Vishwakarma Date: 2026-05-04

System Architecture (the unified CS + SS framework)
Target-Hardware / SUT Architecture (intra-node topologies + cluster fabric)
Design-Space Diagram (axes swept; axes held fixed)
Algorithm / Control Flow Diagrams (workload split, intra-machine flows, async)
Quantitative Results — Empirical Findings by Regime
Configuration-Regime Trade-off Tables
Bottlenecks & Insights Surfaced by the Measurements
Limitations of the Methodology
Note on NCCL Tuning
Analogy

1. System Architecture (the unified CS + SS framework)

BytePS reorganizes distributed DNN training around a single architectural principle: all communication is between a Communication Service (CS) on every GPU machine and a Summation Service (SS) on every machine in the cluster (GPU and CPU alike). This is not a parameter server in the textbook sense and not an all-reduce: it is a parameterized middle ground that contains both as boundary cases. When the number of dedicated CPU machines k = 0, BytePS's traffic pattern matches all-reduce; when k = n (CPU machines equal GPU machines), it matches non-colocated PS; for any 0 < k < n it interpolates between them with provably optimal load assignment.

+------------------------- BytePS Cluster Architecture ------------------------+
|                                                                              |
|  +-------------------------+         +--------------------------+            |
|  |   GPU Machine 0         |         |   GPU Machine n-1        |            |
|  |                         |         |                          |            |
|  | +---------------------+ |   ...   | +----------------------+ |            |
|  | | GPU computation     | |         | | GPU computation      | |            |
|  | | (FP + BP + update)  | |         | | (FP + BP + update)   | |            |
|  | +----------+----------+ |         | +-----------+----------+ |            |
|  |            |            |         |             |            |            |
|  | +----------v----------+ |         | +-----------v----------+ |            |
|  | | Communication Svc   | |         | | Communication Svc    | |            |
|  | | (CS) -- per GPU mc  | |         | | (CS) -- per GPU mc   | |            |
|  | +----------+----------+ |         | +-----------+----------+ |            |
|  |            |            |         |             |            |            |
|  | +----------v----------+ |         | +-----------v----------+ |            |
|  | | Summation Svc       | |         | | Summation Svc        | |            |
|  | | (SS_GPU on GPU mc)  | |         | | (SS_GPU on GPU mc)   | |            |
|  | +----------+----------+ |         | +-----------+----------+ |            |
|  +------------|------------+         +-------------|------------+            |
|               |                                    |                         |
|               +============= 100GbE / RDMA ========+                         |
|                            (RoCEv2, full bisection BW)                       |
|                                    |                                         |
|        +---------------------------+--------------------------+              |
|        |                                                      |              |
|  +-----+----------------+                       +-------------+----------+   |
|  | CPU Machine 0        |                       | CPU Machine k-1        |   |
|  |                      |                       |                        |   |
|  | +------------------+ |          ...          | +--------------------+ |   |
|  | | Summation Svc    | |                       | | Summation Svc      | |   |
|  | | (SS_CPU)         | |                       | | (SS_CPU)           | |   |
|  | | -- AVX summation | |                       | | -- AVX summation   | |   |
|  | | -- < 3 cores @   | |                       | | -- < 3 cores @     | |   |
|  | |    100Gbps       | |                       | |    100Gbps         | |   |
|  | +------------------+ |                       | +--------------------+ |   |
|  +----------------------+                       +------------------------+   |
|                                                                              |
+------------------------------------------------------------------------------+
^ Fig 1: BytePS architecture (Fig. 5 in the paper). CS is per-GPU-machine
  and handles intra-node aggregation; SS runs on every machine (both GPU
  and CPU) and handles cross-node summation. The CPU machines hosting
  SS_CPU are "spare" machines (or spare cores on busy GPU machines)
  donated by the cluster scheduler -- not dedicated parameter servers.

The architecture's load-bearing decision is to separate optimizer execution into two distinct sub-services, placed on hardware that fits each step. Gradient summation — memory-bandwidth-bound, AVX-friendly — runs on the CPU as SS. Parameter update — FLOP-heavy on Adam/RMSProp — runs on the GPU as part of normal computation. This is the Summation Service abstraction, and it is the bridge that lets PS-style topology coexist with all-reduce-class throughput.

+------------------ Component Placement Across Architectures -----------------+
|                                                                              |
|   PS (traditional):                                                          |
|     GPU:  fp -> bp                                                           |
|     CPU:  sum + update     <-- both on CPU; CPU memory BW is bottleneck      |
|                                                                              |
|   All-reduce:                                                                |
|     GPU:  fp -> bp -> sum -> update    <-- everything on GPU; CPU unused    |
|     CPU:  (idle)                                                             |
|                                                                              |
|   BytePS:                                                                    |
|     GPU:  fp -> bp -> update                                                 |
|     CPU:  sum                          <-- the CPU does ONLY the easy half   |
|                                                                              |
+------------------------------------------------------------------------------+
^ Fig 2: Component placement (Fig. 10 in the paper). The BytePS row shows
  the asymmetric split: parameter update -- the FLOP-heavy half -- runs on
  GPU (which is good at FLOPs); summation -- the bandwidth-friendly half --
  runs on CPU (which has AVX). Neither hardware is wasted on the half it
  does poorly.

The internal contract between CS and SS is a hash of partitioned tensor parts:

+---------------------- CS / SS Internal Contract ---------------------+
|                                                                      |
|  1. Tensors are partitioned into <= 4 MB parts                       |
|  2. All CSs index the parts consistently                             |
|  3. A part with index i is hashed into [0, n^2 + k*n - 2*k)          |
|  4. The hash maps to a target SS (CPU or GPU, by Eq. 1 / Eq. 2)      |
|  5. Same part from all GPUs goes to the SAME SS                      |
|     -> SS sums them, returns the result                              |
+----------------------------------------------------------------------+
^ Fig 3: The probabilistic dispatch that approximates the optimal split
  M_SS_CPU and M_SS_GPU under variable tensor sizes.

Two architectural details stand out. First, SS placement is dynamic: a "CPU machine" need not be a dedicated CPU-only node. The cluster scheduler can carve out spare cores from a GPU machine running a non-distributed job and run an SS process there. The 3-month trace from ByteDance's internal cluster showed 20-45% of GPU machines running non-distributed jobs (so their NICs and CPUs were partly idle) and cluster-wide CPU utilization at 20-35% — the resource pool BytePS targets. Second, CS is the only intra-machine controller: it decides traffic volume to each SS (using Eq. 1 and Eq. 2), it picks the local aggregation strategy based on detected GPU/NIC topology, and it owns the RDMA optimization stack.

2. Target-Hardware / SUT Architecture

BytePS is designed and evaluated on two production-cluster hardware regimes that ByteDance routinely operated in 2020. Both share the same inter-machine fabric (100 GbE RoCEv2 with full bisection bandwidth) but differ in intra-machine topology — and the difference dictates a different intra-machine aggregation algorithm. The paper evaluates on up to 256 GPUs (32 machines x 8 V100 each).

+---------------- Cluster: up to 32 nodes x 8 V100 = 256 GPUs ----------------+
|                                                                              |
|  Node 0           Node 1           ...        Node 31                        |
|  +-----------+    +-----------+                +-----------+                 |
|  | 2x Xeon   |    | 2x Xeon   |                | 2x Xeon   |                 |
|  | Platinum  |    | Platinum  |                | Platinum  |                 |
|  | 32 cores  |    | 32 cores  |                | 32 cores  |                 |
|  | hyper-thr |    | hyper-thr |                | hyper-thr |                 |
|  | + MKL     |    | + MKL     |                | + MKL     |                 |
|  +-----+-----+    +-----+-----+                +-----+-----+                 |
|        |                |                            |                       |
|     PCIe 3.0 x16 (128Gbps theoretical)                                       |
|        |                |                            |                       |
|  +-----+-----+    +-----+-----+                +-----+-----+                 |
|  | 8x V100   |    | 8x V100   |                | 8x V100   |                 |
|  | 32 GB     |    | 32 GB     |                | 32 GB     |                 |
|  | (PCIe-only|    | (NVLink   |                | (NVLink   |                 |
|  |  OR       |    |  topology)|                |  topology)|                 |
|  |  NVLink)  |    +-----+-----+                +-----+-----+                 |
|  +-----+-----+          |                            |                       |
|        |          1x 100GbE NIC                 1x 100GbE NIC               |
|  1x 100GbE NIC          |                            |                       |
|        |                |                            |                       |
|        +================+============================+                       |
|              100 GbE RoCEv2 fabric, full bisection                           |
|              DCQCN congestion control                                        |
|              point-to-point goodput ~90 Gbps                                 |
+------------------------------------------------------------------------------+
^ Fig 4: SUT — up to 256 V100 GPUs. Network is uniform; intra-node topology
  varies by hardware vendor. Both topologies share one 100GbE NIC per node
  and PCIe 3.0 x16 trunks; what differs is whether NVLink connects GPUs.

The two intra-node topologies are sufficiently different that they require different aggregation algorithms — and the paper devotes Section 4.2 to each.

2.1 PCIe-only topology

+------------------- PCIe-only Machine (Fig. 6 of paper) --------------------+
|                                                                            |
|     +------ NUMA 0 -------+              +------- NUMA 1 ------+           |
|     |                     |              |                     |           |
|     |  +-------------+    |    QPI       |    +-------------+  |           |
|     |  |   CPU 0     |<---+--->300+Gbps  +--->|   CPU 1     |  |           |
|     |  +------+------+    |    cross-     |   +------+------+  |           |
|     |         |           |    socket     |          |         |           |
|     |  +------+------+    |               |   +------+------+  |           |
|     |  |  Mem 0      |    |               |   |  Mem 1      |  |           |
|     |  +-------------+    |               |   +-------------+  |           |
|     |         |           |               |          |         |           |
|     |    +----+----+      |               |    +-----+---+     |           |
|     |    | PCIe SW |      |               |    | PCIe SW |     |           |
|     |    | P0      |      |               |    | P1      |     |           |
|     |    +-+--+--+-+      |               |    +-+--+--+-+     |           |
|     |      |  |  |        |               |      |  |  |       |           |
|     |     G0 G1 G2 G3     |               |    G4 G5 G6 G7     |           |
|     +---------------------+               +---------------------+           |
|                                                                             |
|     NIC -- 100 GbE -- attached to one PCIe switch (e.g. P0)                |
|                                                                             |
|     Measured GPU<->GPU memcpy:                                              |
|       within same PCIe switch:    ~105 Gbps                                |
|       across PCIe switches:        ~80 Gbps  <-- the slow link             |
|                                                                             |
+----------------------------------------------------------------------------+
^ Fig 5: PCIe-only topology -- two PCIe switches each carrying 4 GPUs;
  cross-switch GPU-to-GPU memcpy is 24% slower than intra-switch. This is
  the gap BytePS's CPU-assisted aggregation closes (Section 4.2.1).

2.2 NVLink-based topology

+--------------------- NVLink-based Machine (Fig. 8 of paper) ----------------+
|                                                                              |
|     +----- NUMA 0 -----+           QPI           +----- NUMA 1 -----+        |
|     |                  |   <--- 300+Gbps --->    |                  |        |
|     |  CPU 0 / Mem 0   |                          |  CPU 1 / Mem 1   |       |
|     +-+----+----+----+-+                          +-+----+----+----+-+      |
|       |    |    |    |                              |    |    |    |        |
|     +-+--+ +-+ +-+ +-+--+                         +-+--+ +-+ +-+ +-+--+     |
|     | P0 | |..| |..| P1 |                         | P2 | |..| |..| P3 |     |
|     +-+--+ +--+ +--+ +-++                         +-+--+ +--+ +--+ +-++     |
|       |              |                              |              |        |
|     +-+--+ +--+ +--+ +-++                         +-+--+ +--+ +--+ +-++     |
|     | G0 | |G1| |G2| |G3|                         | G4 | |G5| |G6| |G7|     |
|     +-+--+ +-+--+--+--+-+                         +-+--+ +-+--+--+--+-+     |
|       |     |        |                              |     |        |        |
|       +=====+========+======== NVLink mesh =========+=====+========+        |
|                          1.2 Tbps GPU<->GPU                                  |
|                                                                              |
|     NIC -- 100 GbE -- attached to P0 ONLY                                    |
|                                                                              |
|     Asymmetry: G0, G1, G2, G3 (under P0) compete with NIC for                |
|                P0 -- CPU0 link (the one ~100 Gbps PCIe trunk)                |
|                                                                              |
+------------------------------------------------------------------------------+
^ Fig 6: NVLink-based topology -- four PCIe switches, each carrying two
  GPUs; NIC sits on P0. The architectural fault line is that the NIC
  shares the P0 -- CPU0 PCIe link with G0 and G1, but G4-G7 are far
  away. NVLink (1.2 Tbps inter-GPU) is so much faster than PCIe that
  it lets BytePS route around the contention by reducing into a GPU
  *not* on the contended switch (the paper picks GPU 2).

2.3 Inter-machine fabric

+----------------- Inter-machine Fabric (RoCEv2) -----------------+
|                                                                  |
|  100 GbE / RoCEv2, full bisection bandwidth                     |
|  DCQCN congestion control [75] (assumed effective)              |
|  Point-to-point measured goodput: ~90 Gbps (vs 100 Gbps line)   |
|                                                                  |
|  RDMA verb: WRITE only (paper measured WRITE > READ > SEND)     |
|  Memory: pre-registered per-tensor, page-aligned                |
|  Loopback (CS<->SS_GPU same machine): SHM, NOT RDMA loopback    |
|     -- the "slow receiver symptom" fix in Section 6.2           |
|                                                                  |
+------------------------------------------------------------------+
^ Fig 7: Fabric. The interesting design choice is the SHM bypass for
  intra-node CS <-> SS_GPU traffic; sticking with RDMA loopback caused
  PFCs to fire and degraded the network.

The fabric is "easy" relative to the two intra-node topologies — uniform bandwidth, full bisection, well-tested DCQCN. The hard problem is fitting BytePS's many-to-one summation pattern onto the asymmetric intra-node topology and the one-NIC-per-node bandwidth ceiling. The PFC-storm work in Section 6.2 is real-world plumbing: BytePS's claim to be "close to theoretical optimal" only holds after the SHM bypass, page-aligned memory, and single-SGE-per-WRITE fixes ship.

3. Design-Space Diagram

The evaluation forms a five-dimensional sweep. Most experiments fix four axes and vary one — Fig. 13 sweeps k, Fig. 14 sweeps the intra-machine strategy, and the scalability section (Fig. 15-16) sweeps n and the model.

                   DESIGN SPACE (5 axes + held-fixed)
  +----------------------------------------------------------------+
  |                                                                |
  |  Axis 1: NUMBER OF GPU MACHINES n  (scaling axis)              |
  |    {1, 2, 4, 8, 16, 32}  -- equivalently {8, 16, ..., 256 GPU} |
  |    8 V100 32GB GPUs per machine                                |
  |                                                                |
  |  Axis 2: NUMBER OF CPU MACHINES k  (BytePS lever)              |
  |    {0, 1, 2, ..., n}                                           |
  |    k = 0    -> falls back to all-reduce                        |
  |    k = n    -> falls back to non-colocated PS                  |
  |    0 < k < n -> the regime BytePS is designed for              |
  |                                                                |
  |  Axis 3: INTRA-NODE TOPOLOGY  (2 levels)                       |
  |    {PCIe-only, NVLink-based}                                   |
  |                                                                |
  |  Axis 4: WORKLOAD MODEL  (6 models)                            |
  |    CV:   ResNet-50 / VGG-16 / UGATIT (GAN)                     |
  |    NLP:  Transformer / BERT-Large / GPT-2                      |
  |                                                                |
  |  Axis 5: FRAMEWORK / BASELINE  (multiple)                      |
  |    {TensorFlow native PS, MXNet native PS,                     |
  |     Horovod 0.19 + NCCL 2.5.7 (all-reduce),                    |
  |     ByteScheduler [55] (over PS or all-reduce),                |
  |     BytePS without CPU machines, BytePS with CPU machines}     |
  |                                                                |
  |  Held FIXED:                                                   |
  |    - GPU model: NVIDIA Tesla V100 32 GB                        |
  |    - NIC: 1x 100 GbE per machine                               |
  |    - Network: 100 GbE RoCEv2, full bisection                   |
  |    - CPU: Intel Xeon Platinum, 32 cores + hyper-threading      |
  |    - Memory: 6-channel DDR4-2666 (~1024 Gbps peak)             |
  |    - DNN computation library: cuDNN, MKL                       |
  |    - Tensor partition size: 4 MB (BytePS default)              |
  |    - Synchronization mode: synchronous (asynchronous covered   |
  |      analytically only; Theorem 2)                             |
  |    - Compression: NONE -- BytePS is lossless                   |
  |    - GDR (GPU-direct RDMA): not used (PCIe-only fails the      |
  |      same-PCIe-switch rule; NVLink already optimized)          |
  +----------------------------------------------------------------+
^ Fig 8: 5-axis design space. The most important held-fixed line is
  "no compression" -- BytePS is positioned as a drop-in lossless
  acceleration; gradient compression (e.g. half-precision) is
  orthogonal and stackable [21, 45].

Two absences define scope. First, BytePS does not sweep NCCL's internal knobs. The "all-reduce baseline" is Horovod + NCCL 2.5.7 at default configuration; protocol / nChannels / chunkSize choices inside NCCL are inherited as-is. The comparison is therefore between architectures (PS vs all-reduce vs BytePS), not between NCCL configurations. Second, the asynchronous training mode is given a correctness proof (Theorem 2) but no benchmark. Async support is shipped; its end-to-end win is not measured.

4. Algorithm / Control Flow Diagrams

BytePS's runtime sits between the framework's autograd backend and the network. Three control flows are load-bearing: (1) the inter-machine workload split (Eq. 1, Eq. 2), (2) the intra-machine aggregation (different per topology), and (3) the asynchronous training rewrite. We present each.

4.1 Inter-machine workload split (one iteration)

   START (one iteration; gradients are ready in GPU memory)
       |
       v
  (1) Each CS PARTITIONS gradient tensors into <= 4 MB parts
        - Indexes parts consistently across all GPU machines
       |
       v
  (2) Each CS HASHES every part index i into [0, n^2 + k*n - 2*k)
        - Hash range maps each part to a unique target SS:
            * k = 0  : every part goes to some SS_GPU
            * k = n  : every part goes to some SS_CPU
            * 0 < k < n : split per Eq. 1 / Eq. 2
       |
       v
  (3) CS ADJUSTS load per Eq. 1 and Eq. 2:
        M_SS_CPU = 2(n-1) / (n^2 + k*n - 2*k) * M    bytes per CPU SS
        M_SS_GPU = (n-k) / (n^2 + k*n - 2*k) * M     bytes per GPU SS
       |
       v
  (4) CS SENDS each part to its assigned SS (RDMA WRITE)
        - Local SS_GPU on same machine: SHM bypass (no NIC)
        - Remote SS (CPU or GPU): RDMA over RoCEv2
       |
       v
  (5) Each SS RECEIVES n parts (one per GPU machine), SUMS via AVX,
      SENDS the sum back to all n CSs
        - SS_CPU on a CPU machine: serves k*M_SS_CPU bytes/iteration
        - SS_GPU on a GPU machine: serves n*M_SS_GPU bytes/iteration
       |
       v
  (6) Each CS RECEIVES the summed tensors from all SSs and
      hands the aggregated gradient to GPU compute
       |
       v
  (7) GPU runs PARAMETER UPDATE locally (one optimizer step per GPU)
       |
       v
  END  -- next iteration starts at step (1)

  Optimal communication time (Eq. 6, paper):
       t_opt = 2 * n * (n-1) * M / ((n^2 + k*n - 2*k) * B)

  Acceleration ratios over baselines (Eq. 7):
       g_a = (n^2 + k*n - 2*k) / n^2          vs all-reduce
       g_p = (n^2 + k*n - 2*k) / (2*k*(n-1))  vs non-colocated PS

  Worked example (n=32, k=16):
       g_a = 1.46 (BytePS is 46% faster than all-reduce)
       g_p = 1.52 (BytePS is 52% faster than non-colocated PS)
^ Fig 9: Inter-machine control flow per iteration. The CS performs no
  computation -- it is purely a dispatcher driven by the hash + Eq. 1/2
  load assignment. SS performs only summation, not parameter update.

4.2 Intra-machine aggregation — PCIe-only topology

The paper's Section 4.2.1 introduces CPU-assisted aggregation, a six-step pipeline that prevents direct cross-PCIe-switch GPU-to-GPU memcpy.

   START (BP just finished; each of 8 GPUs has full gradient tensor)
       |
       v
  (1) REDUCE-SCATTER  -- per PCIe switch (l = 4 GPUs each)
        - 4 GPUs under P0 reduce-scatter -> each holds M/4 of the result
        - 4 GPUs under P1 reduce-scatter -> each holds M/4 of the result
        - traffic: (l-1)M/l = 3M/4 per GPU, ALL inside one PCIe switch
       |
       v
  (2) GPU-CPU COPY  -- each GPU pushes its M/l to CPU memory
        - 4 GPUs under P0 copy 4 * M/4 = M bytes to CPU0 mem
        - similarly P1 -> CPU1 (or CPU0 over QPI)
        - link traffic on P_x -- CPU_x: M bytes total
       |
       v
  (3) CPU-REDUCE  -- CPU sums data from both PCIe switches in DRAM
        - no PCIe traffic
        - CPU now holds the globally summed M bytes
       |
       v
  (4) NETWORKING  -- CS sends M bytes to remote SSs, receives M back
        - RDMA WRITE over 100 GbE NIC
       |
       v
  (5) CPU-GPU COPY  -- each GPU pulls its M/l back from CPU mem
        - traffic on PCIe switch -- CPU link: M bytes per direction
       |
       v
  (6) ALL-GATHER  -- inside each PCIe switch, l GPUs all-gather
        - 4 GPUs under P0 all-gather their M/4 -> all hold full M
        - all traffic stays inside the PCIe switch
       |
       v
  END  -- all GPUs hold the globally averaged gradient

  Per-link traffic budget (steps 1+2+5+6, both directions combined):
    PCIe switch <-> CPU link:  M (out) + M (back) = 2M total
    PCIe switch <-> GPU link:  (2l-1)M/l per direction = 7M/4 (l=4)

  Compared to "naive" all-reduce across 8 GPUs spanning PCIe switches:
    PCIe switch <-> CPU link would carry 7M/4 each direction
    -- which is the bottleneck; CPU-assisted aggregation cuts it to M.
^ Fig 10: PCIe-only intra-machine flow (six stages). The key move is
  step (3): the CPU does the cross-switch reduction in DRAM rather
  than letting GPUs memcpy across the slow ~80 Gbps inter-switch link.

The optimality analysis (Section 4.2.1) compares CPU-assisted aggregation to the "brute-force" alternative where each GPU copies its entire data directly to the CPU. With M = 1 GB and n = 4, the brute-force / CPU-assisted convex combination minimizes job completion time at x* = 1/5 (so optimal is 80% CPU-assisted + 20% brute-force, with J* = 0.129 s). Pure CPU-assisted gives J(0) = 0.141 s, 9% slower than optimal but using 4x less CPU memory bandwidth — the paper chooses pure CPU-assisted because brute-force exhausts DRAM bandwidth, which is already the optimizer bottleneck. Versus ring-based all-reduce, CPU-assisted is provably faster: J_ca = M / b(S_j, C_j) * max(1, (2n-1)/(k*n)) is always less than J_ar = 2(n*p-1)*M / (n*p*b_bottleneck) when b_bottleneck < b(S_j, C_j), 23.7% smaller in the paper's measured configuration.

4.3 Intra-machine aggregation — NVLink-based topology

The NVLink case is asymmetric: only one PCIe switch (P0) carries the NIC, but all eight GPUs are NVLinked. CPU-assisted aggregation is not the right move here — the bottleneck is the P0 -- CPU0 link shared with the NIC, and we should reduce into a GPU on a non-contended switch and let NVLink do the heavy lifting.

   START (BP done; each of 8 GPUs has full gradient tensor; NIC on P0)
       |
       v
  (1) REDUCE  -- all 8 GPUs reduce into GPU 2 (under switch P1, NOT P0)
        - all reductions ride the NVLink mesh (1.2 Tbps GPU<->GPU)
        - GPU 2 ends up holding the locally summed M bytes
       |
       v
  (2) GPU2 -> CPU0 COPY  -- GPU 2 copies M bytes to CPU0 memory
        - link used: P1 -- CPU0   (NOT P0 -- CPU0; NIC has it)
        - the NIC retains its full P0 -- CPU0 PCIe bandwidth
       |
       v
  (3) NETWORKING  -- CS sends/receives M bytes via NIC over RDMA
        - NIC runs at full ~100 Gbps because P0 -- CPU0 is uncontended
       |
       v
  (4) CPU0 -> GPU2 COPY  -- aggregated tensor pulled back to GPU 2
       |
       v
  (5) BROADCAST  -- GPU 2 broadcasts the result to GPUs 0,1,3-7
        - all over NVLink, no PCIe consumption
       |
       v
  END

  Selection of reduce root: the paper chose GPU 2 (under P1, not the
  NIC-bearing P0). GPU 3 also works (round-robin gives near-identical
  performance). GPU 0 is the WORST choice -- it competes hardest with
  the NIC. NCCL's default behavior tends to pick GPU 0 because of
  proximity to the NIC, exactly the wrong choice for BytePS.
^ Fig 11: NVLink-based intra-machine flow. Reducing into GPU 2
  instead of GPU 0 is the entire optimization -- it isolates the
  NIC from PCIe contention. The paper's measurement: this swap
  alone gives a measurable speedup over Horovod's hierarchical
  allreduce, which routes through GPU 0.

Section 4.2.3 generalizes the rule into two principles, parameterized by S_n (PCIe switches with both GPUs and NIC) and S_g (PCIe switches with only GPUs):

+----------- BytePS Intra-Machine Topology Rules (Section 4.2.3) -----------+
|                                                                            |
|  Rule 1: If S_n > 0 AND S_g > 0   (asymmetric, e.g. NVLink-based)         |
|     -- CS uses REDUCE + BROADCAST                                          |
|     -- root = GPU not competing with NIC for PCIe bandwidth               |
|                                                                            |
|  Rule 2: If S_n = 0 OR S_g = 0    (symmetric, e.g. PCIe-only)             |
|     -- CS uses REDUCE-SCATTER + ALL-GATHER                                 |
|     -- CPU-assisted aggregation if no NVLink                               |
|                                                                            |
|  Both rules generalize to multi-NIC topologies (just changes S_n, S_g).   |
|                                                                            |
+----------------------------------------------------------------------------+
^ Fig 12: The two intra-node strategies as a topology classifier. BytePS
  detects topology at startup and dispatches to the right strategy.

4.4 Summation Service decision (the FLOPs/bandwidth diagnosis)

   QUESTION: Where should the optimizer run?
       |
       v
  +----+--------------------------------------------------------+
  |  Profile peak demands vs available bandwidth:                |
  |                                                              |
  |  CPU memory BW (DDR4-2666 6-channel):      ~1024 Gbps        |
  |  Adam optimizer mem traffic per gradient:  ~10x  (read+write)|
  |  100 Gbps NIC into CPU memory:             ~200 Gbps         |
  |                                                              |
  |  Demand: Adam over 100 Gbps stream =                         |
  |     200 Gbps (NIC) + 10 * 100 Gbps (Adam) = 1200 Gbps        |
  |        > 1024 Gbps (DRAM ceiling)  <-- BOTTLENECK            |
  +-------------------------+------------------------------------+
                            |
                            v
  +-------------------------+------------------------------------+
  |  But CPU summation alone:                                    |
  |     2x mem (read both inputs + write sum) = 200 Gbps         |
  |     + AVX FP16/FP32 throughput > 200 Gbps (Fig. 9b)          |
  |                                                              |
  |  -> Summation FITS in CPU bandwidth; full optimizer DOES NOT |
  +-------------------------+------------------------------------+
                            |
                            v
  DECISION: Place SUMMATION on CPU (using AVX), PARAMETER UPDATE on GPU.
       |
       v
  COST: every GPU machine recomputes the parameter update redundantly
        (instead of one PS doing it once). FLOPs overhead vs FP+BP:
            VGG-16:    138 MFLOPs / 32 GFLOPs    = 0.43%
            ResNet-50: 26 MFLOPs / 7.8 GFLOPs    = 0.33%
            BERT-Large: 387 MFLOPs / 494 GFLOPs  = 0.078%
        all < 0.5%. Negligible.

^ Fig 13: The FLOPs/bandwidth argument that produces the Summation
  Service split. The CPU is good at summation (AVX), bad at full
  optimizers (memory-BW-bound on Adam). The argument is symmetric:
  GPU is good at FLOPs, irrelevant for summation -- so let each
  hardware do what it does well.

4.5 Asynchronous training rewrite (Theorem 2)

                        Traditional PS-async (Fig. 11a):
                          GPU:   fp -> bp ->            <--- gt
                          CPU:                   sum -> update -> w_{t+1}
                                       (pushes gt up; pulls w_{t+1} down)

                        BytePS-async (Fig. 11b):
                          GPU:   fp -> bp -> update -> w'_{t+1}
                                                          \
                                                           delta = w'_{t+1} - w_t
                          CPU:                                        sum (delta)
                                                                       \
                                                                        overwrite w_t
                                                                        with w_t + delta

  Key invariant (induction proof): for any iteration t and worker i,
       Delta_w_{i,t} = f(g_{i,t})
  -> the CPU's SS, which only sums deltas, ends up with the same w
     as a traditional async PS that runs the full optimizer centrally.
^ Fig 14: Async workflow comparison. The trick: GPU computes its OWN
  parameter update first (locally), then sends only the DELTA to CPU.
  CPU never runs the optimizer; it just sums the deltas into the
  shared latest-parameters vector. Theorem 2 shows the resulting
  sequence of states is identical to PS-async.

5. Quantitative Results — Empirical Findings by Regime

5.1 Theoretical communication time vs `k` (Table 1 in paper)

Architecture	Communication time per iteration	Optimal when
All-reduce	`2(n-1)M / (n*B)`	`k = 0`
Non-colocated PS	`max(M/B, nM/(kB))`	`k = n`
Colocated PS	`2(n-1)M / (n*B)` (same as all-reduce)	`k = 0`
BytePS (Eq. 6)	`2n(n-1)M / ((n^2 + kn - 2k) B)`	all `0 ≤ k ≤ n`

The BytePS formula reduces to all-reduce when k = 0 (n^2 + 0 - 0 = n^2, so the expression collapses to 2(n-1)M / (n*B)) and to non-colocated PS when k = n (n^2 + n^2 - 2n = 2n(n-1), so the expression collapses to M / B). Between those endpoints it is strictly less than either baseline, with the gap reaching its theoretical maximum at k ≈ n/2.

5.2 Acceleration ratios (Eq. 7)

For n = 32, k = 16:

g_a = (n^2 + k*n - 2*k) / n^2 = (1024 + 512 - 32) / 1024 = 1.46 -> 46% faster than all-reduce
g_p = (n^2 + k*n - 2*k) / (2*k*(n-1)) = 1504 / 992 = 1.52 -> 52% faster than non-colocated PS

In the limit k = n, n -> infinity, g_a -> 2. Adding more CPU machines beyond k = n does not help — the bottleneck moves to the NIC bandwidth on the GPU machines.

5.3 Microbenchmark — communication goodput (Fig. 12 in paper)

8 x 1-GPU machines, varying k:

k	BytePS goodput vs theoretical optimum
0	within 1-9% of optimum
1-8	within 1-9% of optimum
8	within 1-9% of optimum

The point-to-point RDMA goodput is ~90 Gbps in the cluster (vs 100 Gbps line rate), and BytePS sits within 1-9% of the analytical t_opt derived from B = 90 Gbps. All-reduce is flat (no benefit from added CPUs); MXNet PS is mainly bottlenecked by RDMA implementation issues that BytePS's Section 6.2 fixes.

5.4 RDMA optimization staircase (Table 2 in paper)

Optimization stage	Throughput	Speedup vs baseline
Baseline	41 Gbps	1.00x
+ SHM (loopback bypass)	52 Gbps	1.27x
+ SHM + page-aligned mem	76 Gbps	1.85x
+ SHM + aligned + 1 SGE/WRITE	89 Gbps	2.17x

The single biggest implementation lift in BytePS is RDMA tuning: from 41 Gbps to 89 Gbps with three plumbing fixes, a 2.17x improvement before any algorithmic optimization.

5.5 End-to-end training — leveraging CPU machines (Fig. 13)

8 GPU machines x 8 V100 = 64 GPUs. Per-GPU batch sizes: UGATIT = 2 images, GPT-2 = 80 tokens. CPU machines varied 0 to 8.

Topology	Best speedup (BytePS k=8 vs all-reduce)	BytePS k=0 vs k=8 gain
PCIe-only, GPT-2/UGATIT	up to 45%	up to 20%
NVLink-based, GPT-2/UGATIT	up to 45%	up to 20%

Adding CPU machines gives BytePS up to 20% additional speedup over the no-CPU case, and BytePS already beats all-reduce by up to 45% even at k = 0. NVLink-based machines see the larger speedup against all-reduce because NVLink eases the PCIe bottleneck and shifts the dominant cost to the NIC, where BytePS's CPU offload helps more.

5.6 End-to-end training — intra-machine topology adaptation (Fig. 14)

8 GPU machines x 8 V100 = 64 GPUs, k = 0.

For PCIe-only: CPU-assisted aggregation gives up to 20% gain over the strawman strategy (the same as common all-reduce / PS, i.e., reduce-scatter + all-gather across all 8 GPUs).

For NVLink-based, the choice of reduce root:

Reduce root strategy	Performance ranking	Comment
`root = 2`	best (BytePS optimal)	non-NIC PCIe switch
`root = 2,3` (round-robin)	tied with `root = 2`	both isolated from NIC switch
`root = all`	poorer	equivalent to Horovod hierarchical
`root = 0`	worst	hardest contention with NIC; equivalent to Horovod normal mode (plain NCCL all-reduce)

So plain Horovod / NCCL ends up at the worst root choice on this topology — not because NCCL is buggy, but because NCCL has no notion of "which GPU shares a PCIe switch with the NIC."

5.7 Scalability — six models, three frameworks, 8 to 256 GPUs (Fig. 15-16)

256 GPU speedups vs all-reduce baseline (Horovod + NCCL 2.5.7):

Model	Framework	BytePS w/ CPU machines	BytePS w/o CPU machines
ResNet-50	TensorFlow	10% to 84% improvement	9% to 53% improvement
VGG-16	MXNet	(within 10-84% range)	(within 9-53% range)
UGATIT (GAN)	PyTorch	84% (most communication-intensive)	53%
Transformer	TensorFlow	(within range)	(within range)
BERT-Large	MXNet	(within range)	(within range)
GPT-2	PyTorch	(within range)	(within range)

ResNet-50 scaling efficiency at 256 GPU: BytePS = 97.5%; all-reduce = 88%. BytePS achieves at least 91.6% scaling factor for five of the six 256-GPU jobs (UGATIT is the outlier at 74%, since it is the most communication-intensive model in the suite). PS is consistently the worst across all configurations.

5.8 Performance breakdown — VGG-16 at 256 GPUs (Section 7.4)

BytePS is 52% faster than native PS on VGG-16 at 256 GPUs. Decomposition:

Component	Contribution to the 52% speedup
Optimal communication design (intra-server)	19%
Summation Service	18%
Implementation (Sec. 6 — RDMA fixes etc.)	15%
Total	52%

So roughly equal thirds: algorithmic, abstraction, plumbing. None of the three is dominant.

5.9 Headline summary

"For representative DNN training jobs with up to 256 GPUs, BytePS outperforms the state-of-the-art open source all-reduce and PS by up to 84% and 245%, respectively." (Abstract.)

The 245% PS speedup is particularly striking: PS is being beaten not because the architecture is wrong (BytePS contains PS as a special case at k = n) but because production PS implementations are far from PS's theoretical limit. BytePS achieves PS's theoretical limit by virtue of Summation Service + RDMA fixes, then outperforms it everywhere else by interpolating to the right k.

6. Configuration-Regime Trade-off Tables

6.1 Communication architecture (PS vs all-reduce vs BytePS)

Dimension	All-reduce	Non-colocated PS	Colocated PS	BytePS	Winner
Optimal at `k = 0`	YES	No	YES	YES (degenerate)	tie
Optimal at `k = n`	No	YES	No	YES (degenerate)	tie
Optimal at `0 < k < n`	No	No	No	YES (Theorem 1)	BytePS
Uses CPU machines	No	YES (required)	No	YES (optional)	BytePS
Uses async training	No	YES	YES	YES (Theorem 2)	BytePS / PS tie
Asymmetric optimizer split	No (all on GPU)	No (all on CPU)	No (all on CPU)	YES (sum CPU, update GPU)	BytePS
Headline 256-GPU result	baseline	far below	far below	up to 84% over all-reduce	BytePS

For the heterogeneous-cluster regime, BytePS strictly dominates. It is mathematically equivalent to the better of {all-reduce, PS} at the boundaries of k, and uniquely optimal between them.

6.2 Intra-machine aggregation strategy

Dimension	Naive cross-PCIe reduce-scatter	CPU-assisted aggregation	Reduce-into-non-NIC-GPU + broadcast	NCCL hierarchical all-reduce	Winner
PCIe-only topology	suffers cross-switch cost	best (within 9% of optimum)	N/A	suffers cross-switch cost	CPU-assisted aggregation
NVLink-based topology	suboptimal (NIC contention)	unnecessary	best (NIC at full BW)	suboptimal (uses GPU 0)	Reduce-into-GPU 2
Demands CPU memory BW	low	moderate (1x M / iter)	low	low	--
Cross-switch GPU memcpy avoided?	NO	YES	YES (via NVLink)	NO	CPU-assisted / NVLink-reduce
Detected by Sn / Sg classifier	--	Sn=0 or Sg=0	Sn>0 AND Sg>0	--	--
Empirical gain vs strawman	baseline	up to 20% gain (PCIe-only)	measurable gain (NVLink)	small	--

For DynamICCL, the operative observation is that the optimal intra-machine algorithm depends on whether the NIC and GPUs share a PCIe switch. This is the classifier S_n / S_g BytePS uses; an RL-based collective tuner has the same information available (via topology probing) and could in principle make the same call.

6.3 Summation-Service component placement

Dimension	Sum on CPU + Update on CPU (PS)	Sum on GPU + Update on GPU (all-reduce)	Sum on CPU + Update on GPU (BytePS)	Winner
CPU memory BW pressure	severe (10x mem traffic for Adam)	zero	low (~2x for sum only)	BytePS
GPU FLOPs pressure	zero	high (full optimizer)	low (only update step, < 0.5% of FP+BP)	BytePS
CPU FLOPs pressure	high	zero	low (AVX summation, <3 cores @ 100 Gbps)	BytePS
Spare-CPU friendly?	YES (uses dedicated CPU)	NO (no CPU work)	YES (uses spare CPU on GPU machines)	BytePS
Redundant work on every GPU?	NO	YES (every GPU runs optimizer)	YES (every GPU runs update)	--
End-to-end training speed (Fig. 9a)	slow (CPU bottleneck)	medium	fast	BytePS

For the heterogeneous-cluster regime, the BytePS split wins because it routes each step of the optimizer to the hardware that has spare capacity for it. PS has the right placement strategy (CPU does some work) but the wrong split granularity (CPU does too much). All-reduce has the right split granularity (GPU does it all) but the wrong placement strategy (no CPU work). BytePS gets both right.

6.4 Async training compatibility

Dimension	PS-async	All-reduce-async	BytePS-async (Theorem 2)	Winner
Supported in canonical form?	YES	NO (no async all-reduce)	YES	--
Optimizer state held on CPU	YES	--	NO (held on GPU)	--
Convergence equivalent to PS-async?	baseline	--	YES (proved)	--
Impl complexity	low	--	moderate (delta protocol)	PS
Empirical benchmark in paper?	not specifically	--	NO -- only proved correct	--

The async benchmark is not in the paper. The contribution is correctness only — that BytePS-async is mathematically equivalent to PS-async despite the optimizer running on a different machine.

6.5 RDMA implementation choices

Dimension	RDMA WRITE (vanilla)	+ SHM bypass for loopback	+ page-aligned memory	+ 1 SGE per WRITE	Winner
Throughput	41 Gbps	52 Gbps (1.27x)	76 Gbps (1.85x)	89 Gbps (2.17x)	All three together
PFC storms?	YES (slow receiver)	reduced	further reduced	negligible	All three together
Bottleneck	NIC DMA contention	DMA alignment	hardware DMA quirk	none	--
Disclosed vendor root cause?	--	--	"no official answer"	"no official answer"	--

For DynamICCL, the lesson is methodological: even on a perfectly-engineered RoCEv2 fabric with DCQCN, naive RDMA verbs leave >2x throughput on the table. Any tuner that doesn't go through this exercise will undersell its hardware.

7. Bottlenecks & Insights Surfaced by the Measurements

7.1 The CPU's bandwidth ceiling is the hidden constraint

The Section 5 analysis is the paper's most quietly important contribution: a 100 Gbps NIC streaming gradients to a CPU running Adam needs 200 Gbps (NIC) + 10 * 100 Gbps (Adam) = 1200 Gbps of CPU memory bandwidth, exceeding the 1024 Gbps ceiling of a 6-channel DDR4-2666 setup. That mismatch — well below the LP/IP debug surface — is why traditional PS is slow in practice. The bandwidth, not the FLOPs, is the bottleneck. Once Summation Service strips the optimizer from the CPU (leaving only summation, ~200 Gbps demand), the CPU comfortably handles 100 Gbps NIC traffic with <3 cores.

7.2 GPU optimizer recomputation cost is genuinely negligible

Moving parameter update from CPU (run once per iteration in PS) to GPU (run on every GPU machine in BytePS) creates redundant FLOPs work. The paper's measured overhead: 0.078% to 0.43% of FP+BP FLOPs on VGG-16, ResNet-50, BERT-Large with SGD. For batched training (typical), the overhead is even smaller because parameter update fires once per batch but FLOPs scale with batch size. The "redundancy tax" of duplicating the update is so small that the architecture decision is essentially free.

7.3 The intra-machine optimum changes with topology

The paper's Section 4.2 makes the strongest pedagogical case in the entire paper: there is no one-size-fits-all intra-machine algorithm. PCIe-only wants reduce-scatter + CPU-assist; NVLink-based wants reduce-into-non-NIC-GPU. The wrong choice (e.g., Horovod's hierarchical mode on NVLink) costs measurable throughput. The classifier (S_n, S_g) is two integers — and BytePS detects them at startup.

7.4 Tensor partitioning is the unsung pipeline lubricator

The 4 MB partition size (Section 4.1) does three things at once: (a) approximates the optimal M_SS_CPU / M_SS_GPU split via hash, (b) exposes parallelism for CS's pipeline, (c) allows bidirectional NIC utilization via overlapped send/receive. The paper credits ByteScheduler [55] and prior work [34] for the pipelining idea, but BytePS's contribution is making it work over its many-to-one CS-to-SS communication pattern (which is harder than ring all-reduce because it has more receivers per sender).

7.5 RDMA loopback creates a 2:1 incast on the NIC

The most subtle finding in Section 6.2: when CS and SS run on the same GPU machine, naive RDMA loopback creates a 2:1 incast (RX from network + loopback both target the same NIC DMA-to-memory engine). The fix — a shared-memory bypass for same-machine traffic — is conceptually obvious in retrospect but operationally invisible without PFC monitoring. Anyone writing a high-performance RDMA collective library should expect to spend weeks on the slow-receiver symptom.

7.6 Scaling efficiency is dominated by communication intensity, not architecture

The 256-GPU results show ResNet-50 at 97.5% efficiency and UGATIT at 74%, with all six models running on the same BytePS+RoCEv2 stack. The variance is the model, not the system: UGATIT has a much larger communication-to-computation ratio than ResNet-50. For DynamICCL, this confirms the same insight from paper 0030: model intensity is the dominant predictor of how much room there is for communication optimization.

7.7 The BytePS-without-CPU-machines mystery

Section 8 reports that even at k = 0 — where BytePS's communication time is theoretically identical to all-reduce — BytePS still empirically outperforms all-reduce. The authors hypothesize this is because BytePS's many-to-one pattern allows more "asynchronicity" (no global out-of-band synchronization), but explicitly mark this as unproven and call for a distributed profiler to investigate. This is the kind of "asymptotically equivalent in theory, faster in practice" gap that arises from collective synchronization overhead — and it is exactly the kind of effect a within-NCCL tuner could investigate.

7.8 Cluster scheduler should treat CPU machines as elastic

The paper's discussion section explicitly calls for cluster schedulers to dynamically scale k for an active job — adding more CPU machines as they become free, releasing them as other jobs need them. The convergence properties of training are unchanged (only system performance is affected by k), so elasticity is a free win. This is a genuinely novel scheduling primitive that BytePS enables but does not implement.

8. Limitations of the Methodology

Limitation	Implication
Async benchmark missing	Theorem 2 proves correctness; no end-to-end async speedup numbers reported
No NCCL knob sweep	All-reduce baseline is NCCL 2.5.7 default; better-tuned NCCL might close gap
No gradient compression evaluated	BytePS is positioned as orthogonal, but the stack is not measured
No GDR (GPU-direct RDMA)	PCIe-only fails GDR's same-switch rule; NVLink already optimized; cloud doesn't support GDR
Only 2 intra-node topologies	Other topologies (NVSwitch, multiple NICs, EFA) not measured directly
Only 100 GbE RoCEv2 fabric	Other transports (InfiniBand, Slingshot, EFA) not measured
Only 1 CPU model evaluated	Intel Xeon Platinum + 6-channel DDR4; AMD EPYC 8-channel results not given
No per-call telemetry	End-to-end throughput only; no NCCL-level breakdown
4 MB partition size fixed	"Reasonable in our environment"; not swept
Reference job approximation	Hash-based dispatch approximates Eq. 1 / Eq. 2, not exact
`n0 = n` assumption (one GPU per node)	Multi-GPU node analysis is in §4.2 but inter-machine analysis assumes uniform
QPI / memory BW assumed > PCIe	Section 4.2.1 explicitly takes this as given
Convergence-quality results not given	The paper claims drop-in equivalence but does not show training-loss curves
No tail-latency reporting	Means only; queueing-tail behavior not reported
No power / energy measurements	All speedups are throughput-only
Unmeasured NIC vendor effect	Three RDMA-fix root causes "not officially confirmed" by NIC vendor
256 GPU max	Beyond-256-GPU regime asserted to scale further; not demonstrated
Single cluster	All measurements from one production fabric; cross-cluster generalization is asserted

The most consequential methodological gap is the absence of NCCL-internal tuning as a baseline. The paper compares BytePS to default-NCCL all-reduce; whether a better-tuned NCCL (different algo / proto / chunkSize per regime) would close some fraction of the 84% gap is left open. This is exactly the question DynamICCL exists to answer.

9. Note on NCCL Tuning

BytePS shows that the default reduce-root choice in NCCL hierarchical all-reduce is regime-dependent and that the wrong choice on NVLink-based machines costs measurable throughput (Fig. 14b: root = 0 is the worst, root = 2 is the best, and Horovod's normal mode picks the worst). The reason is purely topological: GPU 0 sits on the same PCIe switch as the NIC, so it competes for the P0 -- CPU0 link during local reduction. NCCL has no exposed knob for "pick a non-NIC-sharing root," but a tuner that knows the topology classifier (S_n, S_g) can refuse the default and pick a better root. This is a concrete example of a per-machine-topology configuration choice that NCCL-level tuning could in principle make automatically.

10. Analogy

BytePS is the co-located freight depot at a port that hires the dockyard's idle clerks to handle paperwork. The cluster is the port; each GPU machine is a container ship arriving with its own cargo (gradients) that must be reconciled against every other ship's cargo before any of them can sail again (parameter update). The naive "all-reduce" approach is every captain meeting every other captain on a giant rotating raft to compare manifests — bandwidth-optimal among ships, but it leaves the dockyard's clerks (CPU machines) sitting idle on the pier. The naive "PS" approach is one harbour-master clerk in a single shoreside office handling all paperwork for all ships — the office is overwhelmed and becomes the bottleneck, even though there are unused clerks elsewhere on the dockyard. BytePS sets up a federated paperwork system: any spare clerk in the dockyard (Summation Service on a CPU machine) can be temporarily borrowed to summate manifest entries (gradient summation, the easy half), while each ship's captain remains responsible for signing off the final consolidated manifest in their own ship's office (parameter update on the GPU, the FLOP-heavy half). The number of borrowed clerks k flexes from 0 to n based on what the dockyard scheduler has free, and the workload split (Eq. 1, Eq. 2) is calibrated so the slowest party — clerk or captain — finishes at the same moment, with no idle waiting. The intra-ship logistics (intra-machine algorithms) differ by ship class: a container ship without internal cranes (PCIe-only) needs the dock workers to ferry small bundles between holds and the shoreside office (CPU-assisted aggregation), while a ship with built-in cranes (NVLink-based) can consolidate cargo internally and only send out a single bundle through the secondary ramp that doesn't share the gangway with the harbour-master's NIC line (reduce into a non-NIC-sharing GPU). The paper's most important practical insight is the same as the one any port engineer learns: the bottleneck is rarely the cargo or the ship's engine — it is the gangway shared between paperwork and cargo. Find that shared gangway, route around it, and capacity opens up.