Architecture & Measurement-Design Analysis

PICO: Performance Insights for Collective Operations

Source: Pasqualoni, S.; Bonato, T.; Piarulli, L.; Hoefler, T.; Canini, M.; De Sensi, D. PICO: Performance Insights for Collective Operations. 2025/2026 HPC systems paper (IEEE two-column format; references dated through 2025). Affiliations: Sapienza University of Rome; KAUST; ETH Zurich. Code: https://github.com/HLC-Lab/pico Reader: Native Read tool, chunked (gemini-reader quota exhausted; codex-reader skipped in fallback chain) Analyst: Vishwakarma Date: 2026-07-28

Table of Contents

1. Evaluation Harness Architecture (the "instrument")
2. System-Under-Test Architecture (the "specimen")
3. Design-Space Diagram (collective x message x scale x algorithm)
4. Algorithm & Control-Flow Diagrams (the diagnostic machinery)
5. Measurement Control Flow Through One Experiment
6. Quantitative Results — Empirical Findings by Regime
7. Configuration-Regime Trade-off Tables
8. Bottlenecks & Insights Surfaced by the Measurements
9. Limitations of the Methodology
10. Note on NCCL Tuning
11. Analogy

1. Evaluation Harness Architecture (the "instrument")

PICO is not a microbenchmark that reports one end-to-end number per collective. It is a benchmark-plus-diagnosis framework: a harness whose explicit design goal is to move from "how long did the Allreduce take?" to "which algorithmic step, subsystem, or transport knob is responsible for the time?" The central architectural principle is a clean separation between what to run (a portable, backend-agnostic experiment description) and how to run it (reusable, platform-specific descriptors), so that outputs are simultaneously comparable across machines and diagnosable within a machine.

+-------------------------------------------------------------------+
|                    PICO Framework (Fig. 3 workflow)               |
|                                                                   |
|   +------------------+          +-------------------------+       |
|   | (A) TUI          |          | (B) CLI                 |       |
|   | (interactive)    |          | (scriptable)            |       |
|   +--------+---------+          +-----------+-------------+       |
|            |    both front-ends emit the same spec          |     |
|            +------------------+-----------------------+     |     |
|                               v                             |     |
|   +----------------------+         +----------------------+ |     |
|   | (C) test.json        |         | (D) env.json         | |     |
|   | portable test        |<------->| platform env         | |     |
|   | descriptor           |  bind   | descriptor           | |     |
|   | - collective type    |         | - comm stacks avail. | |     |
|   | - message sizes      |         | - module/env setup   | |     |
|   | - scale (nodes/GPU)  |         | - SLURM launch tmpl. | |     |
|   | - algo/param control |         | - backend knob maps  | |     |
|   +----------+-----------+         +----------+-----------+ |     |
|              |     "what to run"   "how to run it"          |     |
|              +----------------+-----------------+           |     |
|                               v                             |     |
|   +-----------------------------------------------------+   |     |
|   | (E) Benchmark Orchestrator                          |   |     |
|   |   resolves descriptors -> sets up env ->            |   |     |
|   |   builds + launches pico_core instances ->          |   |     |
|   |   manages jobs -> collects results (R4 automation)  |   |     |
|   +--------------------------+--------------------------+   |     |
|                              v                             |     |
|   +-----------------------------------------------------+   |     |
|   | (F) PICO Core (pico_core)  [runs on compute nodes]  |   |     |
|   |   - init communication context                      |   |     |
|   |   - apply requested controls (algo/proto/transport) |   |     |
|   |   - execute target collective over msg x scale      |   |     |
|   |   - internal BARRIER sync for timing alignment      |   |     |
|   |   - emit measurements + metadata (standard format)  |   |     |
|   +------+---------------------------------+------------+   |     |
|          | uses (optional)                 | emits          |     |
|          v                                 v                |     |
|   +----------------------+       +----------------------+   |     |
|   | (G) libpico          |       | (H) results store    |   |     |
|   | backend-neutral      |       | structured + indexed |   |     |
|   | reference collectives|       | measurements +       |   |     |
|   | + tag instrumentation|       | run metadata         |   |     |
|   +----------------------+       +----------+-----------+   |     |
|                                             v               |     |
|                                  +----------------------+   |     |
|                                  | (I) Post-Processing  |   |     |
|                                  | analysis + viz tools |   |     |
|                                  | + network tracer     |   |     |
|                                  +----------------------+   |     |
+-------------------------------------------------------------------+
^ Fig 1: PICO's end-to-end workflow (paper Fig. 3). The control plane
  (A-E) is portable and backend-agnostic; the data plane (F-G) runs on
  compute nodes and does the timing-critical work; the results plane
  (H-I) stores and diagnoses. The test.json / env.json split is the
  load-bearing design decision — it is what makes one experiment
  portable across Leonardo, LUMI, and MareNostrum 5 without editing the
  experiment intent.

The split between test.json and env.json is the architectural pivot. A test.json file encodes control intent — "run Allreduce, sizes 32 B to 512 MiB, at 32/64/.../2048 nodes, sweeping the algorithm" — and nothing cluster-specific. An env.json file encodes platform capability — which communication stacks exist, how to load modules, the SLURM launcher template, and how PICO's abstract knobs map onto each backend's concrete flags. The Orchestrator (E) is the resolver that binds the two together, so the same experiment intent yields a valid, machine-tuned job on any of the three supercomputers. This is the same portability pattern as a build system that separates a Makefile (intent) from a toolchain file (platform): intent is written once, platform mappings are written once per machine, and their product is generated.

The genuinely novel component is libpico (G) paired with tag-based instrumentation. Existing suites (OMB, NCCL-tests, IMB, ReproMPI, CommBench, NetGauge) time the collective as a black box. PICO optionally routes the collective through its own backend-neutral reference implementations, wrapped in PICO_TAG_BEGIN(...)/PICO_TAG_END(...) macros that attribute time to semantically meaningful regions — data staging, per-step communication, per-step reduction. When tags are disabled they compile out to empty statements (measured overhead < 100 ns per tagged region), so instrumentation is free when not in use. This is what upgrades PICO from a benchmark to a diagnostic instrument: it can localize where inside the algorithm the time goes.

   Feature                       OMB IMB NCCL-T CommB NetG ReproMPI PICO
   ---------------------------------------------------------------------
   R1 Fine-grained profiling      X   X    X      X    ~      X      Y
   R2 Backend-neutral references  X   X    X      ~    X      X      Y
   R3 Portable spec & control     ~   ~    X      ~    X      ~      Y
   R4 Automation & usability      ~   ~    ~      X    X      ~      Y
   R5 Metadata-rich reproduce     X   X    X      X    ~      Y      Y
   R6 Extensibility across stacks ~   X    X      ~    X      X      Y
   ---------------------------------------------------------------------
   (Y = built-in, ~ = partial/manual, X = not targeted)  paper Table I
^ Fig 2: Requirements coverage (paper Table I). PICO is the only tool
  claiming built-in support on all six axes; the others each solve a
  subset. The distinguishing axes are R1 (fine-grained profiling) and
  R2 (backend-neutral references) — no prior collective benchmark
  offers both.

A second structural choice: PICO supports heterogeneous communication stacks through a uniform backend-adapter interface, with backend availability selected at compile time (#ifdef NCCL/CUDA gates GPU collectives). Each adapter implements exactly three responsibilities: (i) context initialization, (ii) mapping abstract controls from test.json to backend-specific knobs when those knobs are exposed, and (iii) collective execution plus timing. The adapter contract is deliberately narrow so that adding a backend is a bounded task, and the framework degrades gracefully when a control is unsupported on a given stack rather than requiring every backend to implement every feature.

2. System-Under-Test Architecture (the "specimen")

The specimen is the collective-communication software stack of three production supercomputers, measured through their native libraries. Unlike the flat consumer-GPU testbeds common in DDL surveys, PICO's SUT spans real HPC fabrics: Cray Slingshot on LUMI, a Dragonfly network on Leonardo, and MareNostrum 5. The measured objects are the vendor MPI stacks and the *CCL libraries, running over the machines' actual interconnects at scales up to 2048 nodes.

+------------------- Software Stack Under Test (Fig. 1) ------------+
|                                                                   |
|   +-----------------------------------------------------------+   |
|   |                    Application                            |   | orange
|   +----------------------+------------------------------------+   |
|              |                                  |                 |
|              v                                  v                 |
|   +--------------------+            +------------------------+     |
|   |  MPI               |            |  *CCL                  |     | red/blue
|   |  OpenMPI 4.1.5/6   |            |  NCCL 2.22 / RCCL      |     |
|   |  Cray MPICH 8.1.29 |            |  (oneCCL/xCCL family)  |     |
|   +---------+----------+            +-----------+------------+     |
|             |                                   |                 |
|             v                                   v                 |
|   +--------------------+            +------------------------+     |
|   |  UCC               |            |  UCX                   |     | green
|   +---------+----------+            +-----------+------------+     |
|             |          some MPI / *CCL bypass   |                 |
|             |          middleware -> transport  |                 |
|             +-----------------+-----------------+                 |
|                               v                                   |
|   +-----------------------------------------------------------+   |
|   |  libfabric | libibverbs | sockets | shared memory | ...   |   | cyan
|   +----------------------------+------------------------------+   |
|                                v                                  |
|   +-----------------------------------------------------------+   |
|   |                       Hardware                            |   | black
|   |   Scale-up: NVLink, InfinityFabric, UALink, SUE           |   |
|   |             (up to 72-384 GPU domains, ~10x scale-out BW)  |   |
|   |   Scale-out: InfiniBand / Slingshot / Ultra-Ethernet      |   |
|   |              Dragonfly(+) or tapered fat-tree              |   |
|   +-----------------------------------------------------------+   |
+-------------------------------------------------------------------+
^ Fig 3: The SUT is a layered comm stack, not a single library
  (paper Fig. 1). Two dominant paths: MPI->UCC and *CCL->UCX, both
  bottoming out in libfabric/libibverbs/sockets/SHM. Legacy MPI and
  some *CCL bypass the middleware and hit the transport directly. PICO
  instruments at the top (library call) and attributes downward.

+---- Machines Under Test ------------------------------------------+
|                                                                   |
|  Leonardo (Cray)          LUMI (Cray)        MareNostrum 5        |
|  +-------------------+   +----------------+  +------------------+  |
|  | GPU nodes         |   | GPU nodes      |  | nodes            |  |
|  | Dragonfly network |   | Slingshot      |  |                  |  |
|  | OpenMPI 4.1.6     |   | Cray MPICH     |  | OpenMPI 4.1.5    |  |
|  | UCX 1.15.0        |   | 8.1.29         |  |                  |  |
|  | up to 2048 nodes  |   | up to 1024 nds |  | up to 64 nodes   |  |
|  +-------------------+   +----------------+  +------------------+  |
|                                                                   |
|  Process config in profiling studies: 4 processes / node         |
+-------------------------------------------------------------------+
^ Fig 4: Three-machine SUT. Leonardo's Dragonfly topology is the
  centrepiece of the topology-effect case studies (Sec. 6.3): its
  group structure is exactly what makes distance-halving vs
  distance-doubling Broadcast behave differently despite identical
  cost-model volume.

The topology diversity is the point. On a Dragonfly, communicating rank pairs fall into physically distinct classes — intra-node, intra-group (through a local switch), and inter-group (across the expensive global links). A collective algorithm's partner-ordering determines how much traffic lands on the inter-group links, and that in turn determines congestion. This is invisible to the classical alpha-beta cost model, which counts rounds and bytes but not which physical links carry them. PICO's SUT was chosen specifically so that this gap between the cost model and reality is measurable.

3. Design-Space Diagram (collective x message x scale x algorithm)

PICO sweeps a four-dimensional grid. Every heatmap cell in the headline result (Fig. 6) fixes a machine and reads off (message size, node count); every tuning study fixes all-but-one axis and varies the remaining knob to isolate its effect.

+------------------- DESIGN SPACE (4 axes + held-fixed) ------------+
|                                                                   |
|  Axis 1: COLLECTIVE OPERATION                                     |
|    Allreduce (primary), Broadcast, Allgather, ReduceScatter,     |
|    Reduce  (+ TUI also: Alltoall, Gather, Scatter)               |
|                                                                   |
|  Axis 2: MESSAGE / VECTOR SIZE (9 levels, log grid)              |
|    32 B, 256 B, 2 KiB, 16 KiB, 128 KiB, 1 MiB, 8 MiB,           |
|    64 MiB, 512 MiB                                               |
|                                                                   |
|  Axis 3: SCALE / NODE COUNT (per machine)                        |
|    Leonardo:      32, 64, 128, 256, 512, 1024, 2048             |
|    LUMI:          32, 64, 128, 256, 512, 1024                   |
|    MareNostrum 5: 8, 16, 32, 64                                 |
|    AI traces:     16 GPU, 128 GPU (LLaMA), 64 GPU (MoE)         |
|                                                                   |
|  Axis 4: ALGORITHM x PROTOCOL x TRANSPORT KNOB                   |
|    MPI algo:  all tuned-collection choices per stack            |
|    NCCL algo: {Ring, Tree, Binomial-Butterfly/PAT}             |
|    NCCL proto:{Simple (large-BW), LL (low-latency)}           |
|    Bcast:     {distance-halving, distance-doubling}            |
|    Transport: UCX_MAX_RNDV_RAILS in {2 (default), 4}          |
|                                                                   |
|  Held FIXED per study (to isolate one knob):                    |
|    - Allreduce sweep: vary ALGORITHM only, all else fixed       |
|    - Rendezvous-rail study: algo=Ring @ 32 nodes fixed,         |
|                             vary only UCX_MAX_RNDV_RAILS         |
|    - Processes/node = 4 (profiling studies)                     |
|    - Datatype, operator (reduction op) fixed per run            |
+-------------------------------------------------------------------+
^ Fig 5: The 4-axis sweep. The discipline that makes PICO diagnostic
  rather than merely descriptive is Axis-4 isolation: each tuning
  study freezes three axes and moves one knob, so an observed delta
  is attributable to that knob and not confounded. This is the
  controlled-experiment property most black-box suites lack.

The design-space philosophy inverts a common survey pattern. Where an application-level survey fixes the collective library and varies the training framework above it, PICO fixes the workload shape (collective, size, scale) and varies the library-internal choices below it — algorithm, protocol, and transport rails. This is precisely the layer at which a runtime tuner operates, which is why PICO's measurements read as an atlas of "how much does the library-internal choice matter, and where."

A structural absence worth naming: PICO's headline Allreduce sweep varies only the algorithm. Protocol (Simple vs LL) and the Binomial-Butterfly algorithm enter primarily through the NCCL AI-trace study (Sec. 6.5), and the transport rail count enters through a single focused study on Leonardo. So the design space is deep on MPI algorithm selection at scale and NCCL algo/proto in AI workloads, but it is not a full cross-product of every knob on every machine — a deliberate scoping to keep the campaign tractable.

4. Algorithm & Control-Flow Diagrams (the diagnostic machinery)

Four procedures constitute PICO's diagnostic machinery: the tag-based phase attribution, the phases-rounds-steps decomposition model it produces, the network traffic tracer, and the ATLAHS trace-replay pipeline.

4.1 Tag-based fine-grained attribution (paper Fig. 5)

The instrumented reference Allreduce is realized as ReduceScatter followed by Allgather, with nested tags marking each semantic region. The control flow of a single instrumented call:

  ALLREDUCE(sendbuf, recvbuf, count, op)          [libpico reference]
       |
       v
  PICO_TAG_BEGIN("init:mem-move")
    +-- allocate + copy staging buffers
  PICO_TAG_END                                     <- memory staging cost
       |
       v
  PICO_TAG_BEGIN("phase:redscat")
    | for (step = 0; step < steps; step++):
    |     PICO_TAG_BEGIN("redscat:comm", step)
    |        MPI_Sendrecv(...)                      <- per-step network
    |     PICO_TAG_END
    |     PICO_TAG_BEGIN("redscat:reduction", step)
    |        MPI_Reduce_local(...)                  <- per-step compute
    |     PICO_TAG_END
  PICO_TAG_END
       |
       v
  PICO_TAG_BEGIN("phase:allgather")
    | for (step = steps-1; step >= 0; step--):
    |     PICO_TAG_BEGIN("allgather:comm", steps-1-step)
    |        MPI_Sendrecv(...)                      <- per-step network
    |     PICO_TAG_END
  PICO_TAG_END
       |
       v
  return MPI_SUCCESS
^ Fig 6: Control flow of one tag-instrumented Allreduce (paper Fig. 5).
  Tags nest to mirror the algorithm's structure: a phase contains a
  loop of steps, and each step separates communication from reduction.
  Disabled tags compile to empty statements (< 100 ns overhead each),
  so the same source runs in profiled and unprofiled modes.

The consequence of this design is that a single end-to-end latency number is decomposed, at the granularity of individual send/recv and reduce operations, into communication vs reduction vs data-movement. That decomposition is what lets Fig. 11 show that a "communication-heavy" Allreduce is, at 8 MiB, actually only 35% communication and dominated by local data movement and reduction — a finding no black-box timer could produce.

4.2 The phases-rounds-steps decomposition model (paper Fig. 2)

The tags realize a three-level temporal hierarchy that is PICO's mental model for every collective:

   End-to-end latency              Collective decomposition
   +----------------+       +--------------------------------------+
   |                |       |  ALGORITHMIC PHASES                  |
   |                |  -->  |   [Allgather] [RedScatter] [DataMove]|
   |   Allreduce    |       +--------------------------------------+
   |   (total       |       |  ROUNDS PER PHASE                    |
   |    measured    |  -->  |   Allgather R1,R2  RedScatter R1,R2  |
   |    time)       |       +--------------------------------------+
   |                |       |  STEPS PER ROUND                     |
   |                |  -->  |   Comm R1,R2 | Reduce R1,R2 | DataMove|
   +----------------+       +--------------------------------------+
^ Fig 7: The phase->round->step hierarchy (paper Fig. 2). A "phase"
  is an algorithmic stage (e.g., the ReduceScatter half), a "round"
  is one iteration of that stage's loop, and a "step" is a sub-op
  within a round that is purely communication, purely reduction, or
  purely data movement. PICO's tags let time be summed at any level
  of this tree, which is what makes bottleneck attribution possible.

4.3 Network traffic tracer (Sec. 3.6 analysis toolkit)

A lightweight, topology-aware post-processing tool that estimates how a collective's traffic distributes across the physical fabric — critical for Dragonfly congestion analysis.

   INPUTS                                   PROCEDURE
   +-------------------------+     +-----------------------------+
   | Allocation + rank       |     | 1. classify each communi-   |
   | placement metadata      |---->|    cating rank pair by      |
   | (node list, rank map)   |     |    physical locality:       |
   +-------------------------+     |      intra-node             |
                                   |      intra-switch/group     |
   +-------------------------+     |      inter-group (global)   |
   | Topology description    |---->| 2. sum bytes per class for  |
   | (node->switch->group,   |     |    a given algorithm        |
   |  link hierarchy)        |     | 3. estimate link utilization|
   +-------------------------+     +--------------+--------------+
                                                  v
                                   +-----------------------------+
                                   | OUTPUT: per-class byte split|
                                   | e.g. Bcast doubling ->      |
                                   |   96% inter-group traffic   |
                                   | (topology-level estimate;   |
                                   |  NOT packet-accurate)       |
                                   +-----------------------------+
^ Fig 8: Network traffic tracer. It answers "which physical links
  does this algorithm stress?" without a full network simulation. The
  estimate is coarse (no adaptive routing, no congestion dynamics),
  but it is enough to explain the 2.5x Broadcast gap in Sec. 6.3.

4.4 ATLAHS trace-replay pipeline (Sec. 6.5)

To evaluate NCCL configuration choices for real AI workloads without re-running the training jobs, PICO feeds captured NCCL traces into the ATLAHS toolchain, which replays them as GOAL traces on a network simulator with the collective algorithm/protocol swapped.

  Real training run (LLaMA 7B / Mistral MoE)
       |
       v
  (1) Trace NCCL executions -> raw NCCL logs
       |   (records collective type, size, invocation order)
       v
  (2) ATLAHS: convert raw logs -> replayable GOAL traces
       |
       v
  (3) SWAP algorithm/protocol per collective
       |   (Ring/Tree/Butterfly x Simple/LL)
       |   invocation sequence + message sizes PRESERVED
       v
  (4) Replay on network simulator -> projected per-iteration runtime
       |
       v
  (5) Compare NCCL-default profile vs PICO-optimized profile
           -> projected training-time reduction
^ Fig 9: ATLAHS what-if pipeline. Because the trace fixes the sequence
  and message sizes, only the algorithm/protocol choice varies between
  runs — a controlled A/B test of configuration in simulation. This is
  offline policy evaluation: try a config on a recorded workload
  without paying for a live training run.

5. Measurement Control Flow Through One Experiment

A single benchmark cell is one (collective, message size, node count, algorithm) tuple. The flow below traces one such cell from spec to stored result.

  START (one cell: e.g. Allreduce / 512 MiB / 512 nodes / algo=X)
       |
       v
  (1) User writes/loads test.json (intent) via TUI or CLI
       |
       v
  (2) Orchestrator binds test.json to the machine's env.json
       |   -> resolves module loads, SLURM template, knob mapping
       v
  (3) Orchestrator builds pico_core with the right backend
       |   (#ifdef selects MPI vs NCCL/CUDA adapter at compile time)
       v
  (4) Orchestrator submits the job (SLURM) and waits
       |
       v
  (5) On compute nodes: pico_core initializes comm context
       |   -> applies requested controls (algorithm, protocol,
       |      transport rails) via the backend adapter
       v
  (6) WARMUP + BARRIER: internal barrier synchronization aligns
       |   all ranks before timing (Hoefler-style accurate timing)
       v
  (7) MEASURE: execute the collective over the message-size list,
       |   for the configured iteration count; if libpico is used,
       |   tag regions emit phase/round/step timings
       v
  (8) EMIT: each rank writes measurements + run metadata in the
       |   standardized format at the chosen granularity
       |   (Full | Statistics | Minimal | Summary | None)
       v
  (9) Orchestrator collects outputs into the indexed results store
       |
       v
 (10) Post-processing tools + network tracer render heatmaps,
       |   breakdowns, traffic splits
       v
  END  -> one cell in a Fig. 6 heatmap (or one Fig. 11 breakdown)
^ Fig 10: Control flow for one measured cell. The barrier at step (6)
  and the standardized emit at step (8) are what make cells comparable
  across machines; the metadata capture at (8) is what makes them
  reproducible and regression-diagnosable later.

The result-granularity choice at step (8) is a first-class knob, not an afterthought — it directly trades storage against diagnostic depth:

   Mode         What is stored                          Use
   ------------------------------------------------------------------
   Full         every measurement, every rank, every    deep per-rank
                iteration                                forensics
   Statistics   per-iteration aggregate across ranks    cross-rank
                                                         variance
   Minimal      only the max value per iteration         quick scans
   Summary      one set of aggregates over iterations    campaign roll-up
   None         stdout only, nothing stored              smoke tests
   ------------------------------------------------------------------
^ Fig 11: Result granularity modes (paper Table II). "Full" is the
  forensic mode that feeds the Fig. 11 phase breakdowns; "Summary" is
  the campaign mode that feeds the Fig. 6 heatmaps across thousands of
  cells. The user picks the point on the storage/detail curve.

6. Quantitative Results — Empirical Findings by Regime

6.1 Headline: how far are the defaults from optimal? (paper Fig. 6)

The central measurement is the ratio r = t_best / t_default per (message size, node count) cell, on each machine. r < 1 means the library's default algorithm is suboptimal; r = 0.5 means the default is 2x slower than the best available choice.

Across all three systems the defaults typically fall 30-40% short of the best alternative, and the single worst cell (LUMI, 512 MiB, 512 nodes) delivers only 20% of optimal — a 5x slowdown from an avoidable algorithm choice. The suboptimality is not random noise: it concentrates in structured regions, specifically large messages at large scale, exactly where the default heuristic's broad-portability assumptions diverge most from the platform's actual behavior.

6.2 Transport-rail tuning (paper Fig. 7)

Fixing the Allreduce algorithm to Ring at 32 nodes on Leonardo (Open MPI 4.1.6, UCX 1.15.0) and sweeping only UCX_MAX_RNDV_RAILS from 2 (default) to 4:

More network rails help only in the rendezvous (large-message) regime, where the transfer is bandwidth-bound and can be striped across rails; in the eager (small-message) regime the transfer is latency-bound and unaffected. This is a clean example of a transport knob whose value is entirely regime-conditional.

6.3 Broadcast partner-ordering vs topology (paper Fig. 9-10)

Two binomial-tree Broadcasts that are identical under the alpha-beta cost model — both complete in log2(p) rounds and move the same total volume — behave very differently on Leonardo's Dragonfly at 128 nodes (4 procs/node), because they place traffic on different physical links:

Measured execution time (log-log, 32 B to 512 MiB) is nearly identical up to 16 KiB, then diverges sharply for large messages:

Two findings stack here. First, partner ordering alone causes a 2.5x gap on this topology because distance-doubling forces 96% of the volume onto the scarce inter-group links. Second, Open MPI's own binomial Broadcast is ~an order of magnitude slower than either libpico reference — an implementation inefficiency, independent of algorithm choice, that end-to-end timing alone could never localize but that the backend-neutral reference exposes immediately.

6.4 Phase breakdown: where does the time actually go? (paper Fig. 11)

Instrumented Rabenseifner Allreduce on 8 nodes (Open MPI 4.1.6, libpico), communication as a share of total time across message sizes:

The communication share is non-monotonic: it holds at ~92-94% through 16 KiB (pure latency regime), collapses to 35% at 8 MiB as local data movement and reduction take over (the memory-bandwidth roof), then partially recovers to ~56% at 512 MiB as network bandwidth re-asserts. A high-level "this is a communication collective" label is simply wrong at 8 MiB, where two-thirds of the time is memory movement and arithmetic.

6.5 AI-workload trace replay (paper Fig. 12)

Replaying real NCCL 2.22 training traces through ATLAHS with PICO-optimized collective profiles (AllGather + ReduceScatter switched to Binomial-Butterfly + Simple; AllReduce to Tree + LL):

The optimized profile cuts per-iteration LLaMA runtime by 21% at 16 GPUs and 44% at 128 GPUs — the gain grows with scale because the Ring collectives that dominate LLaMA get relatively worse as node count rises, and the Butterfly replacement scales better. The MoE workload shows no improvement: its collectives are much larger (median 33-67 MiB vs 3-14 MiB for LLaMA), and at that size Ring is already near-optimal, so there is nothing to recover. PICO cannot improve a configuration that is already well-tuned.

7. Configuration-Regime Trade-off Tables

7.1 Algorithm selection vs message size / scale

Best choice by regime: the library default is a safe portable baseline but leaves 30-40% (up to 5x) on the table specifically in the large-message / large-scale corner. That corner is where per-platform algorithm selection pays.

7.2 NCCL protocol (Simple vs LL)

Best choice by regime: protocol optimum inverts at the small/large boundary — LL for the sub-kilobyte AllReduce, Simple for the multi-megabyte AllGather/ReduceScatter. A single fixed protocol cannot win both.

7.3 NCCL algorithm (Ring vs Tree vs Butterfly)

Best choice by regime: Ring wins for the large collectives that dominate MoE; Butterfly/Tree win for the smaller, more numerous collectives in LLaMA — and the Butterfly gain grows with scale (0% -> 21% -> 44% from MoE to L16 to L128).

7.4 Transport rails (UCX_MAX_RNDV_RAILS)

Best choice by regime: raise the rail count only for large-message, rendezvous-regime transfers; it is inert for small messages and needlessly consumes NIC resources there.

8. Bottlenecks & Insights Surfaced by the Measurements

8.1 The default-heuristic gap is structured, not random

The 30-40% (worst-case 5x) suboptimality of default algorithms concentrates in the large-message / large-scale corner of the design space. Default selection logic is built for broad portability and cannot encode per-platform topology or bandwidth characteristics, so it systematically underperforms exactly where the platform's behavior deviates most from the generic model. The insight is that the opportunity map is predictable — you know where to look for it.

8.2 Communication share is non-monotonic (the roofline shift)

The most counterintuitive measurement is that a "communication collective" is only 35% communication at 8 MiB. As message size grows, the dominant limiter migrates from network latency (small messages, ~92-94% comm) to local data movement + reduction (the memory-bandwidth roof, ~8 MiB) and then partially back to network bandwidth (very large messages, ~56% comm). Optimizing the network path alone yields diminishing returns in the mid-size regime, because the bottleneck has moved into memory. This is a roofline crossover made visible only by phase-level attribution.

8.3 Cost-model-equivalent algorithms are not performance-equivalent

Distance-halving and distance-doubling Broadcast are identical under alpha-beta (same rounds, same volume) yet differ 2.5x at 512 MiB on Dragonfly, because one lands 96% of traffic on scarce inter-group links and the other 29%. The insight: topology-aware partner ordering is a first-class performance lever that classic cost models cannot see. A tuner that reasons only about round count and byte volume will pick the wrong algorithm on a real fabric.

8.4 Implementation quality is a separate axis from algorithm choice

Open MPI's internal binomial Broadcast runs ~6x slower than a libpico reference of the same algorithm at 512 MiB. Algorithm selection and implementation efficiency are orthogonal, and only a backend-neutral reference can separate the two. Without libpico, this ~6x implementation penalty would have been misattributed to the algorithm.

8.5 Tuning value is scale-dependent and saturating

The LLaMA optimization gain climbs 21% -> 44% from 16 to 128 GPUs, while the MoE workload sees 0% because its large collectives already favor the default Ring. The insight: configuration tuning pays most at large scale with small/medium collectives, and pays nothing where the default already matches the regime — so the value of a tuner is itself a function of the workload's position in the design space.

9. Limitations of the Methodology

The most consequential limitation is that the fine-grained phase attribution (the framework's signature capability) currently exists only for the MPI reference collectives. For the *CCL libraries — the ones driving modern AI training — PICO can measure end-to-end and swap algorithms/protocols in simulation, but it cannot yet crack open a real NCCL kernel into phase/round/step timings. The diagnostic depth and the AI relevance therefore live in slightly different parts of the framework, and closing that gap (GPU-side reference collectives) is the natural next step the authors identify.

10. Note on NCCL Tuning

PICO is one of the few benchmarking papers that directly quantifies the payoff of selecting NCCL's algorithm and protocol per workload rather than trusting the default. Its ATLAHS trace-replay result is effectively an offline A/B test of configuration: replaying a recorded LLaMA trace with AllGather/ReduceScatter switched to Binomial-Butterfly + Simple and AllReduce to Tree + LL cuts projected per-iteration time by 21% at 16 GPUs and 44% at 128 GPUs, with the gain growing as scale rises. The phase/round/step attribution offers something a latency-only benchmark cannot — a decomposed signal that says whether a slow collective is bound by the network, by local reduction, or by data movement, which is exactly the kind of feedback a configuration selector needs to choose between a bandwidth-oriented protocol (Simple) and a latency-oriented one (LL). The MoE null result is equally instructive: where the default Ring already matches the regime, there is no configuration gain to be had, so any tuner's value is conditional on the workload's message-size and scale profile rather than universal.

11. Analogy

PICO is a chassis dynamometer with a full sensor harness bolted to the drivetrain, run across three different racetracks. A plain collective benchmark is a stopwatch at the finish line: it tells you the lap time and nothing else. A dyno with instrumentation tells you where in the lap the car lost time — was it the engine (network transfer), the transmission (data movement/staging), or the brakes (reduction/compute)? PICO's tag-based phase/round/step attribution is that sensor harness: it decomposes one lap time into per-corner telemetry, which is how it discovers that a "power-limited" lap is actually 65% transmission and brakes at mid-size messages (the non-monotonic communication share). The test.json/env.json split is the standardized test protocol that lets you run the same driver program on Leonardo, LUMI, and MareNostrum 5 without rewriting it per track — you describe the maneuver once and let each track's descriptor supply the local rules. The backend-neutral libpico reference is the spec car: by running the identical algorithm through a clean reference implementation, PICO reveals that Open MPI's factory Broadcast is running ~6x slow not because the racing line (algorithm) is wrong but because the factory engine is detuned. And the Dragonfly Broadcast result is the track-geometry lesson: two racing lines that look identical on paper (same distance, same number of turns) produce a 2.5x gap because one of them keeps sending the car onto the long, congested back straight (the inter-group links) 96% of the time. The stopwatch would have called them equal; only the sensor harness plus the track map explains the difference.