PICO: Performance Insights for Collective Operations

Saverio Pasqualoni, Tommaso Bonato, Lorenzo Piarulli, Torsten Hoefler, Marco Canini, Daniele De Sensi | Sapienza University of Rome / KAUST / ETH Zurich | IEEE-conference-style preprint, 2025 | Open-source: https://github.com/HLC-Lab/pico

Problem

Collective operations are cornerstones of both HPC applications and large-scale AI training/inference, and at scale their communication cost often dominates application runtime. But benchmarking them systematically and reproducibly is hard on modern heterogeneous stacks: collectives intertwine network transfer, memory movement, and reduction/computation across scale-up/scale-out fabrics, multiple communication libraries (MPI implementations and *CCL), evolving network middleware (OFI/libfabric, UCX/UCC), and time-varying runtime conditions (congestion, allocation policies, task-to-node mappings). Each collective has several algorithms, each optimal for different node-count/message-size regimes, and libraries choose among them with conservative general heuristics. Existing tools (OMB, IMB, NCCL Tests, ReproMPI, CommBench) report only end-to-end timings, lack fine-grained phase/step profiling and controlled cross-library comparison, and do not systematically record experimental conditions.

Core Insight

A benchmarking framework should move from end-to-end reporting to diagnosis — controlled experiments that explain why a collective underperforms and which algorithmic step or subsystem dominates. PICO achieves this by decoupling a portable experiment description (test.json) from platform-specific execution (env.json), pairing it with backend-neutral, instrumentable reference collectives and metadata-rich run capture, so that default algorithm/transport choices can be shown to be up to 5x slower than the best available option and the loss localized to a specific phase.

Method

PICO is built around six requirements: R1 fine-grained profiling, R2 backend-neutral references, R3 portable spec and control, R4 automation, R5 metadata-rich reproducibility, R6 extensibility across stacks. Its architecture (Fig. 3) is:

TUI / CLI  ->  test.json (control intent)  +  env.json (platform capabilities)
                              |
                    Benchmark Orchestrator  (resolves + submits jobs)
                              |
                        pico_core  (timing-critical engine, backend adapters)
                          |     \
                    libpico       standardized results + metadata store
             (plain-MPI refs,          |
              tag instrumentation)  Post-processing tools
                                    (network tracer, plotters)

pico_core runs on compute nodes, initializes the communication context, applies requested controls via uniform backend adapters (compile-time selection, e.g. #ifdef NCCL/CUDA), and times target collectives across message sizes and scales. libpico supplies plain-MPI reference collectives (adapted from Open MPI and MPICH, built on point-to-point primitives) that isolate algorithmic differences from backend effects, and hosts optional tag-based instrumentation (PICO_TAG_BEGIN/PICO_TAG_END, nestable, compiled out when disabled, < 100 ns overhead per tagged region) that attributes time to phases, rounds, and steps. Results are stored per test-point with both requested and effective configuration, at configurable granularity (Full / Statistics / Minimal / Summary / None). A network traffic tracer estimates per-algorithm link utilization across topology domains (intra-node / intra-switch / inter-group) from allocation and topology metadata.

Experimental Setup

Headline Quantitative Results

Default-vs-best collective algorithm (MPI_Allreduce sweeps):

• Typical structured shortfall: 30-40% slower than best available algorithm.
• Worst case (LUMI, 64 MiB, largest scales): r = 0.20 — only 20% of optimal, a 5x slowdown.

Backend/transport parameter (UCX_MAX_RNDV_RAILS, Ring Allreduce, Leonardo 32 nodes):

• Raising the rail limit from 2 to 4 gives up to 10% speedup on large (rendezvous-regime) messages; small (eager) messages unaffected.

Broadcast algorithm comparison (Leonardo, 128 nodes, 4 procs/node):

• Distance-doubling vs distance-halving are identical under alpha-beta modeling (both log2(p) rounds, same total volume) but diverge on the Dragonfly network.
• Network tracer (buffer size n): distance-doubling sends 122n external / 5n internal (96% inter-group); distance-halving keeps 90n intra-group and only 37n external (29% inter-group).
• At 512 MiB: distance-doubling = 757 ms vs distance-halving = 304 ms (2.5x). Open MPI's internal Binomial is ~1.9 s (nearly an order of magnitude slower, an implementation inefficiency).

Fine-grained instrumentation (Rabenseifner Allreduce, 8 nodes):

• Small-message runtime nearly constant: 10 us @ 32 B, 11 us @ 256 B, 10 us @ 2 KiB (latency-dominated).
• Communication's relative share is non-monotonic: ~95% at small sizes -> ~35% around 8 MiB -> ~56% at 64/512 MiB, with data movement and reduction absorbing the difference (roofline-style limiter shift from network latency to local memory bandwidth/compute).
• Instrumentation overhead: < 100 ns per tagged region.

End-to-end AI training (ATLAHS trace replay):

• Collective mix (L16/L128): AllGather Ring Simple and ReduceScatter Ring Simple each ~48.3% (L16) / ~45.9% (L128); Allreduce Tree LL and ReduceScatter Ring LL a minority (1-3% L16, 3-6% L128).
• Median collective sizes: Allreduce < 1 KiB; AllGather/ReduceScatter 3-6 MiB (L16), 7-14 MiB (L128); MoE 33-67 MiB.
• PICO-optimized profile (AllGather + ReduceScatter Binomial Butterfly/Simple, Allreduce Tree/LL): 21% faster per iteration on LLaMA 7B 16 GPU, 44% faster on LLaMA 7B 128 GPU.
• Mistral MoE 64 GPU: no measurable improvement (a good profile was already in use; larger collectives favor Ring).

Limitations

• The network traffic tracer gives a topology-level estimate only, not a packet-accurate simulation of congestion, adaptive routing, or protocol behavior.
• libpico currently focuses on MPI (plain-MPI implementations); supporting other communication libraries requires implementing/registering a backend signature.
• Backends lacking a feature degrade gracefully to a defined functional subset — controls are not uniformly available across stacks.
• The MoE optimized profile showed no measurable improvement (already-good profile; large collectives favor Ring).
• Communication-heavy HPC traces (e.g., 3D FFT all-to-all) were unavailable among the open-source traces, limiting AI-side evaluation.
• ATLAHS end-to-end results are projected/simulated per-iteration times, not measured on hardware.

Open Problems

• Extending backend-neutral references (libpico) beyond MPI to *CCL libraries for fully cross-stack controlled comparison.
• Upgrading topology-level traffic estimation toward packet-accurate congestion/routing/protocol modeling.
• Validating benchmark-informed collective profiles on real hardware end-to-end (not only in simulation) and across communication-heavy HPC workloads.

Note on NCCL Tuning

PICO's ATLAHS study is a direct demonstration that per-collective algorithm and protocol selection drives end-to-end training time. Replacing NCCL 2.22's default Ring/Simple choices for AllGather and ReduceScatter with Binomial Butterfly + Simple for those collectives, plus Tree + LL for the small (< 1 KiB) Allreduce, cut simulated LLaMA 7B iteration time by 21% at 16 GPUs and 44% at 128 GPUs. The paper cleanly separates the algorithm-vs-protocol axes that map onto NCCL's own knobs: Simple favors large-message bandwidth while LL (flag-based synchronization) favors small-message latency, and the winning protocol tracks the collective's message-size distribution. The MoE null result — where large collectives already favored Ring — shows the best configuration is workload- and size-dependent rather than a fixed default, exactly the regime a per-collective tuner targets.