PICO: Performance Insights for Collective Operations — Detailed Summary

Saverio Pasqualoni, Tommaso Bonato, Lorenzo Piarulli, Torsten Hoefler, Marco Canini, Daniele De Sensi | Sapienza University of Rome / KAUST / ETH Zurich | IEEE-conference-style preprint, 2025 | Open-source: https://github.com/HLC-Lab/pico

Per-section summary organized by the paper's heading structure. Each section includes paragraph-level bullet points and exact quantitative results where the paper provides them. Index terms: High performance computing, Performance analysis, Computer networks, Message passing, Software Tools.

Abstract

• Collective operations are cornerstones of both HPC applications and large-scale AI training/inference, but benchmarking them systematically and reproducibly is hard on modern heterogeneous hardware/software stacks.
• Existing suites mostly report end-to-end timings and offer limited support for controlled algorithm/configuration selection, fine-grained profiling, and capturing the runtime environment.
• The authors present PICO (Performance Insights for Collective Operations), an open-source framework that: (i) decouples portable experiment setup from platform execution; (ii) provides a backend-adaptive parameter-selection interface across MPI and NCCL; (iii) supplies plain-MPI reference collective implementations, optionally instrumentable; and (iv) records system configuration for reproducible comparisons.
• Evaluated on three major supercomputers, PICO shows default collective algorithms and transport settings can be up to 5x slower than the best available choice. It isolates topology-sensitive algorithmic choices and, through instrumentation, reveals detailed algorithmic breakdowns.
• To assess end-to-end impact, the authors replay open-source LLM training traces in the ATLAHS simulator with PICO-optimized collective profiles, achieving training-time reductions of up to 44%.

I. Introduction

• HPC systems have surpassed exascale, but the disparity between compute and data-movement capability remains a critical obstacle; processor performance improves faster than memory/interconnect, raising the relative cost of communication as systems scale.
• Collective operations are among the most communication-sensitive components of distributed-memory applications; together with point-to-point communication they form the backbone of traditional HPC and of AI training/inference. At large scale, collective cost often dominates application runtime.
• Understanding/optimizing collective performance is difficult because collectives intertwine computation, memory movement, and network transfer, making bottleneck attribution non-trivial. The problem is amplified by heterogeneous scale-up/scale-out fabrics, multiple communication libraries (MPI implementations and *CCL), evolving network stacks/APIs (OFI/libfabric, UCX/UCC), and time-varying runtime conditions (congestion, load-balancing, allocation policies, task-to-node mappings) that are hard to reproduce.
• Each collective can be implemented by multiple algorithms, each optimal for different node-count/message-size combinations; libraries pick among them with predetermined general heuristics. Performance gaps between libraries can invert depending on runtime conditions, so fair, reproducible characterization requires capturing both system context and algorithm-level behavior.
• Existing tools (OMB, NCCL Tests, Intel IMB, ReproMPI, CommBench) report end-to-end performance but only partially address modern needs: they lack fine-grained phase/step profiling, don't support straightforward controlled cross-library comparisons, and don't systematically record experimental conditions (node allocations, environment variables, stack versions, hardware).
• The authors introduce PICO, an open-source, modular, extensible framework for benchmarking and diagnosing collectives across multiple communication libraries. Its contributions: (i) step-level instrumentation attributing time to algorithmic phases; (ii) metadata-rich run capture for reproducibility and regression diagnosis; (iii) backend-neutral reference collectives isolating algorithmic differences from backend effects; and (iv) a unified experiment specification spanning diverse backends (MPI and NCCL) for portable benchmarking.
• PICO is used to analyze collectives on three supercomputers — LUMI, Leonardo, and MareNostrum 5. Default algorithm selections are found to be 30-40% slower than the best alternative, and in the worst case deliver only 0.2x of optimal (5x slower). Metadata-driven post-mortem analysis identifies why similar Broadcast algorithms diverge in scaling, with the slower variant showing a 2.5x slowdown. Fine-grained instrumentation localizes losses to different algorithmic phases (network-limited vs reduction/memory).
• To quantify end-to-end impact, the authors replay traces from LLaMA 7B and Mistral MoE in the ATLAHS simulator with PICO-informed profiles, achieving projected runtime reductions of up to 44%.

II. Motivation and Requirements

• Systematic, fair, reproducible measurement of collective performance is hard because of tight coupling between hardware heterogeneity and software-stack complexity. The authors identify three challenges (C1-C3).

C1 — Hardware heterogeneity:

• Modern platforms are multi-GPU nodes connected by high-bandwidth scale-up fabrics (NVLink, InfinityFabric, UALink, UnifiedBuffer, Scale-Up Ethernet / SUE), whose bandwidth can exceed the scale-out interconnect by up to an order of magnitude. Scale-up domains are growing fast (up to 72 GPUs in NVL72, even 384 GPUs).
• Across nodes, GPUs communicate over scale-out networks (InfiniBand, Slingshot, Ultra-Ethernet), often with tapered topologies (Dragonfly/Dragonfly+, tapered fat-trees). Effective bandwidth/latency depend on whether endpoints share a scale-up domain and, for scale-out, whether they stay within a local switch domain or traverse oversubscribed global links. These non-uniform costs violate the homogeneous-link assumptions behind many traditional collective designs and motivate hierarchical, topology-aware collectives.

C2 — Software-stack complexity:

• Collective execution spans multiple layers (MPI implementations and *CCL, plus network stacks such as OFI/libfabric and UCX), each with tunable parameters and evolving behavior across versions (Fig. 1).
• End-to-end timings mix network transfer, memory staging/movement, and reduction/computation, each depending on one or more layers, and are sensitive to time-varying runtime conditions plus version/parameter changes across layers — complicating reproducibility and regression diagnosis.
• Fig. 1 (software stack): Application -> (MPI | *CCL) -> (UCC | UCX) -> libfabric / libibverbs / sockets / shared mem -> Hardware. Legacy MPI and some *CCL access transport interfaces directly without middleware.

C3 — Measurement methodology:

• Measurement methodology is itself a source of systematic bias: accurate timing needs precise process synchronization, which is non-trivial.
• A common approach uses barriers, but they don't guarantee all processes enter the measured region simultaneously (a process may exit before others, distorting runtimes). Barrier algorithm choice matters — linear approaches (e.g., ring) are worst due to long propagation delay.
• An alternative is window-based schemes where processes agree on a future start time; these reduce barrier artifacts but shift the problem to clock synchronization and drift.

II.A State-of-the-Art Analysis

• Standard suites — OSU Micro-Benchmark (OMB), Intel MPI Benchmark (IMB), NCCL Tests — report end-to-end latency/bandwidth but aren't designed for controlled, reproducible diagnosis: limited support for portable algorithm selection/cross-library baselining (often manual env-var config), no per step/phase measurement, and no systematic recording of system info.
• NCCL Inspector (NVIDIA): a profiler-plugin providing low-overhead, always-on, per-communicator/per-collective performance and metadata logging during real distributed AI runs. Valuable but library-specific (NCCL-only) and aimed at in-workload monitoring rather than portable, backend-neutral benchmarking.
• ReproMPI: a micro-benchmarking framework focused on measurement correctness; offers multiple synchronization methods and reference algorithm implementations, but doesn't track system state, target cross-stack compatibility, or provide per step/phase measurements.
• CommBench: steps toward portability via a library-agnostic API spanning MPI and *CCL, but lacks fine-grained observability and metadata logging, and often requires writing low-level benchmarking logic (high manual effort).
• Netgauge: a modular design decoupling benchmark "patterns" from communication "modules" for extensible network/protocol measurement, but not designed for collective communications.
• Fig. 2 (end-to-end vs decomposition): end-to-end benchmarks report a single latency (left); execution actually decomposes into phases, rounds, and steps mixing communication, reduction/computation, and data movement (right). The illustrated Allreduce splits into an Allgather Phase and a RedScatter Phase with Data-Move segments, rounds per phase (R1, R2), and steps per round.

TABLE I — Qualitative coverage of requirements (checkmark = built-in; partial = external scripting/manual; X = not targeted):

II.B Design Requirements

• The central challenge is moving from end-to-end reporting to diagnosis — controlled experiments that explain why a collective underperforms and which step/subsystem dominates. The primary objective is fine-grained profiling (R1), enabled by backend-neutral instrumentable baselines (R2) and a portable experiment specification (R3); R4-R6 ensure reproducibility and practicality at scale.
• R1 (Fine-grained profiling): profile at the granularity of algorithm phases, rounds, and steps, attributing time to network transfer, memory movement/staging, and reduction/computation. Implementations delineate regions of interest via explicit annotations. Instrumentation must be optional with no measurable overhead when disabled.
• R2 (Backend-neutral references): include reference collectives (plain-MPI ported from major libraries) that isolate algorithmic differences from backend/transport effects and can be instrumented at fine boundaries, without modifying implementation-specific internal code.
• R3 (Portable spec): a portable, declarative specification (collective type, message sizes, scale, algorithm choice, backend parameters) so the same experiment runs and compares across platforms with minimal platform-specific change. It serves as a stable control interface that selects among internal algorithm choices and exposes relevant configuration parameters.
• R4 (Automation/campaigns): support large-scale campaigns with structured bookkeeping and post-processing. Complexity is front-loaded into an infrequent platform-setup step; afterward experiments run from a portable specification, with automatic knob application, job submission/management, result collection, and a standardized output schema.
• R5 (Metadata capture): capture enough metadata to reproduce/audit results and diagnose regressions — software stack versions/build IDs, selected backends/transports, environment variables, hardware details (GPU/NIC model), and allocation/mapping context (node list, rank placement). Metadata verbosity is configurable.
• R6 (Extensibility): accommodate evolving stacks via a clear extension interface for new backends (MPI or *CCL) and new collectives/algorithms while preserving a consistent workflow. When a backend lacks a feature, the framework degrades gracefully to a well-defined functional subset.

II.C Workflows and Usability

• The requirements address three user types: (i) researchers/algorithm developers need backend-neutral baselines and fine-grained instrumentation (R1-R3) with reproducibility (R5); (ii) application developers/users need automated exploration and structured result management (R4) plus metadata to interpret variability (R5-R6); (iii) system administrators need a repeatable workflow for regression testing across upgrades/config changes (R4-R5).

III. Architecture

• PICO's core principle: decouple what to run (a portable experiment description) from how to run it on a given platform (reusable environment descriptors), producing outputs that are both comparable and diagnosable.
• Fig. 3 (workflow pipeline): test and environment specs are defined via descriptive files — test.json (c) and env.json (d); a Benchmark Orchestrator (E) sets up the environment and launches instances of the benchmarking core pico_core (F), which executes test instances and stores performance data and metadata (H) in a structured format. Post-processing / visualization tools (I) analyze the data. The core optionally uses a library, libpico (G), of reference implementations and instrumentation primitives. Front-ends: TUI (A) and CLI (B).

III.A Experiment Specification and Control Plane

• PICO's control plane provides a stable, portable interface for defining experiments and expressing backend control (algorithm choice + parameters) through declarative descriptors, translating intent into backend-specific mechanisms without per-experiment scripting. A platform descriptor (env.json) records platform capabilities and control mappings; a portable test descriptor (test.json) records experiment intent. Together they make experiments executable and comparable across platforms (enabling R4).
• Setup complexity is front-loaded into the infrequent authoring of env.json, which defines available communication stacks (MPI implementations, selected *CCL), module/environment setup, scheduler/launcher templates (e.g., SLURM defaults), and backend-specific control mappings (which algorithm selectors and transport knobs are exposed and how to apply them). Users then write a backend-agnostic test.json encoding control intent ("use algorithm X", "set parameter Y") that PICO resolves against env.json — not cluster-dependent scripts.
• PICO provides a TUI (A) and a CLI (B) as front-ends to the same specification model. The TUI (Fig. 4) guides discovery of available libraries/parameters, applies defaults and validation, and outputs a self-contained test.json.
• Fig. 4 (TUI): interactive experiment specification exposing backends, collectives, algorithms, and control parameters for the current platform. Fields include Environment (e.g., leonardo), Partition, Number of Nodes, Test Time, Exclude Nodes, Data Type (int32), Compress Data, Buffer Sizes, Output Level, and a collectives panel (Allgather, Allreduce, Alltoall, Broadcast, Gather, Reduce, ReduceScatter, Scatter); a second screen shows library selection (Open MPI 4.1.1, NCCL 2.20.1) and GPU tasks-per-node.

III.B Execution Engine and Backend Adapters

• PICO core (F) runs on compute nodes inside the allocated job and handles the timing-critical portion: initializing the communication context, applying requested controls when supported, executing target collective(s) over specified message sizes and scales, and emitting measurements/metadata in the standardized format, using PICO's internal barrier synchronization for timing alignment.
• PICO supports heterogeneous stacks (R6) via a uniform backend-adapter interface, with backend availability selected at compile time (e.g., #ifdef NCCL/CUDA for GPU collectives). Each adapter implements (i) context initialization, (ii) mapping of abstract controls from test.json to backend-specific knobs when exposed, and (iii) collective execution and timing.

III.C Backend-Neutral Baselines

• libpico (G) is a user-space library of reference collective implementations. In the current version it focuses on MPI: plain-MPI implementations built on point-to-point primitives, adapted from Open MPI and MPICH, so algorithmic choices can be evaluated without relying on library-internal collectives — enabling controlled, backend-independent comparison under identical conditions.
• libpico is extensible: developers can write and test new algorithms, and can add support for other communication libraries by implementing the corresponding backend signature and registering the implementation within pico_core.
• Fig. 5 (instrumented Allreduce pseudo-code): nested PICO_TAG_BEGIN/PICO_TAG_END markers annotate an init:mem-move region, a phase:redscat reduce-scatter phase with per-step redscat:comm (MPI_Sendrecv) and redscat:reduction (MPI_Reduce_local) regions, and a phase:allgather phase with per-step allgather:comm (MPI_Sendrecv) regions.

III.D Tag-Based Instrumentation for Fine-Grained Attribution

• To move beyond aggregate timings, PICO supports optional tag-based instrumentation (R1) for libpico collectives (including user-defined ones). Tags delineate semantically meaningful regions — data staging, algorithmic phases, per-step communication/reduction — enabling attribution of where time is spent, without modifying vendor stacks.
• Instrumentation uses lightweight macros (PICO_TAG_BEGIN, PICO_TAG_END), usable flat or nested to capture hierarchical structure. Probes are optional and user-controlled. When enabled, timings for tagged regions use the same structured output model; when disabled, the macros compile out to empty statements, leaving standard benchmarking behavior unaffected. Added cost per timing invocation is negligible — less than 100 ns per tagged region.

III.E Standardized Results and Metadata Capture

• To satisfy R5, PICO emits performance measurements and execution context in a standardized, human-readable output for large campaigns and post-hoc diagnosis. Each campaign stores per-test measurements under a run directory, snapshots the resolved experiment spec (including the effective control settings actually applied), and maintains a lightweight index for automated traversal/aggregation/comparison.
• Each test point (collective type, message size, scale, backend, control settings) is a separate record with timing data and identifiers; the backend-agnostic schema encodes both the requested configuration (from test.json) and the effective configuration after platform resolution (via env.json), preserving comparability even when controls are unsupported or mapped differently across stacks.
• PICO records run context alongside performance data: stack versions/build IDs, selected backends/transports, environment variables and tuning knobs, hardware characteristics (GPU/NIC model), and allocation/mapping context (node list, rank placement). Metadata capture supports configurable verbosity.

TABLE II — Result data granularity modes:

III.F Analysis and Diagnosis Toolkit

• To help interpret performance on tapered, non-uniform interconnects, PICO provides a lightweight network traffic tracer estimating how traffic distributes across topology domains (e.g., Dragonfly groups). Inputs: allocation and rank-placement metadata per run (R5) plus a topology description (node-to-switch-group membership, link hierarchy). It categorizes communicating rank pairs (intra-node, intra-switch, inter-group) and estimates per-algorithm link utilization, letting users correlate observed performance with expected congestion. It provides a topology-level estimate only — not a packet-accurate simulation of congestion, adaptive routing, or protocol behavior.
• For amortized usability (R4), PICO provides scripts that generate standard plots directly from the result schema — heatmaps (message size vs scale), line plots, box/bar summaries across algorithms or backends — keeping visualization consistent across runs and integrable into automated tuning/regression pipelines.

IV. Evaluation

• PICO is evaluated through case studies validating the Section II requirements, answering four questions: (1) how often does library algorithm selection deviate from the best choice, and can PICO orchestrate controlled tuning? (IV.A); (2) can libpico references isolate algorithmic effects and explain cross-platform differences? (IV.B); (3) can instrumentation reveal actionable bottlenecks hidden under aggregate timings? (IV.C); (4) do benchmark-informed choices translate into application-level effects under realistic traces? (IV.D).

IV.A Collective Tuning

• Systematic sweeps of MPI_Allreduce were run on three European supercomputers — Leonardo (Open MPI 4.1.6), LUMI (Cray MPICH 8.1.29), and MareNostrum 5 (Open MPI 4.1.5) — varying only the exposed collective algorithm and holding all else fixed. Outcomes use the best-to-default latency ratio r = t_best / t_def, where t_def is the median runtime under the backend default and t_best the minimum median runtime among non-default algorithms for the same test point. r < 1 means the default is suboptimal; r > 1 means the default is best.
• Fig. 6 (heatmaps, message size vs # nodes): across all three systems there are structured regions (often at larger scales / specific message sizes) where defaults fall short of the best alternative by roughly 30-40%, and in the most pronounced case (LUMI, 64 MiB, largest scales) the default achieves only 20% of optimal performance (r = 0.20, a 5x slowdown). Leonardo spans 32-2048 nodes, LUMI up to 1024, MareNostrum 8-64.
• Default selection heuristics are engineered to be conservative and broadly portable, so they may miss platform-specific characteristics — motivating systematic tuning. PICO's outputs can serve as the empirical basis: Open MPI supports overriding algorithm selection via coll_tuned dynamic decision files, and analogous mechanisms exist across other MPI implementations and *CCL libraries (config files or environment variables).
• Sub-optimality is not limited to algorithm choice; backend/transport parameters can change performance dramatically. Fixing MPI_Allreduce to Ring on Leonardo at 32 nodes and varying only UCX_MAX_RNDV_RAILS (a UCX parameter capping the number of network rails used by the rendezvous protocol for large messages):
• Fig. 7: with times normalized to the default UCX_MAX_RNDV_RAILS=2, raising the rail limit to 4 reduces runtime up to 10% for large messages in the rendezvous regime (mainly 64 MiB and 512 MiB), while small messages (eager regime) are largely unaffected. UCX version 1.15.0.
• Two practical points follow: meaningful tuning requires handling both algorithm selection and backend configuration (R3) — even a strong algorithm can look weak under an unfavorable transport setting — and reproducibility/regression diagnosis depend on recording the effective configuration used in each run (R5). PICO captures both requested and effective settings, enabling controlled single-knob A/B tests.

IV.B Algorithmic Differences

• Performance models (analytic guidance on step count, communication volume, reduction cost) are essential, but modern hierarchical systems exhibit strong topology-dependent bottlenecks. The point is not to argue against modeling but to show that even when two algorithms are equivalent under a cost model, their performance can differ substantially across topologies/allocations.
• Two Broadcast algorithms are compared on Leonardo: distance-doubling binomial tree (Open MPI's binomial broadcast) and distance-halving binomial tree (MPICH's binomial tree). Under classic alpha-beta modeling they appear indistinguishable — both complete in log2(p) rounds and transmit the same total volume. But distance-doubling keeps communication local early and defers longer-distance exchanges to later rounds, whereas distance-halving performs longer-distance exchanges earlier and becomes more local later — shifting how many exchanges traverse local vs global links at each step, and thus where congestion concentrates.
• Fig. 8 (schedules): (a) distance-halving vs (b) distance-doubling; both log2(p) rounds, same total volume, differing in how communication distance evolves. Distance-halving maximizes locality at later rounds, when overall communication volume is greater.
• Fig. 9 (network tracer, 128-node Leonardo Dragonfly allocation, buffer size n): distance-doubling sends almost all volume inter-group (external 122n, internal 5n, total 127n — 96% external), whereas distance-halving keeps 90n intra-group and reduces inter-group traffic to 37n (total 127n — only 29% external). This arises from the interaction of rank placement and the algorithm's communication schedule.
• Fig. 10 (measured MPI_Bcast, 128 nodes, 4 processes/node, log-log): curves are nearly identical for small messages (up to 16 KiB) but diverge sharply once large-message transfers dominate. At 512 MiB, distance-doubling is 2.5x slower at 757 ms vs 304 ms for distance-halving. Open MPI's internal Binomial algorithm is almost an order of magnitude slower at 1.9 s, indicating implementation inefficiency independent of the algorithm chosen.
• Cost models capture step/volume trade-offs but can't distinguish algorithms equivalent in those metrics unless topology/placement is modeled explicitly. PICO complements modeling via backend-neutral comparison (R2) and placement-aware diagnosis from rich metadata (R5), providing structural evidence (tracer) plus measured performance to justify an alternative schedule.

IV.C Fine-Grained Instrumentation

• Collectives comprise multiple algorithmic steps, each stressing different hardware resources (NIC/fabric, caches/DRAM, CPU/GPU arithmetic units). The different scaling of these resources with message size and node count means aggregate end-to-end timing can hide where inefficiencies originate.
• An instrumented run of the libpico reference Rabenseifner Allreduce on an 8-node Leonardo allocation (Open MPI 4.1.6) breaks runtime into tagged components: network communication, reduction computation, and intra-node data movement (staging/copies), plus residual "Other." Fig. 11a shows absolute runtime; Fig. 11b shows relative shares.
• For small messages (up to 128 KiB) the communication curve is nearly indistinguishable from the aggregate, and for sizes up to 2 KiB total runtime is nearly constant (10 us at 32 B, 11 us at 256 B, 10 us at 2 KiB), consistent with a latency-dominated regime where fixed network startup outweighs bandwidth.
• The tagged breakdown shows the "communication-dominated" interpretation doesn't hold uniformly: after 128 KiB, communication's relative cost drops sharply from nearly 95% to 35% (around 8 MiB) before rising again to 56% at 64 MiB and 512 MiB. This non-monotonic trend means the scaling driver changes with message size: once per-message latency is amortized, runtime is no longer governed by network transfer alone — the missing fraction is absorbed by intra-node data movement (staging/copies) and reduction (more bytes combined per step).
• The behavior is interpreted via roofline-model intuition: performance is limited by whichever resource is most constraining. Communication is latency-limited for small messages and bandwidth-limited for large ones, while staging/reduction are limited by local memory bandwidth and compute throughput. As message size grows, the dominant limiter shifts from network latency to local data movement and reduction (moving onto a memory-bandwidth roof), and at very large messages network bandwidth dominates but data movement and reduction still cap end-to-end gains.
• Crossover points and shift magnitudes depend on the machine's hardware, so the same end-to-end Allreduce curve can hide different bottlenecks on different systems — reinforcing the need for fine-grained profiling. PICO's instrumentation (R1) on backend-neutral references (R2) exposes these hidden bottleneck shifts.

Fig. 11b — approximate relative breakdown of instrumented Rabenseifner Allreduce (best-effort reads from the stacked chart):

IV.D Simulation Results with ATLAHS

• PICO is evaluated on real AI workloads via trace replay using the ATLAHS toolchain, which traces NCCL executions and replays them as GOAL traces on network simulators. ATLAHS can generate different replayable traces from the same raw NCCL logs, swapping collective algorithm and protocol while preserving the original invocation sequence and message sizes — enabling controlled what-if analysis without re-running the workload, and testing algorithms not yet in NCCL by adding new translation units in the simulator toolchain.
• Application traces were chosen toward AI workloads (large, repeated collectives where algorithm/protocol/transport decisions produce observable end-to-end effects). Similar sensitivity can arise in communication-heavy HPC apps (e.g., 3D FFTs and their all-to-all exchanges), but such traces were not available among the open-source traces considered.
• Traces were collected with NCCL 2.22, which provides Ring and Tree implementations for AllReduce (Tree = Distance-Halving Reduce followed by Distance-Doubling Broadcast) but only a Ring algorithm for ReduceScatter and AllGather. Newer NCCL releases have since added a Binomial Butterfly (PAT). For each invocation, both the collective algorithm and protocol were recorded, alongside collective size, communicator details, and GPU streams. The algorithm controls overall structure; the protocol controls the low-level transfer/synchronization strategy. Available NCCL protocols are Simple (favors large-message bandwidth) and LL / Low-Latency (reduces small-message latency via flag-based synchronization).
• Chosen workloads: LLaMA 7B on 16 and 128 GPUs (traces L16, L128) and a Mistral MoE (Mixture-of-Experts, 8x8B) on 64 GPUs (trace MoE).
• Fig. 12 (left) — collective mix: for both L16 and L128, the majority are AllGather Ring Simple (L16 48.3%, L128 45.9%) and ReduceScatter Ring Simple (L16 48.3%, L128 45.9%), with Allreduce Tree LL and ReduceScatter Ring LL a small minority (L16 1-3%, L128 3-6%). The MoE trace has fewer invocations, almost equally split among Allreduce Tree LL, ReduceScatter Ring Simple, and Allgather Ring Simple.
• Fig. 12 (center) — size distribution: Allreduce invocations were all small (< 1 KiB); AllGather/ReduceScatter had a median of 3-6 MiB (L16) and 7-14 MiB (L128); MoE was significantly larger at 33-67 MiB. Point-to-point sends were ignored to focus on collectives.
• Using PICO, candidate collective profiles (algorithm/protocol choices) were identified for the observed communicator/size distributions. For L16 and L128 the profile was AllGather and ReduceScatter Binomial Butterfly with Simple protocol, plus Allreduce Tree with LL protocol.
• Fig. 12 (right) — simulated per-iteration runtimes: the PICO-optimized profiles improve over native NCCL by 21% on L16 and 44% on L128. The MoE optimized profile showed no measurable improvement, indicating a good profile was already in use when the trace was instrumented — likely because MoE has larger collectives on average, which tend to perform better with Ring implementations. Suboptimal alternative profiles were replayed alongside the optimal one, confirming runtime variation across algorithm/protocol choices.
• Thanks to PICO's extensible design (R6), the authors could evaluate NCCL algorithms and run a systematic campaign (R4) across both algorithm selection and protocol configuration (R3), observing real-world end-to-end improvements.

V. Conclusions

• PICO is a lightweight, extensible framework for benchmarking collective communication across heterogeneous HPC and AI systems, integrating fine-grained profiling, rich metadata collection, and automated orchestration for reproducible, system-aware analysis. Its modular architecture enables portable comparisons across MPI, *CCL, and user-defined algorithms, with integrated post-processing for high-level summaries and detailed phase breakdowns.
• Case studies demonstrated: revealing suboptimal default algorithm selection and guiding library tuning; highlighting subtle trade-offs between closely related algorithms; quantifying backend-parameter impact; and identifying hidden bottlenecks via reference implementations and instrumentation. PICO-tuned configurations translated to real end-to-end improvements.
• The authors aim for PICO to become a foundational tool for reproducible collective-communication performance analysis in next-generation systems, useful to algorithm designers, application developers, and system administrators.

VI. Acknowledgments

• Supported by EU Horizon Europe grant 101175702 (NET4EXA); Sapienza University grants ADAGIO and D2QNeT; a Microsoft Azure research grant; the FastTrackAI project at the Singapore-ETH Centre (NRF Singapore, MDDI AI Visiting Professorship AIVP-2025-005). Compute access via ISCRA (LEONARDO, EuroHPC JU / CINECA) and EuroHPC JU / LUMI consortium / BSC (LUMI at CSC Finland, MareNostrum 5 at Barcelona Supercomputing Center).

Consolidated Quantitative Record

Limitations and Future Work (as stated)

• The network traffic tracer gives a topology-level estimate only, not a packet-accurate simulation of congestion, adaptive routing, or protocol behavior.
• libpico currently focuses on MPI (plain-MPI implementations); extending to other communication libraries requires implementing/registering a backend signature (framed as an extensibility path).
• Backends lacking a feature degrade gracefully to a defined functional subset — not all controls are uniformly available.
• The MoE optimized profile showed no measurable improvement (a good profile was already in use; larger collectives favor Ring).
• Communication-heavy HPC traces (e.g., 3D FFT all-to-all) were not available among the open-source traces considered, limiting evaluation to AI workloads.
• ATLAHS results are projected/simulated per-iteration times, not measured on hardware.

Note on NCCL Tuning

PICO's ATLAHS study is a direct demonstration that per-collective algorithm and protocol selection drives end-to-end training time. Replacing NCCL 2.22's default choices — Ring/Simple for AllGather and ReduceScatter — with a profile of Binomial Butterfly + Simple for those two collectives and Tree + LL for the small (<1 KiB) Allreduce cut simulated LLaMA 7B iteration time by 21% at 16 GPUs and 44% at 128 GPUs. The paper also isolates the algorithm-vs-protocol distinction that maps onto NCCL's own knobs: Simple favors large-message bandwidth while LL (flag-based synchronization) favors small-message latency, and the winning protocol tracks the collective's message-size distribution. The MoE null result (where large collectives already favored Ring) is the cautionary counterpart: the best configuration is workload- and size-dependent, not a fixed default, which is exactly the regime a per-collective tuner targets.