A Compilation-Based Approach to Performant Reduction and Redistribution Collective Communication Algorithms — Detailed Summary

Andreas Jocksch (ETH Zurich / CSCS), C. Nicole Avans, Riley Shipley, Anthony Skjellum (Tennessee Technological University) | The International Journal of High Performance Computing Applications (IJHPCA), Special Issue Paper, 2026, Vol. 40(2), pp. 219–239 | DOI: 10.1177/10943420251363423

Per-section summary organized by the paper's headings. Each section includes paragraph-level bullet points and exact quantitative results where the paper provides them. This is an expanded version of the authors' PPAM 2024 conference paper "Flexible algorithms for persistent MPI allreduce communication" (LNCS 15579, pp. 273–286). Code: github.com/eth-cscs/ext_mpi_collectives. Funded by NSF Grants 2405142 and 2412182. Benchmarked on the CSCS Alps supercomputer.

Abstract

• Supercomputers keep growing in parallelism, especially cores per node; the authors exploit high core counts in a flexible allreduce built specifically for shared-memory communication.
• The algorithm is a reduce_scatter stage followed by an allgather stage, OR a reduce stage followed by a broadcast stage, with different (multi-radix) factors applicable at each stage; reduce and broadcast are also standalone.
• Barriers are integrated into the algorithm using counters that track progress.
• The implementation splits into a setup phase (run once per parameter set, determining the algorithm) and an execution phase — matching two interfaces: the MPI 4.0 persistent collective interface and a blocking interface where setup happens lazily on the first call.
• Speedups of half an order of magnitude over persistent and blocking allreduce in MPICH and Open MPI on a dual-socket AMD EPYC node, and almost an order of magnitude on a four-socket NVIDIA Grace-Hopper node.
• Reductions run on vectors in both CPU and GPU memory; good multi-node performance; CP2K sped up 2.5% on a standard benchmark; for long messages the implementation matches NCCL.

1. Introduction

• As core counts per node grow on supercomputers and PCs, intra-node collective communication becomes central to efficient resource use. The operations reduce, broadcast, reduce_scatter, allgather, and allreduce are optimised both standalone and as components of inter-node allreduce, building on the authors' prior inter-node work (Jocksch et al., 2021).
• Three existing algorithms are improved (cyclic shift, recursive exchange, tree) and one new algorithm introduced (cyclic copy-in reduction). The first three use flexible factors (multi-radix) instead of a constant radix of two.
• Schemes and benchmarks are presented; cyclic copy-in is presented without benchmarks for completeness. Algorithms are benchmarked mostly for the complex allreduce collective; reduce_scatter and allgather are components of allreduce and not evaluated standalone, whereas on-node reduce and broadcast are.
• The inter-node work extends Jocksch et al. (2021); whereas the earlier work focused on cyclic shift, this contribution focuses on recursive exchange, with an algorithmic scheme for non-trivial factors; again only the complete allreduce is benchmarked, not its components.
• The optimisations suit applications where identical-parameter communication patterns repeat often — repeated allreduce in Neko (Karp et al., 2022) and in distributed AI training — motivating a one-time complex setup in initialisation that is then amortised. This is persistent collective communication (Holmes et al., 2021), an MPI 4.0 feature available in MPICH, Open MPI, and MPC.
• Alternatively, for applications like CP2K (Kühne et al., 2020), classical blocking collectives are supported with lazy initialisation (setup deferred to first call); setup memory is discarded when the communicator is released. Initialised algorithms are cached in 'wisdom' files for reuse.
• Key intra-node features: shared memory, a combination of reduction operations and synchronisation barriers, and flexible factors. Of the four shared-memory algorithms, recursive exchange and tree support non-commutative operations; cyclic shift and cyclic copy-in are commutative-only. Allreduce via recursive exchange / cyclic shift is reduce_scatter then allgather, with an option to recurse an allreduce in between. CPU shared buffers use XPMEM (Hashmi et al., 2018); GPU memory is shared via CUDA inter-process communication.
• Inter-node allreduce recursive exchange runs in three phases: reduce_scatter, a nested allreduce, and allgather (as in the cyclic-shift version of Bruck and Ho, 1993). Inter-node allreduce can run atop intra-node reduce_scatter/allgather ((1) intra reduce_scatter, (2) inter allreduce, (3) intra allgather); for performance it is alternatively built on intra-node reduce/broadcast via the tree algorithm ((1) intra reduce, (2) inter allreduce, (3) intra broadcast). Multi-socket / NUMA nodes are handled so intra-node algorithms work unmodified.
• Roadmap: recursive exchange (intra- and inter-node) plus cyclic shift and tree (intra-node); intra/inter-node interplay and parametrisation; CPU and GPU implementations with single-/multi-node benchmarks vs MPICH, Open MPI, NCCL; and CP2K evaluation using optimised allreduce and reduce_scatter_block.

2. Newly Optimised Algorithms

• Four algorithms optimise collectives for high-core-count, multi-socket nodes, leveraging shared-memory communication, flexible factorisations, and synchronisation barriers for both intra- and inter-node communication.

2.1 Recursive exchange

• A butterfly-type algorithm for non-commutative operators with flexible factors generalising radix two; it maps naturally onto multi-socket nodes when process placement minimises inter-socket transfer. Figure 1 shows data flow for eight tasks ordered for a two-socket node (t0–t3 on socket 0, t4–t7 on socket 1). Allreduce = reduce_scatter stage + allgather stage; in the example reduce_scatter uses factors four then two, allgather uses two then four.

2.1.1 Intra-node communication

• Reduce_scatter stage + allgather stage, repeated per factor in the factorisation of the task count; the optional intermediate allreduce was found not advantageous intra-node. Factors are flexible. Barriers are integrated as counters (flags set/read individually) so each step's data is valid when used.
• Figure 2 shows recursive exchange for the same parameters as Figure 1. All tasks start with data vectors of 8× the task number. Reduce_scatter first sums 0–3 and 4–7 (factor four) then factor two; after the second step the matrix diagonal everywhere holds the sum of 0–7 = 28. Allgather then reconnects vectors so every process owns a full copy. The factorisation is written {−4, −2, 2, 4} (negative = reduce_scatter, positive = allgather).
• The message vector is split into parts (coloured rows), which need not be equal. Two partitioning heuristics: equal-size smaller messages, or a fixed size (typically cache-line) for all parts except the last. With one part non-zero and the rest zero, the algorithm becomes a tree reduce/broadcast with a flexible number of leaves.

2.1.2 Inter-node communication

• Assumes a fully connected network with possibly multiple ports per node. Inter-node allreduce runs in three phases: reduce_scatter, a nested allreduce, then allgather. Recursive exchange is illustrated with one process per node.
• Figure 3 shows the scheme for four nodes, one task per node, factorisation {−2, 2, 2}: an incomplete reduce_scatter {−2}, an allreduce implemented as allgather-with-reduction {2}, and an allgather {2} — a recursive allreduce with reduce_scatter and allgather at the ends and an embedded allreduce.
• In Figure 3, each message is two boxes; per column the top two boxes are sendbuf, middle two recvbuf, bottom two tempbuf. The right side is a script for node zero (close to what the setup phase generates internally), with keywords STEP, FRAC (memory chunk), RECVFROM / SENDTO (non-blocking recv/send with the partner's rank; scheduled then a waitall), and REDUCEFROM (reduction, with two numbers giving the pre- and post-reduction operand lines, counted 0–5).
• Reiterated: overall size split into two equal parts. Incomplete reduce_scatter halves each process's portion ({−2}); the allreduce step ({2}) is allgather-with-reduction (redundant computation on the nodes); the final allgather ({2}) restores full size. It calls MPI_Irecv, MPI_Isend, MPI_Waitall.
• For multiple tasks per node, the algorithm splits into intra- and inter-node parts: shared-memory reduce_scatter, inter-node reduce_scatter, inter-node allgather, shared-memory allgather. The Figure 3 sendbuf/recvbuf become temporary buffers between intra- and inter-node steps because the memory layout differs and requires rearrangement; consequently the cyclic shift of the message vector (Bruck-Ho / Jocksch 2021) can be done simultaneously, and recursive exchange has no preference over cyclic shift here. With multiple sockets, inter-node communication runs in parallel between sockets and one shared-memory segment is allocated per socket.

2.1.3 Parametrisation

• Recursive exchange admits many factor sets; these apply to both reduce_scatter and allgather (allgather is the reversed reduce_scatter). Valid factorisations are any set whose product equals the task count (e.g., for eight: {2,2,2}, {2,4}, {4,2}, {8}). Without proof, for 2ⁿ tasks there are 2ⁿ⁻¹ options.
• Table 1 — combinatorial number of factorisations:

• Constraints when optimising: intra-node, do not split short messages below the cache-line size (avoids cache devalidation across tasks), and per-step message size should not exceed cache size (avoids misses); integrated-barrier complexity also matters. Inter-node, the number of ports enters the factorisation (each factor = ports + 1); usefully-used ports can be message-size dependent, and task order matters on bandwidth/latency-inhomogeneous networks.

2.1.4 In-place versus out-of-place

• Intra-node uses the same algorithms for out-of-place and in-place; the only difference is that the first reduction copies from send to receive buffer, after which all reads are from sendbuf and writes to recvbuf. When combined with inter-node communication, recvbuf data is copied to tempbuf, rearranged, and copied back; for performance a different algorithm may serve inter-node (Section 2.3).

2.2 Cyclic shift

• Works only for commutative operators (cyclical data movement). Implemented by Bruck and Ho (1993) inter-node and by Li et al. (2014) intra-node (dissemination variant). Recursive exchange with one task per socket needs fewer memory copies than cyclic shift. Figure 4 shows seven tasks, six steps all factor two — first three reduce_scatter, last three allgather, with shift direction reversing between stages.

2.2.1 Intra-node communication

• Figure 5 shows the scheme for seven tasks. Unlike Bruck-Ho, no cyclic shift of message vectors is applied before/after communication (no need to coalesce). Messages split into pieces no smaller than the cache line. Key feature: the product of factors for a stage may exceed the task count, e.g. {2,2,2} for 7 tasks, so rows are not simply halved/doubled in the first/last steps.
• Implemented as reduce_scatter then allgather with flexible factors and integrated barriers. With no data to reduce, a stage becomes a dissemination barrier (Hoefler et al., 2006; Nanjegowda et al., 2009). Reduction resembles the OpenMP counterpart; allgather is the reversal of reduce_scatter.
• Recursive application is used for multi-socket nodes and small messages. For multi-socket: cyclic reduce_scatter on each socket, then reduce_scatter between sockets, then allgather between sockets, then allgather on-socket. For short messages, the outer cyclic shift uses fractions ≥ cache line and the inner shift is applied to the smallest messages first.

2.2.2 Parametrisation

• Many factorisations exist; for eight tasks there are ten: {2,2,2}, {2,3,2}, {2,4}, {3,2,2}, {3,3}, {4,2}, {5,2}, {6,2}, {7,2}, {8} — superset of the recursive-exchange options. Factor-set order can be reversed for reduce_scatter vs allgather. Unlike recursive exchange there is no obvious task-count / option-count correlation. Without an intermediate allreduce, the total number of complete-allreduce options is the square of Table 1's entries (reduce_scatter and allgather may use different factors); the allreduce version of Section 2.1.2 adds more.

2.3 Tree algorithm

• Recursive exchange's data layout is good intra-node but needs an extra memory copy before and after inter-node communication. So on-node allreduce is treated as a reduce then broadcast (same operation count). Figure 6 shows the alternative layout that sacrifices data locality but avoids extra copies, with parameters identical to Figure 2. Figure 7 shows the last step with two sockets: half the data reduced to task zero, the other half to task four (only reduce shown; broadcast is analogous in the opposite direction). Parametrisation is as for recursive exchange, though optimal parameters may differ.

2.4 Cyclic copy-in reduction

• For large messages needing a copy-in phase, a combined copy-in-and-reduction step (Figure 8): pieces of different data vectors are copied in and reduced cyclically. Across four steps, tasks' pieces are added cyclically (data of tasks 3,0,1,2 → 0,1,2,3; then 2,3,0,1; then 1,2,3,0), with barrier flags set/checked after each step. Barrier complexity is quadratic (N²) here versus N log N for dissemination, so it is efficient to combine with consecutive shared-memory reductions; multiple task groups do independent copy-in reductions followed by a shared-memory reduction. Group size is a trade-off. Presented without benchmarks for completeness.

2.5 Other collective operations

• The shared-memory allreduce algorithms apply directly to reduce_scatter and allgatherv (phases of allreduce). The banded data layout (Figures 2, 5) needs a data copy to/from the reduce_scatter/allgatherv buffers; reduce_scatter needs a temporary buffer the size of the send buffer — except when radix equals the task count, where data is copied and reduced directly in every task's receive buffer. Broadcast and reduce are obtained by replacing reduce_scatter with scatter and allgather with gather; alternatively the tree allreduce already consists of reduce and broadcast phases.

3. Hardware Mapping

• Algorithmic schemes are mapped onto (hierarchical) multi-socket hardware to maximise data locality; GPU-memory reductions minimise the number of kernel invocations.

3.1 Barrier implementation

• Each task's shared-memory segment stores barrier flag counters. Listing 1 is a dissemination barrier: the shmem array holds pointers to different processes' flags with cyclic indices (index 0 is always the own flag), which is incremented each step and compared to others; memory_fence_store()/memory_fence_load() are memory barriers where the architecture requires them.

3.2 Translation of the algorithm

• The implementation processes collective parameters and generates executable bytecode in several steps; one intermediate step generates and compiles an assembler code (Listing 2) representing allreduce between four nodes (one socket each) with four tasks per node. Colours follow Table 2.
• The example uses factors {−2, −2, 2, 2} for the on-node reduce_scatter–allgather approach and {−2, −2, 2, 2} between nodes, with 1024 elements of 8 bytes (more performant factorisations exist). The code is for task zero; MEMORY_FENCE_STORE / MEMORY_FENCE_LOAD are fences for multi-core execution.
• Table 2 — colour code of assembler listings:

• Serial copy/reduce use SMEMCPY / SREDUCE, where SENDBUF, RECVBUF, SHMEM are the node's send, receive, and temporary shared buffers; bracket numbers are task numbers, added numbers are buffer offsets, the last number is the operation size. SET_NODE_BARRIER increments the task's own flag; WAIT_NODE_BARRIER tests others' flags. ISEND / IRECV / WAITALL map to MPI_Isend/Irecv/Waitall (LOCMEM holds the MPI request handle). REDUCE runs reduction operations in parallel while other tasks reduce neighbouring chunks, so explicit SOCKET_BARRIER and SOCKET_BSMALL memory barriers (both dissemination barriers over the node here) are required. RETURN ends execution. The assembler is translated to bytecode.
• The bytecode covers memory copy/reduce, barrier flag increment/read, MPI point-to-point, and CUDA kernel calls. Kernel parameters live in GPU memory, passed as a pointer to the single copy/reduce kernel used throughout; consecutive operations are fused into one kernel invocation, and any barrier, send, or receive breaks the fusion.
• Assembler for maximum CPU vs GPU performance differs in factors, recursion, pipelining, and barrier-flag handling: on CPU the flag check is immediately before each copy/reduce; on GPU all flags are checked together before a batch. Listing 3 is a GPU-optimised on-node allreduce for task 0 with factors {−4, 4}, executing all copy/reduce in one CUDA kernel; memory fences are unnecessary for the GPU allreduce.

3.3 Mapping on the GPU

• Send/receive buffers are not visible across tasks by default; visibility comes via XPMEM (CPU) or the CUDA runtime (GPU), but those library calls add overhead, so for small messages it is more efficient to copy to/from a temporary buffer.
• Copy, reduce, and send/receive can run on CPU or GPU; the architecture supports CPU access to GPU memory and vice versa (Grace-Hopper specific, Fusco et al., 2024). CPU code has lower latency; GPU code has higher bandwidth. The translator puts consecutive copies/reductions into one kernel where possible (kernels must be embarrassingly parallel over CUDA threads), else falls back to separate kernels; send/receive uses the MPI library's GPU-aware point-to-point.

4. Persistent Versus Blocking Interface

• The algorithms form a library atop any MPI 4.0 library, overriding functions via the MPI profiler hook (PMPI). For the persistent interface, MPI_Allreduce_init is overridden, with the original routine as fallback for unimplemented features. Persistent initialisation is relatively expensive (Section 5), motivating an alternative blocking interface.
• The blocking interface suits applications with varying message sizes but the same processes. Algorithms are lazily initialised on their first call for best-fit parameters and slightly adapted before execution inside MPI_Allreduce; message size is a multiple of the setup's size plus padding, and they are freed on communicator destruction. To avoid expensive re-setup on restart, all bytecodes are stored to disc and read back on re-initialisation ('wisdom' files).
• For the intra-node algorithms, the blocking interface is limited to message sizes needing no padding, or an extra memory copy is required if combined with inter-node communication (padding is generally impossible for the buffers passed to MPI_Allreduce). Buffers of different node processes are made accessible via XPMEM, with pointers cached in a tree data structure as in MPICH.

5. Benchmarks

• Testbeds: an HPE Cray EX with dual-socket 64-core AMD EPYC 7003 nodes, and an HPE EX with nodes of four NVIDIA Grace-Hopper chips (72 CPU cores each). MPICH 4.2.0 and Open MPI 5.0.2 used for comparison on both. Algorithm parameters were chosen empirically.
• One-node allreduce is measured for best and worst case task counts: best = all cores per CPU/node; worst = the largest prime fitting the cores (also motivated by cores occupied for other purposes). They measure allreduce, reduce, and broadcast on one node, and allreduce between nodes; GPU benchmarks compare with NCCL 2.20.3-1 (AWS plugin); concluded with CP2K.

5.1 Standalone communication

• Messages were split into equal parts with a 64-byte (cache-line) minimum; ≤64-byte messages are one segment. Pipelining and recursive-call features are not exploited here for a more direct comparison; maximum performance would need an expensive per-case parameter selection at initialisation.
• Table 3 — empirical allreduce parameters:

• AMD EPYC parameters reflect two sockets, each processor having four 16-core quadrants: the first reduce_scatter step is local within quadrants (largest data), the second between quadrants within a socket, the third between sockets (smallest data); allgather factors reverse this order, so the largest data moves on the highest-bandwidth links. Grace-Hopper uses factor four for the innermost reduce_scatter/allgather. Each datapoint is the arithmetic average of ten OSU microbenchmark 7.4 runs (default parameters); for multiple factor sets, each datapoint is the minimum across sets, still averaged over ten runs.
• Figure 9 (AMD EPYC): with 128 tasks/node ext_mpi outperforms MPICH and Open MPI because factors map directly to available cores; with 127 tasks (prime), cyclic shift is used (recursive exchange only allows {127}), giving lower but still MPICH/Open-MPI-comparable performance.
• NVIDIA Grace CPU benchmarks: single CPU (Figure 10), single node (Figures 11, 12), multiple nodes (Figure 13). The single-CPU test used all 72 cores and 71 (prime, recursive exchange only via factor {71}); unlike AMD EPYC there is no cyclic-shift penalty on a single socket. On the whole node (Figures 11, 12) recursive exchange beats cyclic shift. All benchmarks except Figure 11 are out-of-place.
• On AMD EPYC and Grace, two and four sockets respectively; each socket's inter-node algorithm is set up with factor four. The blocking HPE MPI allreduce (no persistent version in this release) underperforms their implementation on both architectures except at very small messages (equally fast). Bandwidth- optimal intra-node (complete reduce_scatter then allgather) gives marginally higher long-message performance. On Grace the vendor MPI used CMA (not XPMEM) due to current limitations.
• Figure 14 — allreduce initialisation cost on Grace-Hopper: for short messages on one socket, ext_mpi initialisation is four orders of magnitude slower than executing the allreduce; for long messages initialisation and execution take roughly equal time. Whole-node initialisation is two orders of magnitude more expensive than per-socket, owing to disadvantageous algorithmic complexity in the init routine; MPICH and Open MPI have much smaller init times.
• Figures 15–17 — reduce and broadcast. Reduce is only slightly faster than the references because the reduction runs in a memory buffer with a final copy- out to the root (the MPI standard forbids using non-root task buffers directly; an MPI_Info argument permitting this would add speedups). Broadcast performs well vs references on AMD EPYC and shows an advantage for medium/long messages on Grace.
• Figure 13 — four nodes, 128 tasks/node on AMD EPYC and 288 tasks/node on Grace; network is Slingshot with libfabric 1.15.2.0; vendor HPE MPI (MPICH- based) replaces MPICH/Open MPI here. Intra-node factorisation matches the one-node benchmarks; the inter-node part uses latency-optimal parameters (allreduce based on allgather with a reduction).
• Figure 18 — GPU allreduce on one Grace-Hopper node. With one MPI task per GPU (four GPUs), ext_mpi outperforms HPE MPI for short messages and matches NCCL when NCCL's 'blocking' flag is enabled — unsurprising, as a single CUDA kernel per task (Listing 3) is called and this is likely identical to NCCL's approach. With four tasks per GPU (16 tasks/node), ext_mpi outperforms HPE MPI.
• Figure 19 — GPU allreduce on four Grace-Hopper nodes, one MPI task per GPU: for short messages HPE MPI and NCCL are faster (deemed not relevant, since for short messages one would copy GPU data to CPU, reduce, and copy back); for medium messages ext_mpi beats HPE MPI, and vs NCCL the comparison is less clear (NCCL spikes are attributed to a NCCL↔︎libfabric interplay mismatch); for long messages ext_mpi is almost identical to NCCL and faster than HPE MPI. With 16 tasks/node, ext_mpi outperforms HPE MPI for both medium and long messages.

5.2 Application

• CP2K (quantum chemistry / solid-state physics) is benchmarked using allreduce and reduce_scatter_block via the blocking interface. Cases: "QS/H2O-512" (close to many current applications) and "QS_mp2_rpa/64-H2O/H2O-64-RI-dRPA-TZ" (a popular theoretical test). Allreduce runs in CPU memory, reduce_scatter_block in GPU memory; allgather/reduce/bcast are neglected as minor. Both ran on one Grace-Hopper node with 16 MPI tasks × 16 OpenMP threads.
• Table 4 — MPI_Allreduce parameters in CP2K:

• Table 5 — MPI_Reduce_scatter_block parameters in CP2K:

• Not all MPI datatypes/operators used by CP2K are implemented, so allreduce (ext_mpi) calls/message sizes are tabulated separately from the overall values (which include the MPICH fallback); the fallback reduces average speedups.
• Table 6 — average collective performance and overall runtime (± = corrected sample standard deviation over 15 samples):

QS/H2O-512:

QS_mp2_rpa/64-H2O/H2O-64-RI-dRPA-TZ:

• Headline application result: QS/H2O-512 is accelerated by 2.5% — allreduce time is halved and reduce_scatter_block time is cut to 1/20 vs MPICH. QS_mp2_rpa gets 1.5%, limited by the moderate reduce_scatter_block speedup (allreduce again halved). More datatypes (especially for reduce_scatter_block) would improve results; real runs with many more time steps would spend relatively less in initialisation, yielding higher speedups. Table 6 timings are for a restart run using wisdom files; with only 16 tasks/node, wisdom-file generation contributes marginally.

6. Related Work

• Träff et al. (2023) give a uniform circulant-graph framework for designing and analysing collectives, focused on reductions.
• MPI shared-memory approaches: shmget segments needing an extra copy (Peng et al., 2023), application-provided shared buffers (Li et al., 2014), or Cross Memory Attach (CMA, Vienne, 2014). Li et al. (2014) prototyped shared-memory collectives; MPICH, MVAPICH, Open MPI follow this via XPMEM.
• Radix > 2 has been used recently (Fan et al., 2024; Wilkins et al., 2023) to cut communication steps; non-constant radix appears in Jocksch et al. (2021) and Rüfenacht et al. (2017), varying with message size/topology/hardware.
• Special cases of their generalised recursive exchange: Rüfenacht et al. (2017) recursive multiplying for short messages, Thakur et al. (2005) for long messages and non-power-of-two process counts.
• High-data-locality allreduce resembles the Sparbit algorithm (Loch and Koslovski, 2021). GPU intra-node allreduce via CUDA IPC was done by Faraji and Afsahi (2014) with two designs (binomial reduce-broadcast; shared-buffer aware).
• Autotuning MPI collectives with ML (Hunold and Carpen-Amarie, 2018) outperformed Intel MPI and Open MPI defaults. Short-message allreduce on network routers is Flare (De Sensi et al., 2021). Sparse-data allreduce with compression (Li and Hoefler, 2022) and locality-aware sparse exchange (Geyko et al., 2024) are noted.
• Inter-group allgather (m→n tasks) is covered by Kang et al. (2019); multiple sockets are treated as multiple parallel network lanes (Träff and Hunold, 2020; Tran et al., 2024). NCCL is noted as a commercial non-MPI library using the ring algorithm, suited to long messages.

7. Conclusions

• A persistent allreduce implementation uses the persistent interface to separate deriving complex factorisations from executing the allreduce, enabling more complex algorithms than blocking-only implementations and high performance for repeated identical calls. A blocking interface with partial (lazy) initialisation is also provided; knowing the number of subsequent calls helps algorithm selection because initialisation time can be non-negligible and algorithm-dependent.
• Recursive exchange and cyclic shift (previously inter-node) were extended to shared-memory intra-node and mixed intra-/inter-node, with multi-socket support; the tree algorithm serves the on-node part of inter-node allreduce and intra-node broadcast/reduce. Pure intra-node outperforms recent MPICH and Open MPI by up to almost an order of magnitude for medium/large messages on AMD EPYC and Grace-Hopper (CPU and GPU); combined intra-/inter-node gives similar speedups.
• GPU timings are comparable to NCCL; the libraries' inter-node algorithms lack published formal descriptions, and NCCL uses the ring algorithm while ext_mpi does not. reduce_scatter_block is covered by the allreduce algorithms; applied to CP2K, the blocking interface gives a 2.5% overall speedup on a standard benchmark.

8. Future Work and Opportunities

• The current empirical parameter selection is to be replaced or improved by a machine-learning-guided heuristic for extensibility and performance portability, complementing the compiled approach.
• The work may be extended to sparse data (common in deep learning) via data compression (Li and Hoefler, 2022) or novel locality-aware algorithms (Geyko et al., 2024).

9. Limitations

• Initialisation is expensive — up to four orders of magnitude slower than a single short-message execution; whole-node init is two orders costlier than per-socket init due to disadvantageous algorithmic complexity (MPICH/Open MPI init is far cheaper).
• Cyclic shift and cyclic copy-in support commutative operators only.
• Prime task counts (127, 71, 283) force poor factorisation ({prime}), degrading performance (notably on AMD EPYC; not on a single Grace socket).
• The blocking interface is limited to message sizes needing no padding (or an extra memory copy when combined with inter-node communication).
• Reduce is only slightly faster than references because reduction happens in a buffer with a final copy to root (MPI forbids using non-root buffers directly).
• Not all CP2K datatypes/operators are implemented; the MPICH fallback reduces speedups, especially for reduce_scatter_block. On Grace, vendor MPI used CMA not XPMEM due to current limitations.

10. Discussion of NCCL

• NCCL 2.20.3-1 (with the AWS plugin) is the GPU comparison baseline; it is a commercial, non-MPI library that uses the ring algorithm, making it particularly suited to long messages. ext_mpi does not use ring.
• On a single Grace-Hopper node (one MPI task per GPU), ext_mpi matches NCCL when NCCL's 'blocking' flag is enabled — both reduce to a single CUDA kernel per task (Listing 3), a solution the authors judge cannot be beaten and is likely identical to NCCL's.
• On four nodes, for long messages ext_mpi performs almost identically to NCCL; NCCL's short-message advantage is deemed irrelevant (one would offload short- message reductions to the CPU), and NCCL timing spikes between 10³ and 10⁷ bytes are attributed to a mismatch in the NCCL↔︎libfabric interplay.
• The authors note that neither NCCL's nor the vendor MPI's inter-node algorithms have published formal descriptions, in contrast to their own.

Note on NCCL Tuning

The paper's central lever — choosing a multi-radix factorisation per hardware tier (quadrant → socket → node) and a message split whose smallest piece stays at the cache-line size — is structurally the same decision NCCL exposes through algorithm, protocol, and chunkSize selection. The authors show that the optimal factor set is strongly hardware- and message-size-dependent (Table 3: {−16,−4,−2, 2,4,16} on dual-socket EPYC versus {−9,−8,8,9} on single-socket Grace), and that a wrong choice such as the forced {127} prime factorisation collapses performance — evidence that per-collective configuration search, not a fixed default, is what recovers the speedup. Their explicit future-work goal of replacing empirical parameter selection with a machine-learning-guided heuristic is the same problem statement as learned NCCL configuration tuning, applied to a compiled MPI stack.