The Multipath Reliable Connection (MRC) Transport

Rip Sohan, Eric Spada, Eric Davis, Mark Handley, Idan Burstein, Torsten Hoefler, Costin Raiciu, David Riddoch, Amin Tootoonchian, et al. (multi-vendor author list) | AMD / Broadcom / Microsoft / NVIDIA / OpenAI / Intel | Open Compute Project (OCP) specification companion paper | arXiv:2606.18170v1 [cs.NI], 16 Jun 2026 | No DOI (canonical spec: OCP MRC Specification v1.0, 2026)

Problem

The network is now the key bottleneck in large-scale AI/ML training: the fabric must sustain collective communication across thousands of accelerators with low latency, high goodput, and fault tolerance. The dominant open standard in production today, the RoCEv2 Reliable Connection (RC) transport, is fundamentally insufficient for three reasons: (1) it is single-path with go-back-N retransmission; (2) its congestion control (PFC-based lossless Ethernet or DCQCN) does not scale to large clusters; and (3) it is highly vulnerable to link and fabric failures. The two existing responses each carry cost — proprietary transports are hard to evolve across vendor boundaries, and clean-slate open specs require new hardware, new software stacks, and years of interoperability maturation. Because large synchronous training performance is dictated by 100th-percentile (tail) transfer performance, path-level congestion and failures directly stall entire jobs.

Core Insight

The multipath, reliability, congestion-control, and resilience deficiencies of RoCEv2 RC can be closed as a practical, incremental, composable extension of RC — preserving RC's well-understood semantic and software model — rather than requiring a clean-slate transport. MRC adds a coherent set of orthogonal primitives (per-packet multipath via Entropy Values, receiver-bounded in-flight windows, a packet-delivery layer decoupled from RDMA semantics, SACK-clocked sender-based congestion control, and datapath-visible failure signaling) that compose into one control loop over best-effort Ethernet with no lossless-fabric requirement.

Method

MRC extends RoCEv2 RC with a minimal wire delta and a set of mandatory (M) / optional (O) primitives grouped into five categories:

Multipath & Multi-plane:  per-packet EV spraying; ECMP / Structured-EV / SRv6
                          uSID source routing; EV profiles; EV health states
                          (GOOD/SKIP/DENIED/ASSUMED_BAD); multi-plane operation
In-Flight Bounds:         Maximum PSN Range (MPR) window; Dynamic MPR;
                          Max WriteIMM inflight limit
Reliability & Recovery:   SACK (cum-ack + offset + OOO bitmask); NACK; packet
                          trimming (switch truncates to header); reliability
                          probes; linear->exponential ACK timeout; DSCP classes
Congestion Control & LB:  NSCC (sender-based, window-driven, ECN + RTT,
                          per-path window); timestamp reflection; service-time
                          compensation; responder host backpressure; per-path
                          ECN via reflected EVs
Resilience & Fast Fail-over: GID-scoped endpoint ops (reserved QP 0x2);
                          EV probes; Port Status Updates (port_status_mask)

The key architectural idea is a layered decoupling: a Packet Delivery Layer (per-PSN reception bitmaps, SACK/NACK/trim/probe, out-of-order placement) is split from the RDMA Semantic Layer (Write / Write-with-Immediate completions, unchanged vs. RC), so out-of-order sprayed data can land in memory while completion semantics stay ordered. The data plane is deliberately narrowed to Write and WriteIMM only, and RC end-to-end flow control is replaced by explicit bounded-flight (MPR + WriteIMM limits). The control loop closes as: EV picks a path → spraying balances load but reorders → MPR bounds reorder/retransmit state → SACK/NACK/trim/probe recover losses → CC_STATE plus reflected EV/RTT drive the NSCC window and bias EV selection away from congested/failed paths → EV probes and Port Status Updates handle failures at data-plane RTT timescale rather than waiting for control-plane convergence. The wire protocol is a minimal delta over RoCEv2 (distinct 0101 opcode prefix — RC and MRC endpoints are non-interoperable; BTH adds rtx/tsh bits and overloads PSN as request_id; new headers TSETH/METH/SETH/NETH/PETH/ERTH/EETH/CC_STATE). Two APIs are defined: an application API (mrc.h, libibverbs-compatible, mrc_modify_qp() as primary config primitive) and a privileged controller API (mrc_ctl.h, CAP_NET_ADMIN) managing EV/CC profiles and telemetry.

Experimental Setup

This is a transport-specification / design paper. It contains no experimental section — no testbed, workloads, baselines, or metrics are reported. The authors state (Introduction and Conclusion) that comprehensive empirical evaluation (goodput, latency, and failure-recovery) is provided in the companion paper Araujo et al., "Resilient AI Supercomputer Networking using MRC and SRv6," arXiv:2605.04333, 2026.

Headline Quantitative Results

No measured throughput, latency, tail-latency, or FCT numbers appear in this paper (they live in the companion paper). The only quantitative content is protocol constants and enumerations:

• Reserved QP identifier for GID-scoped endpoint operations: 0x2.
• Opcode transport prefix isolating MRC from RC: 0101 (RC and MRC endpoints are non-interoperable).
• EV health states: 4 — GOOD, SKIP, DENIED, ASSUMED_BAD; only GOOD EVs are used for transmission.
• Stated design objective: large synchronous AI training performance is dictated by 100th-percentile (tail) transfer performance — the target MRC's multipath and fast-failover mechanisms optimize for.

Limitations

• No explicit Limitations or Future Work section; boundaries are implied.
• RC and MRC endpoints are non-interoperable (distinct 0101 transport prefix) — no interop with legacy RC deployments.
• The data plane is deliberately narrowed to Write and Write-with-Immediate only, and RC end-to-end flow control is removed — other RDMA operations are out of scope by design.
• Many features are optional and implementation-defined (the load-balancing algorithm, fast-loss-detection hook, alternative CC algorithms, per-packet timers, and whether EV probes / Port Status Updates issue from datapath or controller), so behavior can vary across implementations.
• Empirical validation is entirely deferred to the companion paper; MRC cannot be assessed on measured performance from this document alone.

Open Problems / Boundary Conditions Called Out

The paper frames these as design boundaries rather than a formal open-problems list:

1. Real-world performance (goodput, latency, failure-recovery) must be established by the companion evaluation paper.
2. Interoperability with legacy RC is intentionally unsupported, leaving migration path a deployment concern.
3. The choice of multipath load-balancing algorithm and fast-loss-detection strategy is left open to implementers.