The Multipath Reliable Connection (MRC) Transport — Detailed Summary

Rip Sohan, Eric Spada, Eric Davis, Mark Handley, Idan Burstein, Tony Hurson, Jithin Jose, Vivek Kashyap, Rong Pan, Sayantan Sur, Torsten Hoefler, Costin Raiciu, David Riddoch, Amin Tootoonchian, et al. (multi-vendor author list) | AMD / Broadcom / Microsoft / NVIDIA / OpenAI / Intel | Open Compute Project (OCP) specification companion paper | arXiv:2606.18170v1 [cs.NI], 16 Jun 2026 | No DOI (canonical spec: OCP MRC Specification v1.0, 2026)

Per-section summary organized by the paper's headings. Each section includes paragraph-level bullet points. This is a transport-specification / design paper: it defines a wire protocol and a set of primitives, and it contains no experimental section — all empirical results (goodput, latency, failure-recovery) are deferred to the companion paper Araujo et al., "Resilient AI Supercomputer Networking using MRC and SRv6," arXiv:2605.04333.

Bibliographic / provenance notes

• Cross-industry authorship spanning AMD, Broadcom, Microsoft, NVIDIA, OpenAI, and Intel — signals an open, vendor-neutral standardization effort rather than a single-company product paper.
• Developed under the Open Compute Project (OCP); the paper is the transport-description companion to the formal OCP MRC Specification v1.0 [9] and to the empirical evaluation paper [10].
• Acknowledgments disclose that LLMs were used for language editing during manuscript preparation.

Abstract

• MRC is an open, production-grade transport for large-scale AI/ML training running over best-effort Ethernet (no lossless-fabric / PFC requirement).
• It extends RoCEv2 with explicit, composable primitives for per-packet multipath and sender-based congestion control.
• It decouples packet delivery from semantic (RDMA) processing, adding capabilities for accelerated packet-loss recovery.
• It adds resilience against port and path failures.
• The paper's stated purpose is to describe MRC and detail its core capabilities and mechanisms (not to evaluate it empirically).

I. Introduction & Motivation

• The network is now a key bottleneck in large-scale AI/ML training: as cluster scale for frontier models grows, the fabric must sustain collective communication across thousands of accelerators with low latency, high goodput, and fault tolerance. Industry responded with high-performance proprietary transports and emerging clean-slate open specifications.
• Both existing responses have drawbacks: proprietary transports are hard to evolve across vendor boundaries; clean-slate open specs require new hardware, new software stacks, and years of interoperability maturation before large-scale deployment.
• The dominant open standard in production AI/ML clusters today is the RoCEv2 Reliable Connection (RC) transport. RC provides ordered, point-to-point delivery and is well understood, but is fundamentally insufficient for large-scale training for three reasons:
  1. RC is single-path with go-back-N retransmission.
  2. RC congestion control (PFC-based lossless Ethernet or DCQCN) does not scale to large clusters.
  3. RC is highly vulnerable to link and fabric failures.
• The paper presents MRC (Multipath Reliable Connection), an OCP open specification. Unlike both proprietary and clean-slate designs, MRC extends RC directly, preserving RC's well-understood semantic and software model while adding orthogonal primitives that close the three gaps. The result is a transport adoptable incrementally, spanning fixed-peer single-tier accelerator fabrics to arbitrary-peer multi-tier NIC deployments.
• Three contributions are claimed:
  1. An explicit per-connection multipath and multi-plane framework enabling controlled packet spraying plus congestion-control primitives for best-effort Ethernet — selective acknowledgments, reasoned negative acknowledgments, and wire-visible host backpressure signals.
  2. Reliability primitives offering scalable, datapath-usable capabilities for accelerated packet-loss recovery.
  3. Endpoint-scoped operations providing scalable, datapath-usable reachability and liveness signaling, enabling fast failover without waiting for control-plane convergence.
• MRC is already in production; the companion paper [10] provides full empirical evaluation. Roadmap: §II design & mechanisms, §III wire protocol, §IV controller and application API, §V conclusion.

II. Design Overview

• MRC modifies/extends RC to support per-connection multipath packet spraying and fast failover under link/fabric faults, while preserving RC's reliable, ordered delivery and completion software abstraction. It combines concepts from UltraEthernet (UEC) and IBTA. The design is a set of orthogonal primitives forming a coherent control loop: endpoints select per-packet paths, bound outstanding traffic to control reordering/buffering, exchange selective reliability and congestion feedback, and mitigate port/path failures by broadcasting datapath-visible reachability signals.
• Table I categorizes MRC primitives as mandatory (M) — required for correctness — or optional (O) — improving efficiency and resilience. Implementations adapt optional primitives to their hardware design point and expected network configuration.
• The primitives are tightly co-designed: an Entropy Value (EV) dictates a packet's path; EV spraying balances load but introduces reordering; Maximum PSN Range (MPR) bounds reordering and retransmit state; SACK/NACK/Trimmed feedback keeps recovery tractable; reliability probes elicit connection state; SACK-clocked congestion control uses EV reflection and ECN to tune load balancing and avoid failed paths; traffic classes isolate control traffic from congested data and prioritize retransmits; and EV probes plus Port Status Updates signal path/port health at line speed.
• MRC deliberately simplifies the transport for AI training: it narrows data-plane operations to Write and Write-with-Immediate (WriteIMM) and removes RC end-to-end flow control in favor of explicit bounded-flight mechanisms. It permits out-of-order data placement at the responder to tolerate packet spraying and decouples packet delivery from semantic processing; requester/responder completion semantics remain unchanged.

Table I — MRC Transport Primitives (M = mandatory, O = optional)

II.A Multipath and Multi-Plane Operation

• RC assigns each QP to one network path by ECMP hashing over the connection 5-tuple. This underutilizes multi-path fabric capacity and gives no way to steer, rebalance, or respond to path-level congestion or failure.
• MRC provides explicit per-connection multipath via packet spraying: the requester carries an Entropy Value (EV) in each packet and rotates it per-packet, distributing a single QP's packets across many fabric paths. EVs are a first-class transport primitive — carried on the wire, interpreted by the network to select a path, and managed by endpoints for load balancing and resilience.
• MRC supports three EV mechanisms under a common abstraction, letting systems swap between implicit hashing and explicit source routing without altering transport-layer multipath logic (contingent only on NIC capabilities):
  1. Legacy ECMP hashing (as used today).
  2. Structured EV — encodes a deterministic hop-by-hop source route in the UDP source port and IPv6 flow label.
  3. SRv6 uSID source routing — explicit path selection via SRv6 micro-segments (RFC 8986 / RFC 9800).
• EV selection is configured through EV profiles, defining the EV "universe" for a connection and how EVs are generated (explicit lists, NIC-generated, or SRv6-derived). Profiles are controller-managed and shared across QPs to bound per-QP state. At runtime, implementations keep lightweight per-EV state to bias selection away from congested/unhealthy paths. Each EV is classified into one of four states: GOOD, SKIP, DENIED, ASSUMED_BAD — only GOOD EVs are used for transmission; the others represent progressively stronger exclusion triggered by datapath observations or controller intervention. Implementations may allow EVs to transition back from ASSUMED_BAD via EV probing (§II.E). In Clos-style networks this active end-to-end load balancing can rapidly avoid congested/flaky links — even beyond the point where switches no longer have alternate paths to the destination.
• MRC extends spraying to multiple fabric planes via multi-port NICs, partitioning EVs across physical ports (planes). A single QP can spray across multiple physical planes, maximizing path diversity and isolating failures to a single port/plane. MRC decouples response-path selection from the forward request path — this asymmetry lets control packets use different paths/planes from data, reinforcing resilience against partial fabric failures and asymmetric congestion. The aim: network failures should not cause QP failure or greatly impact transfer tail latency, because large synchronous AI training job performance is strongly dictated by 100th-percentile transfer performance.

II.B Receiver-Driven Bounded In-Flight Transmission

• Sprayed packets often arrive out-of-order. Without structural bounds on outstanding traffic, endpoints face responder buffer exhaustion and requester retransmission-state explosion. MRC mitigates via the Maximum PSN Range (MPR) — a sliding receive window that strictly bounds in-flight request packets: a requester cannot send a packet with a sequence number beyond the upper edge of the responder's advertised packet-tracker bitmap. MPR is negotiated at connection establishment and reflects the responder's request-packet buffering capacity.
• Both endpoints maintain a PSN-fidelity, MPR-sized bitmap window to track in-flight packets. This structure underpins MRC's SACK/retransmission mechanisms (§II.C), bounding requester retransmit state directly to the negotiated MPR size; responder out-of-order reception state is likewise proportional to MPR.
• MPR can remain static, but MRC optionally supports Dynamic MPR, letting the responder update window size at runtime via SACKs. This optimizes hardware resources on two axes: (1) elastic bitmap-memory scaling based on actual demand rather than worst-case static provisioning; (2) NIC oversubscription/multiplexing of finite bitmap capacity across concurrent QPs by reclaiming allocations from idle/low-rate connections.
• An MRC responder also advertises a limit on concurrent outstanding WriteIMM operations. Whereas MPR bounds the PSN window at request-packet fidelity, WriteIMM limiting bounds the number of semantic-level operations the responder will process concurrently. Together these decoupled mechanisms give independent dimensions of in-flight bounding, limiting both packet and semantic-operation buffering/state.

II.C Reliability and Recovery

• MRC targets best-effort Ethernet, where loss and reordering are expected. It introduces a packet-delivery layer that tracks packet reception at the responder decoupled from RDMA semantic processing.
• SACK & NACK: MRC enforces reliability via selective acknowledgments (SACK). Each SACK carries a cumulative acknowledgment, a bitmap offset, and a bitmask of out-of-order arrivals relative to that offset — letting the requester distinguish loss from transient reordering. The responder advances the cumulative ack as gaps fill, sliding the requester's transmission horizon forward. Negative acknowledgments (NACKs) provide proactive non-delivery signaling for immediate retransmission; operating at the packet-delivery layer, NACKs are driven by deterministic events such as trimmed-packet arrival or local resource exhaustion.
• To minimize head-of-line blocking and speed window advancement, MRC optimizes retransmission of the earliest missing PSNs: responders prioritize reporting the oldest incomplete bitmap regions, and requesters retransmit on a differentiated, higher-priority traffic class (§II.E).
• Trimmed Packet Support: switches truncate logically dropped packets, forwarding only their headers via a high-priority traffic class. Responders process these headers to generate a NACK. This explicit, low-latency loss signal enables fast retransmission that bypasses retransmit-timeout timers.
• Reliability Probes: requesters issue out-of-band probes to query the responder's current delivery state. Probes do not consume PSNs or alter connection state (replies are decoupled from forward progress); they can be fire-and-forget or timer-driven. Responders reply with a standard SACK, letting the requester audit connection health proactively, independent of ACK timeouts.
• Timeout and Loss Detection: MRC refines RC's connection timer and Local ACK timeout, adding a higher-resolution Local ACK timeout with linear-to-exponential backoff plus an optional per-packet timer. MRC also allows implementation-specific loss detection/recovery; full details in the specification [9].

II.D Congestion Control and Load Balancing Integration

• MRC is designed to support congestion control over best-effort Ethernet, focused on NSCC — a sender-based, window-driven algorithm using ECN and RTT-derived queuing delay to regulate a per-fabric-path congestion window. MRC standardizes the transport signals that drive NSCC: every SACK encapsulates CC metadata in a dedicated sub-header conveying forward-path ECN markings, responder accounting metrics (e.g., cumulative bytes received), and responder-side congestion-window penalties. Alternative algorithms may use a subset of these signals; the controller enables per-connection CC profile configuration (analogous to the EV profile).
• RTT estimation via two mechanisms: requester local state, or explicit timestamps in packet headers reflected back by the responder in the CC sub-header. MRC also supports per-connection service-time compensation: the responder reports its internal processing latency so the requester can subtract this overhead and compute accurate network RTT even under responder processing delay.
• MRC supports host backpressure: responders signal memory-subsystem queuing in an implementation-agnostic way; NSCC uses this to modulate the requester's window, preventing responder-side memory contention from degrading end-to-end performance.
• Responders reflect forward-path EVs within return SACKs, letting the requester correlate ECN markings with specific paths. Multi-path load-balancing algorithms use this per-path feedback to dynamically adjust path-selection probabilities (§II.A). The load-balancing algorithm itself is implementation-defined.

II.E Resilience and Fast Fail-over

• Large-scale fabrics suffer frequent disruptions — link flaps, blackholes, path asymmetries — degrading performance and sometimes crashing jobs. Baseline RC is ill-equipped, relying on control-plane convergence and end-to-end timeouts.
• MRC introduces endpoint operations: lightweight, GID-scoped control exchanges identified on the wire by a reserved QP identifier 0x2. Receivers process them statelessly without connection context, replying by simply swapping Layer-2 and Layer-3 fields. GID scoping lets a single exchange update state across all connections sharing that GID, amortizing overhead and improving scalability.
• Two endpoint operations are defined. EV probes are path-scoped, validating specific EVs' viability; used to drive EV selection and health state (§II.A). Implementations may issue them from the datapath or via the controller API (§IV).
• Port Status Update is an asynchronous operation announcing port-level reachability in multi-plane networks. It carries a port_status_mask bitmap of the sender's local port health; the receiver logs this to avoid degraded ports. It may be issued from datapath or controller API. This accelerates reachability signaling from slow control-plane timescales to data-plane round-trip time.
• Together these form a fast failover capability: Port Status Updates proactively signal link/fabric failures while EV probes autonomously and continually query viability of degraded paths — no control-plane intervention. Exposing both via controller API and datapath lets implementations tailor to use-case/deployment demands.

Architecture (control-loop closure)

+-------------------- REQUESTER (sender) NIC ---------------------+
|  EV generator/selector  (per-packet Entropy Value)            |
|    states: GOOD / SKIP / DENIED / ASSUMED_BAD                 |
|  Multi-plane port partitioner (EVs split across NIC ports)    |
|  MPR transmit bitmap  (won't send beyond responder window)   |
|  NSCC engine  (per-path window; ECN + RTT; consumes CC_STATE)|
|  Retransmit engine (linear->exp timeout; prioritized oldest) |
+--------------------------------|-------------------------------+
   Request: BTH -> METH -> [TSETH] -> RETH -> [ImmDt]  (per-packet EV)
                                 v
+----------------- ETHERNET FABRIC (Clos, multi-plane) ----------+
|  Path select via ECMP / Structured-EV / SRv6 uSID             |
|  Packet TRIMMING: truncate-to-header, high-priority class     |
|  DSCP classes: data | retransmit | control                    |
+--------------------------------|-------------------------------+
                                 v
+-------------------- RESPONDER (receiver) NIC -------------------+
|  MPR receive bitmap (out-of-order tolerant)                   |
|  Advertises MPR / Dynamic MPR + WriteIMM inflight limit       |
|  Out-of-order data PLACEMENT into memory (ordered completion) |
|  SACK gen: cum-ack + offset + OOO bitmask + CC_STATE          |
|            + reflected EV + reflected timestamp/service-time  |
|  NACK gen (on trimmed-header arrival / resource exhaustion)   |
|  Host backpressure signaler (memory-subsystem queuing)        |
+--------------------------------|-------------------------------+
   Return path (ASYMMETRIC, independent plane/path):
     SACK: BTH->SETH->CC_STATE  |  NACK: BTH->NETH
     Probe: BTH->PETH           |  Endpoint op: BTH->ERTH / ->EETH (QP 0x2)
                                 v
+---------------------- CONTROLLER (CAP_NET_ADMIN) --------------+
|  EV profiles + CC profiles (per-QP, shared) | device/port qry|
|  EV state events | EV probes | Port Status Updates            |
+---------------------------------------------------------------+

Loop:  EV picks path -> spraying balances load but reorders ->
       MPR bitmap bounds reorder/retransmit state ->
       SACK/NACK/trim/probe recover losses ->
       CC_STATE + reflected EV/RTT drive NSCC window & bias EV
       selection away from congested/failed paths ->
       EV probes + Port Status Updates handle failure at RTT
       timescale (not control-plane convergence).

• Layered decoupling (key architectural idea): a Packet Delivery Layer (per-PSN reception bitmaps; SACK/NACK/trim/probe; out-of-order placement) is split from the RDMA Semantic Layer (Write/WriteIMM completions, unchanged vs. RC). Out-of-order data can land in memory while completion semantics stay ordered.

III. Wire Protocol

• MRC's wire protocol is a minimal delta over RoCEv2 RC: a new opcode space, minor BTH modifications, new control packet types, and a new class of endpoint operations, via new headers (Table II) encapsulated within standard RoCEv2/UDP/IP. The MRC UDP destination port and checksum match RoCEv2; parsers must additionally inspect the DSCP field.
• Opcode Space: MRC isolates its packet types from RC via a distinct 0101 transport prefix. RC and MRC endpoints are non-interoperable.
• BTH Modifications: adds an rtx bit (flags retransmissions), a tsh bit (signals a TSETH header present), and repurposes the PSN field as a requester-local request_id for probe/endpoint correlation.
• Request Packets: uniform header stack BTH → METH → [TSETH] → RETH → [ImmDt]. RETH is present in every packet; METH is populated only for immediate operations; TSETH presence is per-packet, signaled by the BTH tsh bit; ImmDt is appended post-RETH for immediate variants.
• Control Packets: bifurcated into Reliability control packets (SACK, NACK) and Endpoint operations, both using new headers appended to BTH. SACKs use SETH → CC_STATE (delivery state + congestion telemetry); NACKs use NETH (non-delivery signaling); Reliability Probes use PETH (elicit a SACK-encoded response); Endpoint operations map Endpoint Requests (ERTH) to Endpoint Responses (EETH).
• Detailed header layouts and field definitions are deferred to the specification [9].

Table II — MRC Header Modifications

IV. Application and Controller API

• MRC defines two APIs: an application API (mrc.h) and a privileged controller API (mrc_ctl.h).
• Application API: applications use libibverbs for device discovery and memory registration; MRC adds new entry points only where transport semantics diverge. MRC resources (mrc_cq, mrc_qp) mirror existing verbs naming/lifecycle. Functions are restricted to transport-essential operations with strict signature/parameter parity with verbs. mrc_modify_qp() is the primary configuration primitive — managing Dynamic MPR, Trimmed Packet support, service-time compensation, and EV/CC profile assignment. Extended queries mrc_query_device() and mrc_query_port() expose MRC-specific capability and per-port feature flags absent from verbs. QP connection setup is out-of-band; every QP is bound to pre-configured EV and CC profiles.
• Controller API: a privileged process (CAP_NET_ADMIN) governing policy and telemetry. Mandatory capabilities: device/port queries and EV/CC profile management. Optional: EV state events and EV probes.

V. Conclusion

• MRC demonstrates that the multipath, reliability, congestion-control, and resilience deficiencies of RoCEv2 RC are resolvable as a practical, incremental, composable extension, built on existing hardware designs, to deliver an open, production-ready transport for large-scale AI/ML training. Comprehensive empirical evaluation is in companion paper [10].

Named-entity / quantitative inventory

Because this is a specification paper, "quantitative" content is limited to protocol constants and enumerations rather than measured results:

• Reserved QP identifier for endpoint operations: 0x2.
• Opcode transport prefix isolating MRC from RC: 0101.
• EV states (4): GOOD, SKIP, DENIED, ASSUMED_BAD (only GOOD transmits).
• Design objective metric: 100th-percentile (tail) transfer performance dictates synchronous AI training job performance.
• Named systems/algorithms/protocols: MRC, RC, RoCEv2, NSCC (sender-based window ECN+RTT CC from UltraEthernet), DCQCN, PFC, ECMP, SRv6 uSID, Structured EV, MPR, Dynamic MPR, WriteIMM, SACK, NACK, Trimmed packets, Reliability Probes, EV Probes, Port Status Update, UltraEthernet (UEC), IBTA; contemporaneous transports cited: Falcon (Google, SIGCOMM '25), Alibaba Stellar (SIGCOMM '25).
• Headers: BTH, TSETH, RETH, METH, SETH, NETH, PETH, ERTH, EETH, CC_STATE, ImmDt.
• APIs / symbols: mrc.h, mrc_ctl.h, mrc_cq, mrc_qp, mrc_modify_qp(), mrc_query_device(), mrc_query_port(), libibverbs, CAP_NET_ADMIN.
• No equations, no measured throughput/latency/FCT numbers appear in this paper.

Experimental setup & results

• None in this paper by design. The authors state (Intro ¶6 and Conclusion) that comprehensive empirical evaluation — goodput, latency, and failure-recovery — is provided in the companion paper Araujo et al., "Resilient AI Supercomputer Networking using MRC and SRv6," arXiv:2605.04333, 2026 ([10]). No testbed hardware, software versions, workloads, baselines, or metrics are reported in 0061 itself.

Limitations / boundary conditions (as stated or implied)

• No explicit "Limitations" or "Future Work" section. Implicit boundaries:
• RC and MRC endpoints are non-interoperable (distinct 0101 transport prefix) — MRC cannot interoperate with legacy RC.
• MRC deliberately narrows the data plane to Write and Write-with-Immediate only and removes RC end-to-end flow control — other RDMA operations are out of scope by design.
• Many features are optional and left implementation-defined: the load-balancing algorithm, the fast-loss-detection hook, alternative CC algorithms, per-packet timers, and whether EV probes / Port Status Updates issue from the datapath or the controller.
• Empirical validation and quantitative performance are explicitly deferred to the companion paper [10]; this paper cannot be assessed on measured results.

Positioning vs. related transports

• Contrasted against RoCEv2 RC (single-path, go-back-N, DCQCN/PFC that "do not scale" per Gangidi et al., SIGCOMM '24), against clean-slate UltraEthernet (which MRC borrows NSCC and design concepts from but avoids the clean-slate hardware/software cost by extending RC), and against proprietary transports Falcon (Google) and Alibaba Stellar (both SIGCOMM '25). MRC's distinguishing bet is incremental adoptability on existing RoCE hardware plus an open OCP specification.
• Packet trimming (switch truncates instead of drops, forwarding headers as an explicit loss signal) situates MRC in the NDP/EQDS lineage of receiver-driven, trim-based loss recovery.