Also: Brief Summaries Detailed Summaries

Architecture & Design Analysis

The Multipath Reliable Connection (MRC) Transport

Source: Sohan, R.; Spada, E.; Davis, E.; Handley, M.; Burstein, I.; Hurson, T.; Jose, J.; Kashyap, V.; Pan, R.; Sur, S.; Hoefler, T.; Raiciu, C.; Riddoch, D.; et al. arXiv:2606.18170v1 [cs.NI], 16 Jun 2026. Open specification developed under the Open Compute Project (OCP). Authors' affiliations: AMD, Broadcom, Microsoft, NVIDIA, OpenAI, Intel (multi-vendor consortium). Companion (evaluation) paper: J. Araujo et al., "Resilient AI Supercomputer Networking using MRC and SRv6," OCP Foundation, 2026 (ref [10]) — holds the empirical goodput / latency / failure-recovery numbers absent from this paper. Reader: gemini-reader quota exhausted; extracted via native Read tool (5 pages, all pages incl. references). Analyst: Vishwakarma Date: 2026-07-28

System Architecture — the transport as a composed control loop
System-Under-Test Architecture — MRC as a delta over RoCEv2 RC
Design-Space Diagram — the primitives that are configurable vs. fixed
Algorithm / Control Flow Diagrams — spraying, bounding, recovery, CC, failover
Design Findings by Primitive Category (the "no measurements here" section)
Configuration-Regime Trade-off Tables
Bottlenecks & Insights Surfaced by the Design
Limitations of the Methodology
Note on NCCL Tuning
Analogy

1. System Architecture — the transport as a composed control loop

MRC is not a monolithic protocol; it is a set of orthogonal primitives that compose into one coherent sender/receiver control loop. The design intent, stated by the authors, is that reliability, multipath, congestion control, and resilience are co-designed but separable concerns: entropy values pick a path, spraying balances load but introduces reordering, a bounded window caps that reordering and the state it costs, selective/negative acknowledgments keep recovery tractable, and endpoint-scoped probes signal path health at line speed. The whole thing runs over best-effort Ethernet — explicitly not PFC-based lossless Ethernet or DCQCN — while preserving the ordered, reliable, completion semantics that applications already expect from RoCEv2 Reliable Connection (RC).

+-------------------------------------------------------------------+
|                     MRC Transport (per QP)                        |
|                                                                   |
|   REQUESTER (sender)                    RESPONDER (receiver)      |
|  +---------------------------+        +------------------------+  |
|  | Path Selection            |        | Out-of-Order Placement |  |
|  |  - Entropy Value (EV)     |        |  - decouple delivery   |  |
|  |    per-packet rotation    |        |    from RDMA semantics |  |
|  |  - GOOD EVs only          |        |  - MPR-sized bitmap    |  |
|  +------------+--------------+        +-----------+------------+  |
|               |                                    ^              |
|      == request pkts (sprayed) ==============>     |              |
|               |    BTH->METH->[TSETH]->RETH->[ImmDt]              |
|               v                                    |              |
|  +---------------------------+        +-----------+------------+  |
|  | In-Flight Bound           |        | Feedback Generator     |  |
|  |  - Max PSN Range (MPR)    |<-------|  - SACK (SETH+CC_STATE)|  |
|  |  - Max WriteIMM inflight  |  SACK  |  - NACK (NETH)         |  |
|  +------------+--------------+  slides |  - Probe reply (PETH)  |  |
|               |                window  +-----------+------------+  |
|               v                                    ^              |
|  +---------------------------+        == control pkts ==+         |
|  | NSCC Congestion Control   |   (may take a different EV/plane)  |
|  |  - window-driven          |                                    |
|  |  - ECN + RTT + host bp     |                                    |
|  |  - per-path EV reflection |                                    |
|  +------------+--------------+                                    |
|               |                                                   |
|               v                                                   |
|  +-----------------------------------------------------------+    |
|  | Resilience / Fast Failover (GID-scoped, reserved QP 0x2)   |    |
|  |   - EV Probes (ERTH -> EETH)                               |    |
|  |   - Port Status Update (port_status_mask)                  |    |
|  |   - stateless, processed without connection context        |    |
|  +-----------------------------------------------------------+    |
+-------------------------------------------------------------------+
^ Fig 1: MRC as one composed control loop. Data is sprayed across many
  paths (double arrows); feedback (SACK/NACK/probe) rides a separate,
  possibly different, path back (single arrows) and drives three
  couplers at once: the MPR window, the NSCC window, and per-path EV
  health. Failover signaling is lifted out of the connection entirely
  onto a stateless, GID-scoped endpoint channel (QP 0x2).

The architecture places three feedback consumers behind a single return signal (the SACK). This is the key structural decision: rather than run independent control loops for reliability, congestion, and load balancing — each with its own probe traffic — MRC overloads one SACK to carry cumulative ACK + out-of-order bitmap + a CC_STATE sub-header (forward-path ECN, cumulative bytes received, responder window penalties) + reflected forward-path EVs. One packet feeds the delivery tracker, the congestion window, and the path-selection probabilities simultaneously. The consequence is low control overhead at the cost of a wider, more complex SACK format.

A second structural decision is the bifurcation of the return path from the forward path. Control packets are not required to retrace the EV of the data they acknowledge; they may traverse a different path or even a different physical plane. This decouples "which way did the data go" from "which way did the acknowledgment come back," which is what lets MRC keep signaling alive when part of the fabric has failed asymmetrically.

The API surface reflects the same separability. Applications use libibverbs through mrc.h, where mrc_cq and mrc_qp mirror standard verbs lifecycle, and mrc_modify_qp() is the single configuration primitive (Dynamic MPR, trimmed packet, service-time compensation, EV/CC profile binding). A privileged controller running with CAP_NET_ADMIN uses mrc_ctl.h to manage EV and CC profiles and to consume telemetry. Connection setup itself happens out-of-band — every QP is bound to a pre-configured EV profile and CC profile before traffic flows, so the datapath never negotiates policy inline.

2. System-Under-Test Architecture — MRC as a delta over RoCEv2 RC

MRC is a proposed transport, and the cleanest way to read its architecture is as a minimal, deployable delta over RoCEv2 Reliable Connection (RC). The authors deliberately reject a clean-slate transport: MRC reuses RC's software model, verbs API, and completion semantics, and encapsulates inside standard RoCEv2/UDP/IP. The wire underneath is restructured; the abstraction above is preserved.

+---------------------------------------------------------------+
|  Application (collective libs, AI/ML training frameworks)     |  <- unchanged
+---------------------------------------------------------------+
|  libibverbs  (mrc.h: mrc_qp, mrc_cq, mrc_modify_qp)          |  <- verbs-compatible
+---------------------------------------------------------------+
|                                                               |
|  MRC Transport  (opcode prefix 0101 -- NOT RC-interoperable)  |
|  +---------------------------------------------------------+  |
|  | narrowed data plane:  WRITE  +  WRITE-with-IMM only      |  |
|  | RC end-to-end flow control  ->  REMOVED                  |  |
|  | replaced by explicit bounded-flight (MPR + WriteIMM lim) |  |
|  | out-of-order placement at responder                     |  |
|  | per-packet EV multipath + SACK/NACK + NSCC + failover    |  |
|  +---------------------------------------------------------+  |
+---------------------------------------------------------------+
|  RoCEv2 / UDP / IPv6   (MRC UDP dport + checksum per RoCEv2;   |
|                         parsers also inspect DSCP)            |
+---------------------------------------------------------------+
|  Path layer:  ECMP hash | Structured EV (UDP sport + IPv6     |
|               flow label) | SRv6 uSID source route (RFC 8986,  |
|               RFC 9800)                                        |
+---------------------------------------------------------------+
|  Best-effort Ethernet  (NO PFC, NO DCQCN; ECN + RTT signals   |
|                         used instead of lossless backpressure)|
+---------------------------------------------------------------+
^ Fig 2: MRC's layered position. Everything above the transport is
  RC-compatible; everything at and below the transport is rebuilt for
  multipath + best-effort operation. The "0101" opcode prefix makes RC
  and MRC endpoints mutually non-interoperable -- both ends must speak
  MRC.

Three simplifications define the delta. First, the data plane is narrowed to two operations, WRITE and WRITE-with-Immediate; the general RC operation set is dropped. Second, RC end-to-end flow control is removed and replaced by two explicit, decoupled in-flight bounds — a packet-fidelity bound (Maximum PSN Range) and a semantic-operation bound (Maximum WriteIMM inflight). Third, the responder is allowed to place data out of order, decoupling packet delivery from RDMA semantic processing so that packet spraying does not force in-order buffering. Completion semantics at requester and responder remain unchanged, which is what makes the transport a drop-in for existing software.

The header stack encodes this delta precisely. The Base Transport Header is modified rather than replaced, and a family of new headers carries the new signals.

  REQUEST packet header stack:
    BTH -> METH -> [TSETH] -> RETH -> [ImmDt]
     |      |        |         |        |
     |      |        |         |        +- Immediate Data (immediate ops)
     |      |        |         +---------- RETH: present in EVERY packet
     |      |        +-------------------- TSETH: iff BTH.tsh bit set
     |      +----------------------------- METH: iff immediate operation
     +------------------------------------ BTH: +rtx bit, +tsh bit,
                                            PSN overloaded as request_id
                                            for probes/endpoint ops

  CONTROL packets bifurcate:
    (a) reliability:  SACK = SETH -> CC_STATE   (delivery + CC telemetry)
                      NACK = NETH               (non-delivery signal)
                      probe reply = SACK triggered by PETH
    (b) endpoint ops: request = ERTH  ->  response = EETH
^ Fig 3: MRC header composition. BTH is modified (Table II); TSETH,
  METH, SETH, NETH, PETH, ERTH, EETH, CC_STATE are new; RETH is recast
  for MRC WRITE; ImmDt is unmodified. Overloading BTH.PSN as a
  requester-local request_id is what lets probes and endpoint ops
  correlate without consuming sequence-number space.

Table II — MRC header modifications (reproduced):

Header	Status	Role
BTH	Modified	adds rtx, tsh bits; overloads PSN for probes/ops
TSETH	New	timestamp / service-time
RETH	Recast	adapted for MRC WRITE semantics
METH	New	message header; tracks WriteIMM ops
SETH	New	SACK header; carries CC_STATE
NETH	New	NACK header
PETH	New	probe request header
ERTH	New	endpoint request header
EETH	New	endpoint response header
CC_STATE	New	congestion-control telemetry
ImmDt	Unmod.	immediate data

The header design is itself an architectural statement: by modifying BTH instead of defining a new base header, and by keeping RETH/ImmDt recognizable, MRC stays parseable by hardware that already understands RoCEv2 framing. The new headers are additive and appear only where their function is invoked.

3. Design-Space Diagram — configurable vs. fixed

Because MRC is a specification, its "design space" is not a sweep of measured configurations but a catalog of primitives each marked Mandatory (M) or Optional (O). The M set is the minimum an implementation must ship to be MRC; the O set is where hardware design points and network configurations diverge. This M/O partition is the paper's real independent-variable structure: it defines which behaviors are guaranteed to exist across the ecosystem and which are implementation-defined.

                MRC DESIGN SPACE (primitive catalog)
  +---------------------------------------------------------------+
  |                                                               |
  |  MULTIPATH & MULTI-PLANE                                      |
  |    [M] per-connection packet spraying (per-packet EV rotate) |
  |    [M] EV profiles (controller-managed, shared across QPs)   |
  |    [O] source routing (SRv6 uSID | Structured EV)            |
  |    [O] EV states & denylisting (GOOD/SKIP/DENIED/ASSUMED_BAD)|
  |    [O] multi-plane operation (spray across physical ports)   |
  |                                                               |
  |  IN-FLIGHT BOUNDS                                            |
  |    [M] Maximum PSN Range (MPR): responder-advertised window  |
  |    [M] Maximum WriteIMM inflight: concurrent WRIMM limit     |
  |    [O] Dynamic MPR (runtime resize via SACK)                 |
  |                                                               |
  |  RELIABILITY & RECOVERY                                      |
  |    [M] SACK + NACK reliability control packets               |
  |    [M] Linear + Exponential ACK timeout                      |
  |    [M] Differentiated DSCP traffic classes                   |
  |    [O] trimmed packet support (needs switch trimming)        |
  |    [O] reliability probes                                    |
  |    [O] per-packet timer                                      |
  |    [O] fast-loss detection hook (implementation-specific)    |
  |                                                               |
  |  CONGESTION CONTROL & LOAD BALANCING                         |
  |    [O] NSCC (window-based, SACK-clocked, ECN+RTT)           |
  |    [O] timestamp header (RTT measurement)                    |
  |    [O] service-time reporting / compensation                 |
  |    [O] responder host backpressure                          |
  |                                                               |
  |  RESILIENCE & FAST FAILOVER                                  |
  |    [O] EV probes (path-scope liveness)                       |
  |    [O] Port Status Update (asynchronous link-state signal)   |
  |                                                               |
  |  HELD FIXED (non-negotiable substrate):                      |
  |    - encapsulation: RoCEv2 / UDP / IP                        |
  |    - fabric: best-effort Ethernet (NO PFC, NO DCQCN)         |
  |    - opcode prefix: 0101 (RC non-interoperable)             |
  |    - reserved endpoint QP id: 0x2                            |
  |    - data-plane ops: WRITE + WRITE-with-IMM only            |
  +---------------------------------------------------------------+
^ Fig 4: The design space is an M/O primitive catalog, not a measured
  sweep. Note the striking imbalance: nearly all congestion-control and
  resilience machinery is OPTIONAL. The mandatory core is spraying +
  bounded flight + SACK/NACK + timeouts + traffic classes -- enough to
  be reliable and multipath, but the "make it fast and self-healing"
  layer is left to the implementer.

Two properties of this catalog matter. First, the mandatory core is small: an MRC implementation must spray per-packet, bound in-flight PSNs, generate SACK/NACK, back off on timeout, and separate traffic classes by DSCP — and that is essentially all. Everything that makes MRC competitive on tail latency (NSCC, host backpressure, EV denylisting, trimmed-packet fast loss, EV probes, Port Status Updates) is optional. This is deliberate: the specification "adapts optional primitives depending on hardware design point and expected network configuration." Second, several optional primitives carry infrastructure dependencies — trimmed packets need switches that truncate-and-forward; multi-plane needs multi-port NICs; SRv6 source routing needs SRv6-capable fabric. The optionality is therefore not arbitrary; it tracks what the surrounding hardware can support.

The three EV mechanisms form a sub-axis of their own, unified behind one abstraction so an operator can move between implicit and explicit path control without touching transport logic:

   EV ABSTRACTION (one interface, three realizations)
   +---------------------------------------------------------+
   |  interpret(EV) -> path                                  |
   +---------------------------------------------------------+
        |                    |                     |
        v                    v                     v
  +-----------+     +------------------+   +------------------+
  | ECMP hash |     | Structured EV    |   | SRv6 uSID        |
  | (legacy,  |     | src route in UDP |   | source route     |
  | implicit) |     | sport + IPv6     |   | (RFC 8986/9800,  |
  |           |     | flow label       |   | explicit)        |
  +-----------+     +------------------+   +------------------+
     implicit  <-------------------------------->  explicit
     (NIC/fabric picks)              (endpoint dictates hop-by-hop)
^ Fig 5: The EV abstraction spans a spectrum from implicit hashing to
  fully explicit source routing. Swapping among them is contingent only
  on NIC capability, not on rewriting multipath logic -- the transport
  always just asks "which path for this packet?" and the EV answers.

4. Algorithm / Control Flow Diagrams

4.1 EV state machine — per-path health

Every EV a connection may use is classified into one of four states. Only GOOD EVs are eligible for transmission; the other three encode progressively stronger exclusion. Transitions are driven either by datapath observation (ECN, loss, probe outcome) or by explicit controller intervention.

                     datapath signal / controller
          +---------------------------------------------+
          |                                             |
          v                                             |
      +--------+   transient issue    +--------+        |
      | GOOD   |--------------------->| SKIP   |        |
      | (used  |<---------------------|(tempor.|        |
      |  for TX)|   issue clears       | avoid) |        |
      +---+----+                       +--------+        |
          |                                              |
          | persistent fault / controller               |
          v                                              |
      +--------------+   probe verifies  +-------------+ |
      | ASSUMED_BAD  |------------------>| (back to    | |
      | (excluded,   |   viability       |  GOOD)      |-+
      |  probeable)  |<------------------| via EV probe|
      +------+-------+   still bad        +-------------+
             |
             | controller hard-denylist
             v
      +--------------+
      | DENIED       |  (administratively excluded, strongest)
      +--------------+
^ Fig 6: EV state machine. GOOD is the only transmit-eligible state.
  ASSUMED_BAD is recoverable -- an EV probe (Sec 4.5) can promote it
  back to GOOD -- whereas DENIED is an administrative hard exclusion
  set by the controller. This gives the datapath autonomy to route
  around flaky links while leaving the operator a hard override.

4.2 Per-packet spraying under the MPR window

Spraying without a bound would exhaust responder buffers and explode requester retransmit state. The Maximum PSN Range is the governor: the requester may not send a packet whose sequence number lies beyond the upper edge of the responder's advertised packet-tracker bitmap. Both endpoints hold a PSN-fidelity, MPR-sized bitmap; the responder's cumulative-ACK progress slides the window forward.

  START: requester has WRITE/WriteIMM work queued
     |
     v
  (1) pick next PSN; is PSN <= responder window upper edge (MPR)?
     |                                   |
     | yes                               | no -> STALL until SACK
     v                                   |     advances the window
  (2) select a GOOD EV (rotate per pkt)  |
     |                                   |
     v                                   |
  (3) emit request pkt (sprayed) --------+
      BTH(rtx=0) -> METH? -> [TSETH] -> RETH -> [ImmDt]
     |
     v
  (4) responder places data OUT OF ORDER, marks bitmap[PSN]=1
     |
     v
  (5) responder emits SACK: cumulative ACK + bitmap offset + bitmask
     |                       + CC_STATE (ECN, bytes, penalties)
     |                       + reflected forward-path EVs
     v
  (6) requester: advance MPR window on cumulative ACK;
      feed NSCC window; update per-EV health; free retransmit state
     |
     v
  (7) any gap in bitmap?  yes -> retransmit EARLIEST missing PSN
      on a higher-priority DSCP class (rtx=1); no -> continue
     |
     v
  END (loop)
^ Fig 7: Spraying + bounded-flight control flow. MPR converts an
  unbounded reordering problem into a fixed-size bitmap problem. The
  requester always retransmits the OLDEST gap first, on a prioritized
  traffic class, to minimize head-of-line blocking at the responder.

4.3 Loss recovery — SACK vs. NACK vs. trimmed packet

MRC has three loss signals of increasing speed and increasing infrastructure cost. A SACK reports gaps discovered by reordering; a NACK is a proactive non-delivery signal from a deterministic event; a trimmed packet is a switch-generated, near-instant loss signal that bypasses timers entirely.

  Requester            Fabric/Switch          Responder
     |                     |                      |
  (1)|== data PSN=n =======|=====================>| place, bitmap[n]=1
     |                     |                      |
  (2)|== data PSN=n+1 ==== X  (congestive drop)   |   gap at n+1
     |               switch TRIMS to headers,      |
     |               forwards on high-prio class   |
     |                     |==== hdr(n+1) ========>| NACK from header
     |<==== NACK(NETH) n+1 =======================| (fast, no timeout)
     |                     |                      |
  (3)|== rtx PSN=n+1 ======|=====================>| fills gap
     |   (rtx=1, hi-prio DSCP)                     |
     |                     |                      |
  (4)|<==== SACK: cumACK=n+1, bitmap advances ====| window slides
     |                     |                      |
^ Fig 8: Three-tier loss recovery. Path (2) shows trimmed-packet fast
  recovery: instead of waiting for an ACK timeout, the switch truncates
  the dropped packet to its headers and forwards them, letting the
  responder emit a NACK almost immediately. Retransmits ride a
  higher-priority DSCP class so recovery is not queued behind new data.

If trimming is unavailable, the same gap is eventually caught by the Linear + Exponential ACK timeout (linear resolution first, then exponential backoff) or by an optional per-packet timer, and recovery proceeds via the slower timeout path. Reliability probes provide a fourth, proactive route: the requester can fire a lightweight probe (PETH) that elicits a standard SACK without consuming PSNs or altering connection state, auditing delivery health independently of any timeout.

4.4 NSCC congestion-control loop (data flow)

NSCC is sender-based and window-driven. Its distinguishing move is that it reads everything from the SACK's CC_STATE sub-header rather than from a separate telemetry channel: forward-path ECN markings, cumulative bytes received, responder-side window penalties, reflected forward-path EVs, and (optionally) service-time compensation to subtract responder host overhead from measured RTT.

  +-------------------+  == data (ECN-eligible) ==>  +----------------+
  | Requester         |                              | Responder      |
  | NSCC window W      |                              | - mark ECN     |
  |                   |                              | - count bytes  |
  | reads from SACK:  |  <== SACK(SETH+CC_STATE) ==  | - measure svc  |
  |  * ECN fraction   |     * ECN marks (fwd path)   |   time         |
  |  * RTT (ts / svc  |     * cumulative bytes       | - reflect EVs  |
  |    compensation)  |     * window penalty         | - host bp flag |
  |  * host backpress.|     * reflected fwd EVs      |                |
  |  * per-EV ECN     |                              |                |
  +---------+---------+                              +----------------+
            |
            v
   adjust W (fair, high-fidelity) AND re-bias EV selection
   probabilities per path using per-EV ECN correlation
^ Fig 9: NSCC closed loop. One SACK carries both the congestion signal
  (ECN + RTT + host backpressure) and the load-balancing signal
  (per-path EV ECN), so the window controller and the path controller
  are clocked by the same feedback. Service-time compensation removes
  responder host latency so the RTT reflects the network, not the host.

4.5 Fast failover — endpoint-scoped, connection-independent

The resilience machinery deliberately sits outside any single connection. Endpoint operations are GID-scoped, use reserved QP identifier 0x2, and are processed statelessly (replies formed by swapping L2/L3 fields). One exchange updates state across all connections sharing that GID, amortizing per-connection cost and pulling reachability signaling down from control-plane convergence timescales to a single data-plane round trip.

  Node A                                   Node B
    |                                        |
 (1)|== Port Status Update (port_status_mask)=>| log degraded port;
    |   "my ports p2,p5 are down"              | avoid those planes
    |                                        |
 (2)|== EV Probe request (ERTH, GID-scoped) ==>| stateless: swap L2/L3
    |   "is this EV/path alive?"               |
    |<===== EV Probe response (EETH) =========| reply on QP 0x2
    |                                        |
 (3) probe OK  -> ASSUMED_BAD EV promoted to GOOD
     probe bad -> EV stays excluded, spray away from it
    |                                        |
    +-- effect applies to ALL QPs sharing the GID (one exchange) --+
^ Fig 10: Failover signaling on the stateless endpoint channel (QP
  0x2). Port Status Updates proactively announce local link failure;
  EV probes autonomously verify whether a degraded path is usable
  again. Because both are GID-scoped, a single exchange heals every
  connection between the two nodes at once -- no control-plane
  convergence, no per-QP re-negotiation.

5. Design Findings by Primitive Category

This paper reports no empirical measurements. It is explicit that MRC "is already in production" and that "a companion paper presents a full empirical evaluation including goodput, latency, and failure-recovery results [10]. This paper describes the transport itself." There are no throughput, flow-completion, buffer-occupancy, or loss numbers, no testbed, no ns-3 model, and no swept result tables. The single quantitative claim is motivational and qualitative: the performance of large synchronous AI training jobs is "strongly dictated by 100th-percentile (worst-case / tail) transfer performance," which sets the design target that individual link or fabric failures "should not cause QP failure, or even greatly impact the tail latency of transfers."

In place of a results-by-regime table, the paper's findings are best read as a map of which primitive addresses which structural weakness of RC. The three RC deficiencies MRC targets are: single-path with go-back-N retransmission; non-scalable PFC/DCQCN congestion control; and vulnerability to link/fabric failures with slow control-plane convergence.

  RC weakness                MRC primitive that answers it
  +-------------------------------------------------------------+
  | single-path            -> per-packet EV spraying +          |
  |                           multi-plane operation             |
  |                                                             |
  | go-back-N retransmit   -> SACK bitmap + NACK + trimmed-pkt  |
  |                           fast loss + earliest-gap rtx on    |
  |                           high-prio DSCP                     |
  |                                                             |
  | PFC/DCQCN won't scale  -> NSCC (window-based, SACK-clocked, |
  |                           ECN+RTT over best-effort Ethernet)|
  |                           + responder host backpressure     |
  |                                                             |
  | unbounded reorder from -> Maximum PSN Range window +        |
  |   spraying                out-of-order placement + Dynamic   |
  |                           MPR (elastic bitmap)              |
  |                                                             |
  | slow failover          -> EV probes + Port Status Update on |
  |   (control-plane)         stateless GID-scoped QP 0x2       |
  +-------------------------------------------------------------+
^ Fig 11: The "findings" of a spec paper are its problem-to-primitive
  mapping. Each RC weakness is met by a specific, co-designed
  primitive; the value claim (tail-latency resilience) is asserted by
  construction and validated elsewhere in the companion paper.

The one place the paper does argue a concrete performance mechanism is active end-to-end EV load balancing: because path selection is driven end-to-end by per-EV ECN feedback, MRC claims it can steer traffic away from congested or flaky links "even beyond the point on the path where switches no longer have any alternative paths to the destination." That is a statement about where the control authority lives (the endpoints, correlating ECN to paths) rather than a measured number, but it is the sharpest performance argument in the paper.

6. Configuration-Regime Trade-off Tables

The verdict column below records MRC's own design choice and the rationale the paper gives — this is a specification, so the "winner" is the option the authors selected, not a measured outcome.

6.1 Transport strategy — clean-slate vs. incremental extension

Dimension	Clean-slate transport	Extend RoCEv2 RC	MRC's choice
Deployability	Low (new HW/SW)	High (reuse model)	Extend RC
Verbs / app compatibility	Broken	Preserved	Extend RC
Design freedom	Maximal	Constrained by RC	(accepted cost)
Interop with RC	None	None (0101 prefix)	tie (both break RC)
Hardware reuse	Minimal	High	Extend RC

MRC extends RC because deployability and preserving the software/completion model outweigh the freedom of a clean slate; the cost accepted is that RC and MRC endpoints are non-interoperable.

6.2 Reordering vs. state — the MPR knob

Dimension	Small MPR	Large MPR	MRC's choice
In-flight parallelism	Low	High	large, bounded
Responder bitmap memory	Low	High	Dynamic MPR (elastic)
Requester rtx state	Low	High	bounded by MPR
Reordering tolerated	Low	High	enough for spraying
Multi-QP multiplexing	Rigid	Wasteful if static	Dynamic MPR reclaims

MRC's choice is bounded-but-elastic: MPR caps reordering and state, while optional Dynamic MPR resizes the bitmap at runtime via SACK and reclaims capacity from idle QPs — trading a little signaling complexity for large memory savings versus worst-case static provisioning.

6.3 Loss-signal latency vs. infrastructure dependence

Dimension	ACK timeout	Reliability probe	Trimmed-packet NACK	MRC's choice
Detection latency	Slowest	On-demand	Fastest	all three (tiered)
Fabric support needed	None	None	Switch trimming	tiered by fabric
State cost	Low	None (no PSN)	Low	probe is cheapest
Proactive vs reactive	Reactive	Proactive	Near-instant	mix

MRC ships a tiered stack: mandatory linear+exponential timeout as the floor, optional probes for proactive auditing, optional trimmed-packet NACKs for fast recovery where switches support truncation. The faster the tier, the more fabric cooperation it requires.

6.4 Congestion control — lossless fabric vs. best-effort

Dimension	PFC / DCQCN (lossless)	NSCC (best-effort)	MRC's choice
Scales to frontier size	No	Yes	NSCC
Requires PFC config	Yes (fragile)	No	NSCC
Signal source	Fabric backpressure	ECN + RTT (in SACK)	NSCC
Host contention handling	External	Responder host bp	NSCC
Per-path awareness	None	Reflected EV per-path	NSCC

MRC chooses NSCC over best-effort Ethernet precisely because PFC/DCQCN does not scale to frontier clusters; NSCC rides the SACK it already sends and adds per-path ECN correlation plus host backpressure that lossless schemes lack.

6.5 Failover authority — control plane vs. data plane

Dimension	Control-plane converge	Endpoint ops (QP 0x2)	MRC's choice
Reaction time	Seconds (convergence)	One RTT	endpoint ops
Per-connection cost	High	Amortized (GID-scoped)	endpoint ops
Connection context req'd	Yes	No (stateless)	endpoint ops
Scope of one exchange	Fabric-wide reroute	All QPs sharing GID	endpoint ops

MRC pushes failover into the datapath: EV probes and Port Status Updates on the stateless, GID-scoped endpoint channel heal every connection between two nodes in a single round trip, bypassing slow control-plane convergence entirely.

7. Bottlenecks & Insights Surfaced by the Design

The paper's central premise is that the network is the key bottleneck in large-scale AI/ML training, and that as models and clusters grow the binding constraint is not average throughput but tail transfer performance. A synchronous training step cannot complete until its slowest transfer completes, so a single congested or failed link taxes the entire job through the 100th-percentile transfer. This reframing — optimize the tail, not the mean — is what motivates every resilience primitive in the design.

The sharpest architectural insight is that control-plane convergence is too slow to be the failover mechanism. MRC's answer is to relocate reachability and liveness signaling into the datapath at RTT timescales via stateless, GID-scoped endpoint operations. The same instinct — do it at the endpoints, at line rate, without connection context — recurs across the design: EV load balancing is end-to-end (not hop-local), so it can steer around congestion even past the last switch that still has alternative paths; loss signaling via trimmed packets is near-instant rather than timeout-gated; and one SACK carries delivery, congestion, and path-health telemetry at once.

A second insight is the separation of packet-fidelity from semantic-fidelity bounds. Maximum PSN Range governs raw packet buffering and reordering; Maximum WriteIMM inflight governs concurrent semantic operations. Keeping these independent lets the transport spray aggressively at the packet level while still bounding the responder's semantic-processing queue — two different resources, two different limits, no conflation.

The design also surfaces the head-of-line blocking risk that spraying creates and answers it directly: responders prioritize reporting the oldest missing PSNs and requesters retransmit the earliest gaps first, both on a higher-priority DSCP traffic class. Reordering is embraced (out-of-order placement) but its worst consequence — a stalled cumulative ACK — is attacked by always healing the front of the window first.

Finally, the responder host is treated as a first-class congestion source. Rather than assume the network is the only place packets queue, MRC surfaces responder-side memory-subsystem contention on the wire (host backpressure) so NSCC can modulate the sender window before host contention degrades end-to-end performance. This closes a loop that fabric-only congestion control (PFC/DCQCN) leaves open.

8. Limitations of the Methodology

Because this is a specification paper rather than a measurement study, its limitations are limitations of evidence and scope rather than of a testbed.

Limitation	Consequence for the reader
No empirical evaluation in this paper	All performance claims are by design argument;
	goodput/latency/failover numbers live in [10]
Most CC & resilience primitives are Optional	Real behavior varies widely by implementation
Several primitives need fabric support	Trimmed packets need switch trimming; multi-plane
	needs multi-port NICs; SRv6 needs SRv6 fabric
RC and MRC are non-interoperable (0101)	Deployment is all-or-nothing per connection
Header/field layouts deferred to spec [9]	Paper is a summary; exact wire format elsewhere
Load-balancing algorithm implementation-def.	No canonical path-selection policy to compare
CC algorithm choice is pluggable	NSCC is a default, not the only option
LLMs used for language editing (disclosed)	Prose provenance noted by the authors

The most consequential limitation is the first: the paper argues correctness and resilience by construction and defers all quantitative validation to the companion paper. A reader cannot, from this document alone, judge how much tail latency MRC actually recovers, how fast failover is in wall-clock terms, or how NSCC compares numerically to DCQCN. The second limitation compounds the first — because the primitives that most affect performance (NSCC, host backpressure, trimmed-packet fast loss, EV denylisting, EV probes) are all Optional, two conforming MRC implementations can behave very differently, and the mandatory core alone (spray + bound + SACK/NACK + timeout + DSCP) is reliable and multipath but not necessarily fast or self-healing.

9. Note on NCCL Tuning

MRC sits below a collective library, so it changes the ground rules that collective-level configuration assumes. Today, using many communication channels per collective is partly a way to buy path diversity — several queue pairs hash onto several fabric paths, so more channels spreads load and hedges against a bad link. MRC provides that path diversity inside a single connection via per-packet spraying and out-of-order placement, which weakens the case for inflating channel counts purely to spread traffic and shifts the remaining channel-count decision toward host-side parallelism (kernel occupancy, PCIe/NIC posting) rather than network hedging. Likewise, MRC's own bounded-flight window and window-driven congestion control overlap with the message-chunking and pipelining that collective libraries tune for throughput, so those knobs should be co-tuned with, not on top of, an assumption of single-path in-order delivery. The non-obvious point is that a transport which absorbs reordering and reroutes around failures autonomously removes some of the burden that channel-count and chunk-size tuning have historically carried at the collective layer.

10. Analogy

MRC is a fleet of parcel couriers serving one customer, coordinated by a single tracking receipt. A traditional Reliable Connection is one courier who always drives the same road and, if a package goes missing, restarts the entire day's deliveries from that package onward (go-back-N, single path). MRC instead hands each parcel to whichever courier has a clear road right now (per-packet EV spraying), and lets them arrive at the loading dock in any order because the dock sorts them later (out-of-order placement). To keep the dock from overflowing, the customer will only release as many parcels as the dock has numbered shelves open (the Maximum PSN Range window), and it reclaims shelves from quiet customers when busy ones need them (Dynamic MPR). The customer's one signed receipt does five jobs at once: it lists exactly which parcels arrived and which are missing (SACK bitmap), notes how crowded the roads were (ECN), how long the round trip took minus the time spent inside the warehouse (RTT with service-time compensation), whether the dock itself is jammed (host backpressure), and which roads each courier actually used (reflected EVs) — so the customer can both slow down and reroute future parcels from that single receipt. When a road washes out, nobody waits for the city traffic authority to redraw the map (control-plane convergence); instead the couriers themselves post a quick "road p2 is out" notice and send a scout down questionable roads to see if they have reopened (Port Status Update and EV probes on the stateless endpoint channel), and that one notice updates every route the two depots share at once (GID-scoped). The whole scheme runs on ordinary public roads with no reserved lanes (best-effort Ethernet, no PFC) — the couriers achieve reliable, timely delivery not by commandeering the road but by spreading out, sorting at the dock, and reading a single, very informative receipt.