RDMA storage networking
RDMA storage networking uses remote direct memory access for data placement within storage protocols such as iSER and NVMe/RDMA.
Layering: RDMA is a data-transfer facility, not a storage command set. iSER carries SCSI semantics; NVMe/RDMA carries NVMe semantics. Direct placement can avoid intermediate copies, but the protocol name does not prove a copy-free application path or persistent storage. [2] [3]
Providers: NVMe/RDMA supports reliable queue pairs over providers including InfiniBand, RoCE and iWARP. The storage binding and its network provider are separate choices. [2]
Addressing: TRSVCID is connection-management information, not a packet-filter recipe. RDMA Transport 1.3 describes a decimal TCP-port string used by RDMA IP CM, separately from RDMA endpoint addressing. Its default-port discussion mentions 4420; NVIDIA's RoCEv2 documentation identifies UDP destination port 4791 for packet encapsulation. Report the field and layer, not just a number. [2] [4]
Implementation example: Linux v6.12's NVMe/RDMA host creates an RDMA_PS_TCP reliable-connected queue pair. For IPv4/IPv6, rdma_get_service_id combines the port space with the supplied port. RDMA_PS_TCP does not mean that a RoCE provider becomes NVMe/TCP. This is a source inspection, not a packet capture or interoperability test. [5] [6]
Congestion policy: Do not assign one Ethernet flow-control requirement to all RDMA providers. Cumulus Linux 5.12, for example, offers ECN-only lossy RoCE and ECN-plus-PFC lossless RoCE modes; support and performance remain implementation-dependent. [7]
Relations
- related
- NVMe over Fabrics, Fibre Channel, SCSI initiator and target, iSCSI, iSCSI Extensions for RDMA (iSER), NVMe/RDMA, RDMA steering tag, RoCEv2, iWARP, InfiniBand, FaRM, Infiniswap, Clio, Clover, Hotpot, eRPC, Tebis
Advantages
- Direct placement can reduce intermediate data-copy work in the storage transport. [2, 3]
- The NVMe/RDMA binding can use several RDMA providers. [2]
Limitations
- A compatible RDMA provider is required; the binding is not an ordinary TCP socket protocol. [2, 3]
- Discovery service identifiers and RoCEv2 packet ports describe different layers; neither alone establishes connectivity or security. [2, 4]
Citations
- NVM Express NVMe over RDMA Transport 1.3 (31 July 2026), sections 2 and 3.1.2
- RFC 7145: Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification, IETF, April 2014
- NVIDIA MLNX_OFED 5.8-6.0.4.2 LTS: RoCE modes and RoCEv2 UDP encapsulation
- Linux v6.12 NVMe/RDMA host: nvme_rdma_alloc_queue and nvme_rdma_alloc_ctrl
- Linux v6.12 RDMA connection manager: rdma_get_service_id
- NVIDIA Cumulus Linux 5.12: RoCE lossy and lossless modes
Further reading
Sources & caveats
- Reviewed storage binding, provider encapsulation and Linux source separately. No hardware, packet capture or benchmark was performed.