rdma-storage · shipping

RDMA storage networking

RDMA storage networking uses remote direct memory access for data placement within storage protocols such as iSER and NVMe/RDMA.

Related Observatory article: Read on hesela.com

Layering: RDMA is a data-transfer facility, not a storage command set. iSER carries SCSI semantics; NVMe/RDMA carries NVMe semantics. Direct placement can avoid intermediate copies, but the protocol name does not prove a copy-free application path or persistent storage. [2] [3]

Providers: NVMe/RDMA supports reliable queue pairs over providers including InfiniBand, RoCE and iWARP. The storage binding and its network provider are separate choices. [2]

Addressing: TRSVCID is connection-management information, not a packet-filter recipe. RDMA Transport 1.3 describes a decimal TCP-port string used by RDMA IP CM, separately from RDMA endpoint addressing. Its default-port discussion mentions 4420; NVIDIA's RoCEv2 documentation identifies UDP destination port 4791 for packet encapsulation. Report the field and layer, not just a number. [2] [4]

Implementation example: Linux v6.12's NVMe/RDMA host creates an RDMA_PS_TCP reliable-connected queue pair. For IPv4/IPv6, rdma_get_service_id combines the port space with the supplied port. RDMA_PS_TCP does not mean that a RoCE provider becomes NVMe/TCP. This is a source inspection, not a packet capture or interoperability test. [5] [6]

Congestion policy: Do not assign one Ethernet flow-control requirement to all RDMA providers. Cumulus Linux 5.12, for example, offers ECN-only lossy RoCE and ECN-plus-PFC lossless RoCE modes; support and performance remain implementation-dependent. [7]

Relations

related
NVMe over Fabrics, Fibre Channel, SCSI initiator and target, iSCSI, iSCSI Extensions for RDMA (iSER), NVMe/RDMA, RDMA steering tag, RoCEv2, iWARP, InfiniBand, FaRM, Infiniswap, Clio, Clover, Hotpot, eRPC, Tebis

Advantages

  • Direct placement can reduce intermediate data-copy work in the storage transport. [2, 3]
  • The NVMe/RDMA binding can use several RDMA providers. [2]

Limitations

  • A compatible RDMA provider is required; the binding is not an ordinary TCP socket protocol. [2, 3]
  • Discovery service identifiers and RoCEv2 packet ports describe different layers; neither alone establishes connectivity or security. [2, 4]

Citations

  1. NVM Express NVMe over RDMA Transport 1.3 (31 July 2026), sections 2 and 3.1.2 (accessed 2026-10-21)
  2. RFC 7145: Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification, IETF, April 2014 (accessed 2026-10-16)
  3. NVIDIA MLNX_OFED 5.8-6.0.4.2 LTS: RoCE modes and RoCEv2 UDP encapsulation (accessed 2026-10-21)
  4. Linux v6.12 NVMe/RDMA host: nvme_rdma_alloc_queue and nvme_rdma_alloc_ctrl (accessed 2026-10-21)
  5. Linux v6.12 RDMA connection manager: rdma_get_service_id (accessed 2026-10-21)
  6. NVIDIA Cumulus Linux 5.12: RoCE lossy and lossless modes (accessed 2026-10-23)

Further reading

  1. RDMA storage networking (accessed 2026-03-26)
Sources & caveats
  • Reviewed storage binding, provider encapsulation and Linux source separately. No hardware, packet capture or benchmark was performed.

JSON: /data/terms/rdma-storage.json

Published by Hesela team · Updated

Report a correction · Editorial standards · Reuse guidance

Cite this page
Hesela team. RDMA storage networking. Hesela. 2026-11-16. https://hesela.dev/terms/rdma-storage/