nvme-of · shipping · also NVMe-oF, NVM Express over Fabrics

NVMe over Fabrics

An extension that carries NVMe commands across RDMA, Fibre Channel, or TCP networks to remote namespaces.

Related Observatory article: Read on hesela.com

Architecture: NVMe over Fabrics separates NVMe command semantics from network transport. TCP, RDMA and Fibre Channel use distinct bindings rather than one interchangeable wire format [2] [3] [4].

Discovery: A discovery controller supplies connection information, not a namespace data path. Namespace identity and path selection remain separate concerns [5] [7].

Operational scope: Transport choice alone does not establish encryption, failover time or application durability. Check security configuration and recovery behavior for the actual host and target stack [6].

Specification details: A capsule carries a command or response and optionally data, with 64-byte command and 16-byte response capsules. Queues are created by Connect and deleted by Disconnect, SGLs replace PRPs, and there is no completion-queue flow control, so the host must keep completion slots free. [4]

Overhead: A study of remote flash over TCP, not using NVMe over Fabrics, measured 21 µs of added read latency for an optimised dataplane against more than 100 µs for a conventional server. [8]

Relations

related
NVMe, Fibre Channel, Block storage, ALUA, All-flash array, RDMA storage networking, NVMe/TCP, NVMe/RDMA, NVMe/FC, NVMe discovery controller, NVM subsystem, NVMe multipath, NVMe base specification, NVMe Keep Alive

Limitations

  • A subsystem may require a fabric secure channel, in-band authentication or both, so access control depends on configuration. [4]

Citations

  1. NVM Express NVMe over TCP Transport 1.3, sections 2, 3.1.2, 3.2 and 3.6 (accessed 2026-10-17)
  2. NVM Express NVMe over RDMA Transport 1.3, section 2 (accessed 2026-10-17)
  3. NVM Express NVMe over Fabrics 1.1a, section 7.2: Fibre Channel transport (accessed 2026-10-17)
  4. NVM Express Base Specification 2.2, section 3.1.3.3: Discovery Controllers (accessed 2026-10-17)
  5. SPDK NVMe over Fabrics target: transport support, TLS and in-band authentication (accessed 2026-10-17)
  6. Linux native NVMe multipath: ANA states and path-selection policies (accessed 2026-10-17)
  7. Klimovic, Litz and Kozyrakis. ReFlex: Remote Flash = Local Flash. ASPLOS 2017. (accessed 2026-10-16)

Further reading

  1. NVMe over Fabrics (accessed 2026-03-26)
Sources & caveats
  • TCP/RDMA details use transport revision 1.3; discovery uses Base 2.2. The 2021 FC mapping reference is historical. No hardware or performance measurements are claimed.
  • The remote-flash study is university work on a different system over TCP; it shows the order of network overhead, not NVMe over Fabrics performance.
  • Arithmetic: up to 65,535 I/O queues of 65,535 commands allow about 4.29 billion outstanding commands per controller, a protocol ceiling that no implementation is shown to reach.

JSON: /data/terms/nvme-of.json

Published by Hesela team · Updated

Report a correction · Editorial standards · Reuse guidance

Cite this page
Hesela team. NVMe over Fabrics. Hesela. 2026-11-08. https://hesela.dev/terms/nvme-of/