NVMe over Fabrics
An extension that carries NVMe commands across RDMA, Fibre Channel, or TCP networks to remote namespaces.
Architecture: NVMe over Fabrics separates NVMe command semantics from network transport. TCP, RDMA and Fibre Channel use distinct bindings rather than one interchangeable wire format [2] [3] [4].
Discovery: A discovery controller supplies connection information, not a namespace data path. Namespace identity and path selection remain separate concerns [5] [7].
Operational scope: Transport choice alone does not establish encryption, failover time or application durability. Check security configuration and recovery behavior for the actual host and target stack [6].
Specification details: A capsule carries a command or response and optionally data, with 64-byte command and 16-byte response capsules. Queues are created by Connect and deleted by Disconnect, SGLs replace PRPs, and there is no completion-queue flow control, so the host must keep completion slots free. [4]
Overhead: A study of remote flash over TCP, not using NVMe over Fabrics, measured 21 µs of added read latency for an optimised dataplane against more than 100 µs for a conventional server. [8]
Relations
- related
- NVMe, Fibre Channel, Block storage, ALUA, All-flash array, RDMA storage networking, NVMe/TCP, NVMe/RDMA, NVMe/FC, NVMe discovery controller, NVM subsystem, NVMe multipath, NVMe base specification, NVMe Keep Alive
Limitations
- A subsystem may require a fabric secure channel, in-band authentication or both, so access control depends on configuration. [4]
Citations
- NVM Express NVMe over TCP Transport 1.3, sections 2, 3.1.2, 3.2 and 3.6
- NVM Express NVMe over RDMA Transport 1.3, section 2
- NVM Express NVMe over Fabrics 1.1a, section 7.2: Fibre Channel transport
- NVM Express Base Specification 2.2, section 3.1.3.3: Discovery Controllers
- SPDK NVMe over Fabrics target: transport support, TLS and in-band authentication
- Linux native NVMe multipath: ANA states and path-selection policies
- Klimovic, Litz and Kozyrakis. ReFlex: Remote Flash = Local Flash. ASPLOS 2017.
Further reading
Sources & caveats
- TCP/RDMA details use transport revision 1.3; discovery uses Base 2.2. The 2021 FC mapping reference is historical. No hardware or performance measurements are claimed.
- The remote-flash study is university work on a different system over TCP; it shows the order of network overhead, not NVMe over Fabrics performance.
- Arithmetic: up to 65,535 I/O queues of 65,535 commands allow about 4.29 billion outstanding commands per controller, a protocol ceiling that no implementation is shown to reach.