hesela.devRoCEv2 congestion notification packet
A RoCEv2 congestion notification packet carries congestion feedback from a receiving endpoint to the sender after congestion-marked traffic arrives.
Direction: The receiving endpoint acts as a notification point, sending a CNP back toward the transmitting endpoint after receiving congestion-marked RoCEv2 traffic. The transmitting endpoint acts as a reaction point that reduces its sending rate. [1] [2]
Timing: The implementation and configured control algorithm determine the rate response. Feedback coalescing and rate-update intervals mean that a trace must not assume one CNP per marked packet or one rate reduction per CNP. [2] [3]
Scope: A CNP reports congestion; it is not a storage acknowledgement or proof that data reached persistent media. It differs from PFC, which pauses a selected Ethernet priority on a link. [4] [2]
Counters: Cumulus Linux 5.12 publishes interface transmit ECN and CNP counts and a counter-reset operation. Its displayed ECN label does not specify whether previously marked forwarded packets are included. Diagnostic recommendation: identify interface or queue-pair scope, direction, time interval, reset state and counter definition before comparing observed CE data packets, receiver-generated CNPs, sender-received CNPs and rate changes. A switch transmit CNP count is not by itself a count of CNPs generated by that switch. [1] [3]
Historical timing: The 2015 DCQCN deployment used a receiver-side CNP interval of 50 microseconds per flow, not a universal RoCE timer. [5]
Interoperability: Bai et al. report that a Gen1 sender receiving per-marked-packet CNPs from a Gen2/Gen3 receiver could make excessive rate reductions. Their generation labels describe that deployment, not every mixed-NIC deployment. Check both endpoints' coalescing policies. [3]
Relations
Citations
- NVIDIA Cumulus Linux 5.12: default configuration and RoCE lossy mode
- NVIDIA WinOF-2 25.10.50020: RoCEv2 Congestion Management and RP decrease
- Bai et al., Empowering Azure Storage with RDMA, NSDI 2023, section 7.2
- IEEE 802.1Qbb project scope: priority-based flow control
- Zhu et al., Congestion Control for Large-Scale RDMA Deployments, SIGCOMM 2015, section 3; DOI 10.1145/2785956.2787484
Sources & caveats
- Versioned implementation documentation and the NSDI 2023 Azure deployment report establish mechanism and feedback timing scope; no NIC experiment reproduced.
- The storage-completion distinction is an architectural inference from congestion feedback, not a measured durability result.
- RFC 3168 defines ECN, not DCQCN. WinOF-2's wording conflating the names is not adopted; only its notification/reaction-point description and timing controls are used.
- Counter comparison is a diagnostic recommendation, not a claim about undocumented hardware counter increment semantics or an executed network measurement.