{
  "status": "shipping",
  "advantages": [],
  "limitations": [],
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-11-03",
  "slug": "roce-congestion-notification-packet",
  "term": "RoCEv2 congestion notification packet",
  "aliases": [
    "RoCE CNP",
    "RoCEv2 CNP"
  ],
  "definition": "A RoCEv2 congestion notification packet carries congestion feedback from a receiving endpoint to the sender after congestion-marked traffic arrives.",
  "note": "Direction: The receiving endpoint acts as a notification point, sending a CNP back toward the transmitting endpoint after receiving congestion-marked RoCEv2 traffic. The transmitting endpoint acts as a reaction point that reduces its sending rate. [roce-qos] [roce-rcm]\n\nTiming: The implementation and configured control algorithm determine the rate response. Feedback coalescing and rate-update intervals mean that a trace must not assume one CNP per marked packet or one rate reduction per CNP. [roce-rcm] [azure-rdma]\n\nScope: A CNP reports congestion; it is not a storage acknowledgement or proof that data reached persistent media. It differs from PFC, which pauses a selected Ethernet priority on a link. [pfc] [roce-rcm]\n\nCounters: Cumulus Linux 5.12 publishes interface transmit ECN and CNP counts and a counter-reset operation. Its displayed ECN label does not specify whether previously marked forwarded packets are included. Diagnostic recommendation: identify interface or queue-pair scope, direction, time interval, reset state and counter definition before comparing observed CE data packets, receiver-generated CNPs, sender-received CNPs and rate changes. A switch transmit CNP count is not by itself a count of CNPs generated by that switch. [roce-qos] [azure-rdma]\n\nHistorical timing: The 2015 DCQCN deployment used a receiver-side CNP interval of 50 microseconds per flow, not a universal RoCE timer. [dcqcn]\n\nInteroperability: Bai et al. report that a Gen1 sender receiving per-marked-packet CNPs from a Gen2/Gen3 receiver could make excessive rate reductions. Their generation labels describe that deployment, not every mixed-NIC deployment. Check both endpoints' coalescing policies. [azure-rdma]",
  "related": [
    "roce-v2",
    "explicit-congestion-notification"
  ],
  "citations": [
    {
      "id": "roce-qos",
      "title": "NVIDIA Cumulus Linux 5.12: default configuration and RoCE lossy mode",
      "url": "https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-512/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE/",
      "accessed": "2026-10-31"
    },
    {
      "id": "roce-rcm",
      "title": "NVIDIA WinOF-2 25.10.50020: RoCEv2 Congestion Management and RP decrease",
      "url": "https://networking-docs.nvidia.com/winof2driverum/251050020/ethernet-network",
      "accessed": "2026-10-31"
    },
    {
      "id": "azure-rdma",
      "title": "Bai et al., Empowering Azure Storage with RDMA, NSDI 2023, section 7.2",
      "url": "https://www.usenix.org/system/files/nsdi23-bai.pdf",
      "accessed": "2026-11-03"
    },
    {
      "id": "pfc",
      "title": "IEEE 802.1Qbb project scope: priority-based flow control",
      "url": "https://1.ieee802.org/dcb/802-1qbb/",
      "accessed": "2026-10-31"
    },
    {
      "id": "dcqcn",
      "title": "Zhu et al., Congestion Control for Large-Scale RDMA Deployments, SIGCOMM 2015, section 3; DOI 10.1145/2785956.2787484",
      "url": "https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/lossless.pdf",
      "accessed": "2026-11-03"
    }
  ],
  "provenance": [
    "Versioned implementation documentation and the NSDI 2023 Azure deployment report establish mechanism and feedback timing scope; no NIC experiment reproduced.",
    "The storage-completion distinction is an architectural inference from congestion feedback, not a measured durability result.",
    "RFC 3168 defines ECN, not DCQCN. WinOF-2's wording conflating the names is not adopted; only its notification/reaction-point description and timing controls are used.",
    "Counter comparison is a diagnostic recommendation, not a claim about undocumented hardware counter increment semantics or an executed network measurement."
  ]
}