cxl-memory · shipping · also CXL-attached memory

CXL memory expansion

CXL memory expansion adds device-attached memory to a host's address space for coherent access through CXL.mem.

Related Observatory article: Read on hesela.com

Type-3 expanders use CXL.io and CXL.mem, not the device-originated CXL.cache role. The host can still cache their memory. Capacity may be volatile, persistent, or both; coherent access is not a power-loss survival guarantee. [9] [8]

Linux 6.17 offers application-managed mmap() access through /dev/daxN.Y, or dax_kmem conversion into hotplug blocks for the kernel page allocator. Firmware, kernel-build and boot settings can also expose system RAM during early boot. NUMA association alone does not establish page-allocator availability. [6] [7]

Possible NUMA nodes are identified during early boot; memory tiers group nodes with online memory by performance. HMAT/CDAT information can distinguish their tier from the default DRAM tier. ZONE_MOVABLE excludes most kernel allocations. [7]

A 2023 study evaluated three expanders individually on one platform. In memo's batched temporal-load test, latency was 35 percent above remote-socket DDR5 for one device and about 2 and 3 times its latency for the others. All-read bandwidth efficiencies were 70, 46, 47 and 20 percent of theoretical bandwidth for remote DDR5 and the three devices. With all pages on the first expander, a key-value workload with equal reads and updates had p99 latency 10, 73 and 105 percent above all-local-DDR allocation at 25,000, 45,000 and 85,000 target queries per second. [5]

CXL 2.0 pooling permits each physical memory location to be mapped to at most one host at a time. CXL 3.0 supports shared regions with software-managed or multi-host hardware coherence; sharing does not automatically select the hardware-coherent model. Neither model replaces application synchronization or guarantees crash consistency. Check system-wide feature support. [11] [12] [14]

Relations

related
PCIe storage attachment, Storage-class memory (SCM), Byte-addressable storage, PCIe lanes (storage), CXL Type-3 device, Host-managed device memory decoder, Persistent memory, Direct Access, Device DAX, Pond, Memory stranding, CXL memory pooling, CXL memory sharing

Limitations

  • In the 2023 study, Caption improved throughput of the mcf and roms mix by 24 percent over the better of the tested all-DDR and 50:50 DDR:CXL static policies. A separate DLRM result was 4 percent lower than the 50:50 policy. The figure's normalization and the textual best-baseline comparison are distinct; the gain is workload-specific, not universal. [5]
  • In Linux 6.17, when global and per-device memmap_on_memory are not both enabled, a large movable pool needs local ZONE_NORMAL capacity for its memory map. Enabling both reserves hotplugged capacity for metadata; high-latency metadata access can have performance costs. [13]
  • For Linux 6.17 hotplug, ZONE_NORMAL capacity should be treated as permanently attached to the allocator. ZONE_MOVABLE helps retain removability, but successful offlining is not guaranteed. DAX-added capacity is driver-managed and excluded from kexec placement because a device reset can overwrite the new kernel; this does not guarantee application data retention. [13]
  • On the study's Sapphire Rapids platform with four-way sub-NUMA clustering, a 32 MB buffer fit the socket-wide cache but exceeded one node's cache. MLC reported 41 ns for CXL-A placement versus 76.8 ns for local DDR placement. These are cache-mediated observations, not isolated off-chip memory latency; other active SNC nodes could reduce the cache advantage. [5]

Citations

  1. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices, Y. Sun et al., arXiv:2303.15375v4, 2023 (accessed 2026-10-29)
  2. Linux 6.17: DAX Driver Operation, Device DAX and kmem conversion. (accessed 2026-10-27)
  3. Linux 6.17: Linux Init, early-boot RAM, NUMA nodes and memory tiers. (accessed 2026-10-27)
  4. Linux 6.17: Devices and Protocols, CXL.mem and Type-3 capacity. (accessed 2026-10-27)
  5. CXL Consortium: Introducing CXL 3.1, webinar Q&A (2024), traditional device types. (accessed 2026-10-27)
  6. CXL Consortium: CXL 2.0 memory pooling, webinar Q&A, part 2 (2021). (accessed 2026-10-27)
  7. CXL Consortium: Compute Express Link 3.0 white paper, page 3, shared memory. (accessed 2026-10-27)
  8. Linux 6.17: Memory Hotplug, memory-map placement and enabling settings. (accessed 2026-10-29)
  9. CXL Consortium: Flash Memory Summit tutorial (August 2023), slides 27-28; pooled/shared FAM and two coherence models. (accessed 2026-11-16)

Further reading

  1. CXL memory expansion (accessed 2026-03-26)
  2. Compute Express Link: Devices and Protocols, The Linux Kernel documentation (accessed 2026-10-16)
  3. Compute Express Link: Linux Init (Early Boot), The Linux Kernel documentation (accessed 2026-10-16)
  4. Compute Express Link: Memory Hotplug, The Linux Kernel documentation (accessed 2026-10-16)
  5. CXL Consortium: Supporting Persistent Memory, webinar Q&A (2021). (accessed 2026-10-27)
Sources & caveats
  • This record uses CXL Consortium explainers and version-pinned Linux 6.17 documentation, not a direct review of the normative CXL specification. Legacy citation identifiers remain stable.
  • The kernel documentation was written by kernel developers, some employed by hardware makers.
  • The measurement study is by two universities with two co-authors at a processor maker lab, so it is not fully vendor-independent.
  • Device and platform results come from one 2023 platform generation with three particular devices; PCIe 4.0 against DDR4 latency and energy figures are quoted by the study from other works.
  • Arithmetic: 22 / 6 is about 3.7 for the cited DDR4-to-PCIe-4.0 energy-per-bit ratio, not the reverse. These are cited background estimates, not this study's measurements. [5]
  • No cost, price or market claims are made.
  • Architecture and OS exposure are scoped to Linux 6.17 documentation, not an October 2026 support matrix. The 2023 benchmark is version-pinned to arXiv v4 and its tested platform, not generalized to current CXL devices.
  • The limits of coherency and NUMA association are engineering inferences from the separate access, allocation and persistence mechanisms, not measured durability results.
  • The v4 introduction summarizes tail-latency increases as 10-82 percent, while section 5.1/Figure 6a reports Redis YCSB-A p99 increases of 10, 73 and 105 percent. The note follows the detailed experiment. [5]
  • The PCIe per-lane and DDR per-pin rates quoted in section 1 use different denominators; dividing 16 by 3.2 does not establish a like-for-like per-pin advantage. [5]

JSON: /data/terms/cxl-memory.json

Published by Hesela team · Updated

Report a correction · Editorial standards · Reuse guidance

Cite this page
Hesela team. CXL memory expansion. Hesela. 2026-11-16. https://hesela.dev/terms/cxl-memory/