CXL memory expansion
CXL memory expansion adds device-attached memory to a host's address space for coherent access through CXL.mem.
Type-3 expanders use CXL.io and CXL.mem, not the device-originated CXL.cache role. The host can still cache their memory. Capacity may be volatile, persistent, or both; coherent access is not a power-loss survival guarantee. [9] [8]
Linux 6.17 offers application-managed mmap() access through /dev/daxN.Y, or dax_kmem conversion into hotplug blocks for the kernel page allocator. Firmware, kernel-build and boot settings can also expose system RAM during early boot. NUMA association alone does not establish page-allocator availability. [6] [7]
Possible NUMA nodes are identified during early boot; memory tiers group nodes with online memory by performance. HMAT/CDAT information can distinguish their tier from the default DRAM tier. ZONE_MOVABLE excludes most kernel allocations. [7]
A 2023 study evaluated three expanders individually on one platform. In memo's batched temporal-load test, latency was 35 percent above remote-socket DDR5 for one device and about 2 and 3 times its latency for the others. All-read bandwidth efficiencies were 70, 46, 47 and 20 percent of theoretical bandwidth for remote DDR5 and the three devices. With all pages on the first expander, a key-value workload with equal reads and updates had p99 latency 10, 73 and 105 percent above all-local-DDR allocation at 25,000, 45,000 and 85,000 target queries per second. [5]
CXL 2.0 pooling permits each physical memory location to be mapped to at most one host at a time. CXL 3.0 supports shared regions with software-managed or multi-host hardware coherence; sharing does not automatically select the hardware-coherent model. Neither model replaces application synchronization or guarantees crash consistency. Check system-wide feature support. [11] [12] [14]
Relations
- related
- PCIe storage attachment, Storage-class memory (SCM), Byte-addressable storage, PCIe lanes (storage), CXL Type-3 device, Host-managed device memory decoder, Persistent memory, Direct Access, Device DAX, Pond, Memory stranding, CXL memory pooling, CXL memory sharing
Limitations
- In the 2023 study, Caption improved throughput of the mcf and roms mix by 24 percent over the better of the tested all-DDR and 50:50 DDR:CXL static policies. A separate DLRM result was 4 percent lower than the 50:50 policy. The figure's normalization and the textual best-baseline comparison are distinct; the gain is workload-specific, not universal. [5]
- In Linux 6.17, when global and per-device memmap_on_memory are not both enabled, a large movable pool needs local ZONE_NORMAL capacity for its memory map. Enabling both reserves hotplugged capacity for metadata; high-latency metadata access can have performance costs. [13]
- For Linux 6.17 hotplug, ZONE_NORMAL capacity should be treated as permanently attached to the allocator. ZONE_MOVABLE helps retain removability, but successful offlining is not guaranteed. DAX-added capacity is driver-managed and excluded from kexec placement because a device reset can overwrite the new kernel; this does not guarantee application data retention. [13]
- On the study's Sapphire Rapids platform with four-way sub-NUMA clustering, a 32 MB buffer fit the socket-wide cache but exceeded one node's cache. MLC reported 41 ns for CXL-A placement versus 76.8 ns for local DDR placement. These are cache-mediated observations, not isolated off-chip memory latency; other active SNC nodes could reduce the cache advantage. [5]
Citations
- Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices, Y. Sun et al., arXiv:2303.15375v4, 2023
- Linux 6.17: DAX Driver Operation, Device DAX and kmem conversion.
- Linux 6.17: Linux Init, early-boot RAM, NUMA nodes and memory tiers.
- Linux 6.17: Devices and Protocols, CXL.mem and Type-3 capacity.
- CXL Consortium: Introducing CXL 3.1, webinar Q&A (2024), traditional device types.
- CXL Consortium: CXL 2.0 memory pooling, webinar Q&A, part 2 (2021).
- CXL Consortium: Compute Express Link 3.0 white paper, page 3, shared memory.
- Linux 6.17: Memory Hotplug, memory-map placement and enabling settings.
- CXL Consortium: Flash Memory Summit tutorial (August 2023), slides 27-28; pooled/shared FAM and two coherence models.
Further reading
- CXL memory expansion
- Compute Express Link: Devices and Protocols, The Linux Kernel documentation
- Compute Express Link: Linux Init (Early Boot), The Linux Kernel documentation
- Compute Express Link: Memory Hotplug, The Linux Kernel documentation
- CXL Consortium: Supporting Persistent Memory, webinar Q&A (2021).
Sources & caveats
- This record uses CXL Consortium explainers and version-pinned Linux 6.17 documentation, not a direct review of the normative CXL specification. Legacy citation identifiers remain stable.
- The kernel documentation was written by kernel developers, some employed by hardware makers.
- The measurement study is by two universities with two co-authors at a processor maker lab, so it is not fully vendor-independent.
- Device and platform results come from one 2023 platform generation with three particular devices; PCIe 4.0 against DDR4 latency and energy figures are quoted by the study from other works.
- Arithmetic: 22 / 6 is about 3.7 for the cited DDR4-to-PCIe-4.0 energy-per-bit ratio, not the reverse. These are cited background estimates, not this study's measurements. [5]
- No cost, price or market claims are made.
- Architecture and OS exposure are scoped to Linux 6.17 documentation, not an October 2026 support matrix. The 2023 benchmark is version-pinned to arXiv v4 and its tested platform, not generalized to current CXL devices.
- The limits of coherency and NUMA association are engineering inferences from the separate access, allocation and persistence mechanisms, not measured durability results.
- The v4 introduction summarizes tail-latency increases as 10-82 percent, while section 5.1/Figure 6a reports Redis YCSB-A p99 increases of 10, 73 and 105 percent. The note follows the detailed experiment. [5]
- The PCIe per-lane and DDR per-pin rates quoted in section 1 use different denominators; dividing 16 by 3.2 does not establish a like-for-like per-pin advantage. [5]