This is the top-level repository of the CXL Memory Resource Kit (CMRK) which is a collection of documentation, tools, and benchmarks for use with CXL memory.
The CMRK comprises both original content and CXL-related enhancements to external projects that we plan to submit to upstream where appropriate. Our goal is to expand the CMRK over time, in collaboration with the community, to help enable applications to take full advantage of CXL memory.
When making CXL-related enhancements to external projects, we source those repositories rather than flattening the code. This makes it easy to see what we modified, and also simplifies the process of submitting patches to upstream maintainers.
The CMRK documentation and code is currently focused on Linux only.
We recommend the following system configuration to use CXL memory:
- A CPU and system BIOS that supports CXL memory
- The latest release of Fedora or Ubuntu LTS
- Currently: Fedora 36 or Ubuntu 22.04.1 LTS
- The latest Linux kernel package available for the distro
- Upgrade on Fedora:
dnf update kernel - Upgrade on Ubuntu:
apt upgrade linux-base
- Upgrade on Fedora:
- The
daxctlpackage installed on your system (version 72 or later) - The
numactlpackage installed on your system
Clone this repository and run the bootstrap.sh script to fetch the external content:
./bootstrap.sh- Examining Your CXL Memory
- About Kernel Support for CXL Memory
- About Raw Command Support
- About BIOS Support for Specific Purpose Memory
- Enabling Specific Purpose Memory
- About NDCTL and DAXCTL
- Using CXL Memory as a DAX Device
- Using CXL Memory as System RAM
- Converting Between Device DAX and System RAM Modes
- Testing CXL Memory
- About MXCLI
The cxlstat tool in the root directory of this repository is intended to tell you everything you
might need to know about a system configured with CXL memory, for example:
- Does your kernel contain support for CXL?
- Are there CXL memory devices present in your system?
- How is your CXL memory currently configured?
$ sudo ./cxlstat
System booted using UEFI
Detected Ubuntu
Minimum kernel version requirement met - 5.19.0-21-generic vs. 5.15
Kernel boot configuration located (/boot/config-5.19.0-21-generic)
The package daxctl is installed and at a sufficient version (found 74.0)
Detecting CXL devices ...
CXL Device : mem0
Vendor : Micron Technology Inc
Device ID : Device 6400
Address : 0000:7f:00.0
Size : 128 GB
Mailbox Present : yes
Detecting DAX devices ...
Name : dax0.0
Path : /dev/dax0.0
Size : 128 GB
Memory Range : 0x1050000000-0x304FFFFFFF (128 GB)
Mode : devdax (memory directly usable only via DAX access)
The daxctl command appears usable for all normal functions.
Standard CXL mailbox commands may be used (no vendor-specific commands allowed).
The cxlstat tool is intended both as a useful way of checking your configuration, and as an example of how to
do so in your own scripts or programs.
Important note: If you run cxlstat as root (or via sudo), all information is available.
If you run it from a non-privileged account, some information (e.g. host physical address ranges)
may be omitted. For this reason we recommend running it as root or under sudo.
Linux supports CXL memory starting with the 5.12 kernel, though it continues to receive CXL-related updates with each release.
The kernels in the recommended distros are built with the required options to support CXL memory.
In particular, as of 5.12, the CXL_CONFIG_BUS and CXL_CONFIG_MEM options are required.
cxlstat checks your kernel config file (/boot/config-`uname -r`) to verify that your kernel
supports CXL memory.
Linux supports CXL mailbox commands defined in the CXL specification. However, unsupported commands may be required to use vendor specific features or perform maintenance operations such as firmware upgrade on the memory device.
A kernel built with the CXL_MEM_RAW_COMMANDS option allows allows vendor-specific CXL mailbox commands
(i.e., "raw commands") to be sent to the device. This is disabled by default, but the option exists
so that you can enable it in a custom kernel build.
cxlstat checks your kernel config file (/boot/config-`uname -r`) to report whether the
CXL_MEM_RAW_COMMANDS option is enabled or not.
The CXL consortium has counseled system BIOS and firmware developers to configure CXL memory as the
EFI_CONVENTIONAL_MEMORY type with the EFI_MEMORY_SP attribute.
Linux configures CXL memory during boot as follows.
- If your BIOS has applied the
EFI_MEMORY_SPattribute:- Your CXL memory will appear as a DAX device (e.g.
/dev/dax0.0). - Your CXL memory can be used only by applications that call
mmapto memory-map the device. See Using CXL Memory as a DAX Device.
- Your CXL memory will appear as a DAX device (e.g.
- If your BIOS did not apply the
EFI_MEMORY_SPattribute:- Your CXL memory will appear in one or more new NUMA nodes which have no local CPU cores.
- You can use the
numactltool to set the memory placement policy for any application that needs to use the CXL memory. See Using CXL Memory as System RAM.
It may be possible to convert the mode that the CXL memory is in after the system has already booted. See Converting Between Device DAX and System RAM Modes.
On Intel systems that support the EFI_MEMORY_SP attribute, it is always enabled. No action is needed.
On an AMD system, you may need to enable it in the BIOS configuration:
- Advanced → AMD CBS → CXL Common Options → CXL SPM → set to "Enabled"
The ndctl project originally contained only tools for
administering NVDIMMs and other non-volatile memory. These tools are now being generalized to also
support CXL memory devices. One of these tools is daxctl, which now serves as the primary tool
for configuring CXL memory.
cxlstat checks whether the daxctl package is installed, and that the installed version is new
enough to support CXL memory.
See the NDCTL User Guide
for more information on installing daxctl.
Note that on the recommended distros, installing from packages
is sufficient. Older distros may require building and installing ndctl or its sub-packages (daxctl, the cxl CLI) from source to get a new enough version.
When CXL memory is in device DAX mode, it is only usable by applications that can map memory from the DAX device.
Applications that use mmap on CXL memory must do the following:
- Open the DAX device (e.g.
/dev/dax0.0) - Specify a length and offset with a 2 MiB alignment
The following is an example of a minimal C program that writes to CXL memory via mmap.
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
int main()
{
/* DAX mapping requires a 2MiB alignment */
size_t page_size = 2 * 1024 * 1024;
int fd = open("/dev/dax0.0", O_RDWR);
if (fd == -1) {
perror("open() failed");
return 1;
}
void *dax_addr = mmap(NULL, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (dax_addr == MAP_FAILED) {
perror("mmap() failed");
close(fd);
return 1;
}
/* Write something to the memory */
strcpy(dax_addr, "hello world");
munmap(dax_addr, page_size);
close(fd);
return 0;
}More examples can be found in the patches applied in the cxl branch of these benchmark repositories:
When CXL memory is in system RAM mode, Linux maps it to one or more NUMA nodes that can be used by any application.
You can run numactl -H to view the NUMA nodes on your system. In this example, NUMA node 1 (which has no local CPUs) corresponds to
the CXL memory device.
$ numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 63696 MB
node 0 free: 60568 MB
node 1 cpus:
node 1 size: 128505 MB
node 1 free: 127588 MB
node distances:
node 0 1
0: 10 50
1: 255 10
You can use numactl with the --membind option to run your app while constraining its memory allocations to a CXL NUMA node.
In this example, the CXL memory is on NUMA node 1.
numactl --membind 1 <my app command line>Depending on the BIOS configuration, Linux will bring up CXL memory devices in either device DAX mode or system RAM mode. See About BIOS Support for Specific Purpose Memory.
Use daxctl list to list the current state of CXL memory devices on the system.
If the output of daxctl list is empty, it may be necessary to reboot the system and
configure the BIOS to enable specific purpose memory.
To convert from device DAX mode to system RAM mode:
sudo daxctl reconfigure-device --mode=system-ram dax0.0 --forcedaxctl also provides a command to convert from system RAM mode to Device DAX mode. However, this
is not guaranteed to succeed, depending on the state of the system.
sudo daxctl reconfigure-device --mode=devdax dax0.0 --forceIf daxctl reconfigure-device --mode=devdax does not succeed, a reboot may be needed to get the
CXL memory device back into device DAX mode.
More extensive documentation for daxctl is available in the
NDCTL User Guide.
After you run the bootstrap script, the benchmarks subdirectory will contain several benchmark tools for testing CXL memory:
The tools that are open source (multichase, STREAM, stressapptest) have been modified to be able to test CXL memory in device DAX as well as system RAM modes.
Usage documentation for all the benchmark tools is in benchmarks/README.md.
mxcli, included as a seperate repository , is a handy application to debug and operate CXL device/s.
'mxcli support sending cxl commands over in-band ( Mailbox ) as well sideband ( MCTP/SMBUS ). 'mxcli is useful for firmaware upgrade as well deferent debuging features.
mxcli has an interactive mode with indexed menus, auto-discovery of CXL devices and
auto-completion of commands/fields, making it a self-documenting and intuitive tool.
- More details : https://github.com/cxl-micron-reskit/mxcli#