NAME¶

daxctl-reconfigure-device - Reconfigure a dax device into a different mode

SYNOPSIS¶

daxctl reconfigure-device <dax0.0> [<dax1.0>...<daxY.Z>] [<options>]

DESCRIPTION¶

Reconfigure the operational mode of a dax device. This can be used to convert a regular devdax mode device to the system-ram mode which arranges for the dax range to be hot-plugged into the system as regular memory.

# daxctl reconfigure-device --mode=system-ram --region=0 all
libdaxctl: daxctl_dev_disable: dax3.0: error: device model is dax-class
dax3.0: disable failed: Operation not supported
error reconfiguring devices: Operation not supported
reconfigured 0 devices

daxctl-reconfigure-device nominally expects that it will online new memory blocks as movable, so that kernel data doesn’t make it into this memory. However, there are other potential agents that may be configured to automatically online new hot-plugged memory as it appears. Most notably, these are the /sys/devices/system/memory/auto_online_blocks configuration, or system udev rules. If such an agent races to online memory sections, daxctl checks if the blocks were onlined as movable memory. If this was not the case, and the memory blocks are found to be in a different zone, then a warning is displayed. If it is desired that a different agent control the onlining of memory blocks, and the associated memory zone, then it is recommended to use the --no-online option described below. This will abridge the device reconfiguration operation to just hotplugging the memory, and refrain from then onlining it.

In case daxctl detects that there is a kernel policy to auto-online blocks (via /sys/devices/system/memory/auto_online_blocks), then reconfiguring to system-ram will result in a failure. This can be overridden with --force.

THEORY OF OPERATION¶

The kernel device-dax subsystem surfaces character devices that provide DAX-access (direct mappings sans page-cache buffering) to a given memory region. The devices are named /dev/daxX.Y where X is a region-id and Y is an instance-id within that region. There are 2 mechanisms that trigger device-dax instances to appear:

In the Soft Reservation case the expectation for EFI + ACPI based platforms is that in addition to the EFI_MEMORY_SP attribute the firmware also creates distinct ACPI proximity domains for any address range that has different performance characteristics than default "System RAM". So, the SRAT will define the proximity domain, the SLIT communicates relative distance to other proximity domains, and the HMAT is populated with nominal read/write latency and read/write bandwidth data. That HMAT data is emitted to the kernel log on bootup, and also exported to sysfs. See NUMAPERF <https://www.kernel.org/doc/html/latest/admin-guide/mm/numaperf.html>, for the runtime representation of CPU to Memory node performance details.

Outside of the NUMA performance details linked above the other method to detect the presence of "Soft Reserved" memory is to dump /proc/iomem and look for "Soft Reserved" ranges. If the kernel was not built with CONFIG_EFI_SOFT_RESERVE, predates the introduction of CONFIG_EFI_SOFT_RESERVE (v5.5), or was booted with the efi=nosoftreserve command line then device-dax will not attach and the expectation is that the memory shows up as a memory-only NUMA node. Otherwise the memory shows up as a device-dax instance and DAXCTL(1) can be used to optionally partition it and assign the memory back to the kernel as "System RAM", or the device can be mapped directly as the back end of a userspace memory allocator like LIBVMEM <https://pmem.io/vmem/libvmem/>.

EXAMPLES¶

# daxctl reconfigure-device --mode=system-ram --no-online dax0.0
[


  {


    "chardev":"dax0.0",


    "size":16777216000,


    "target_node":2,


    "mode":"system-ram"


  }
]

# daxctl reconfigure-device --human --mode=devdax --force dax0.0
{


  "chardev":"dax0.0",


  "size":"15.63 GiB (16.78 GB)",


  "target_node":2,


  "mode":"devdax"
}

# daxctl reconfigure-device --mode=system-ram --region=0 all
[


  {


    "chardev":"dax0.0",


    "size":16777216000,


    "target_node":2,


    "mode":"system-ram"


  },


  {


    "chardev":"dax0.1",


    "size":16777216000,


    "target_node":3,


    "mode":"system-ram"


  }
]

# daxctl reconfigure-device --mode=system-ram dax0.0
[


  {


    "chardev":"dax0.0",


    "size":16777216000,


    "target_node":2,


    "mode":"system-ram"


  }
]
# numactl --cpunodebind=0-1 --membind=2 -- some-service --opt1 --opt2

# daxctl reconfigure-device dax0.1 -s 16G
reconfigured 1 device
# daxctl reconfigure-device dax0.1 -s 0
reconfigured 1 device

OPTIONS¶

-r, --region=

-s, --size=

The size must be a multiple of the region alignment.

This option is mutually exclusive with -m or --mode.

-a, --align

This option is mutually exclusive with -m or --mode.

-m, --mode=

-N, --no-online

--no-movable

-f, --force

-u, --human

-v, --verbose

COPYRIGHT¶

Copyright © 2016 - 2022, Intel Corporation. License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

SEE ALSO¶

daxctl-list(1),daxctl-migrate-device-model[1]