v1.13.0

This is a promotion of the v1.13.0-rc.3 release to GA.

This release of the NVIDIA Container Toolkit adds the following features:

• Improved support for the Container Device Interface (CDI) specifications for GPU devices when using the NVIDIA Container Toolkit in the context of the GPU Operator.
• Added the generation CDI specifications on WSL2-based systems using the nvidia-ctk cdi generate command. This is now the recommended mechanism for using GPUs on WSL2 and podman is the recommended container engine.

NOTE: This release is a unified release of the NVIDIA Container Toolkit that consists of the following packages:

• libnvidia-container 1.13.0
• nvidia-container-toolkit 1.13.0
• nvidia-container-runtime 3.13.0
• nvidia-docker2 2.13.0

The packages for this release are published to the libnvidia-container package repositories.

Full Changelog: v1.12.0...v1.13.0

v1.13.0-rc.3

• Only initialize NVML for modes that require it when runing nvidia-ctk cdi generate.
• Prefer /run over /var/run when locating nvidia-persistenced and nvidia-fabricmanager sockets.
• Fix the generation of CDI specifications for management containers when the driver libraries are not in the LDCache.
• Add transformers to deduplicate and simplify CDI specifications.
• Generate a simplified CDI specification by default. This means that entities in the common edits in a spec are not included in device definitions.
• Also return an error from the nvcdi.New constructor instead of panicing.
• Detect XOrg libraries for injection and CDI spec generation.
• Add nvidia-ctk system create-device-nodes command to create control devices.
• Add nvidia-ctk cdi transform command to apply transforms to CDI specifications.
• Add --vendor and --class options to nvidia-ctk cdi generate

Changes from libnvidia-container v1.13.0-rc.3

• Fix segmentation fault when RPC initialization fails.
• Build centos variants of the NVIDIA Container Library with static libtirpc v1.3.2.
• Remove make targets for fedora35 as the centos8 packages are compatible.

Changes in the toolkit-container

• Add nvidia-container-runtime.modes.cdi.annotation-prefixes config option that allows the CDI annotation prefixes that are read to be overridden.
• Create device nodes when generating CDI specification for management containers.
• Add nvidia-container-runtime.runtimes config option to set the low-level runtime for the NVIDIA Container Runtime

v1.13.0-rc.2

• Don't fail chmod hook if paths are not injected
• Only create by-path symlinks if CDI devices are actually requested.
• Fix possible blank nvidia-ctk path in generated CDI specifications
• Fix error in postun scriplet on RPM-based systems
• Only check NVIDIA_VISIBLE_DEVICES for environment variables if no annotations are specified.
• Add cdi.default-kind config option for constructing fully-qualified CDI device names in CDI mode
• Add support for accept-nvidia-visible-devices-envvar-unprivileged config setting in CDI mode
• Add nvidia-container-runtime-hook.skip-mode-detection config option to bypass mode detection. This allows legacy and cdi mode, for example, to be used at the same time.
• Add support for generating CDI specifications for GDS and MOFED devices
• Ensure CDI specification is validated on save when generating a spec
• Rename --discovery-mode argument to --mode for nvidia-ctk cdi generate

Changes from libnvidia-container v1.13.0-rc.2

• Fix segfault on WSL2 systems. This was triggered in the v1.12.1 and v1.13.0-rc.1 releases.

Changes in the toolkit-container

• Add --cdi-enabled flag to toolkit config
• Install nvidia-ctk from toolkit container
• Use installed nvidia-ctk path in NVIDIA Container Toolkit config
• Bump CUDA base images to 12.1.0
• Set nvidia-ctk path in the
• Add cdi.k8s.io/* to set of allowed annotations in containerd config
• Generate CDI specification for use in management containers
• Install experimental runtime as nvidia-container-runtime.experimental instead of nvidia-container-runtime-experimental
• Install and configure mode-specific runtimes for cdi and legacy modes

v1.13.0-rc.1

• Include MIG-enabled devices as GPUs when generating CDI specification
• Fix missing NVML symbols when running nvidia-ctk on some platforms [#49]
• Add CDI spec generation for WSL2-based systems to nvidia-ctk cdi generate command
• Add auto mode to nvidia-ctk cdi generate command to automatically detect a WSL2-based system over a standard NVML-based system.
• Add mode-specific (.cdi and .legacy) NVIDIA Container Runtime binaries for use in the GPU Operator
• Discover all gsb*.bin GSP firmware files when generating CDI specification.
• Align .deb and .rpm release candidate package versions
• Remove fedora35 packaging targets

Changes in toolkit-container

• Install nvidia-container-toolkit-operator-extensions package for mode-specific executables.
• Allow nvidia-container-runtime.mode to be set when configuring the NVIDIA Container Toolkit

Changes from libnvidia-container v1.13.0-rc.1

• Include all gsp*.bin firmware files if present
• Align .deb and .rpm release candidate package versions
• Remove fedora35 packaging targets

v1.13.0-rc.3

• Only initialize NVML for modes that require it when runing nvidia-ctk cdi generate.
• Prefer /run over /var/run when locating nvidia-persistenced and nvidia-fabricmanager sockets.
• Fix the generation of CDI specifications for management containers when the driver libraries are not in the LDCache.
• Add transformers to deduplicate and simplify CDI specifications.
• Generate a simplified CDI specification by default. This means that entities in the common edits in a spec are not included in device definitions.
• Also return an error from the nvcdi.New constructor instead of panicing.
• Detect XOrg libraries for injection and CDI spec generation.
• Add nvidia-ctk system create-device-nodes command to create control devices.
• Add nvidia-ctk cdi transform command to apply transforms to CDI specifications.
• Add --vendor and --class options to nvidia-ctk cdi generate

Changes from libnvidia-container v1.13.0-rc.3

• Fix segmentation fault when RPC initialization fails.
• Build centos variants of the NVIDIA Container Library with static libtirpc v1.3.2.
• Remove make targets for fedora35 as the centos8 packages are compatible.

Changes in the toolkit-container

• Add nvidia-container-runtime.modes.cdi.annotation-prefixes config option that allows the CDI annotation prefixes that are read to be overridden.
• Create device nodes when generating CDI specification for management containers.
• Add nvidia-container-runtime.runtimes config option to set the low-level runtime for the NVIDIA Container Runtime

Full Changelog: v1.13.0-rc.2...v1.13.0-rc.3

v1.13.0-rc.2

• Don't fail chmod hook if paths are not injected
• Only create by-path symlinks if CDI devices are actually requested.
• Fix possible blank nvidia-ctk path in generated CDI specifications
• Fix error in postun scriplet on RPM-based systems
• Only check NVIDIA_VISIBLE_DEVICES for environment variables if no annotations are specified.
• Add cdi.default-kind config option for constructing fully-qualified CDI device names in CDI mode
• Add support for accept-nvidia-visible-devices-envvar-unprivileged config setting in CDI mode
• Add nvidia-container-runtime-hook.skip-mode-detection config option to bypass mode detection. This allows legacy and cdi mode, for example, to be used at the same time.
• Add support for generating CDI specifications for GDS and MOFED devices
• Ensure CDI specification is validated on save when generating a spec
• Rename --discovery-mode argument to --mode for nvidia-ctk cdi generate

Changes in the toolkit-container

• Add --cdi-enabled flag to toolkit config
• Install nvidia-ctk from toolkit container
• Use installed nvidia-ctk path in NVIDIA Container Toolkit config
• Bump CUDA base images to 12.1.0
• Set nvidia-ctk path in the
• Add cdi.k8s.io/* to set of allowed annotations in containerd config
• Generate CDI specification for use in management containers
• Install experimental runtime as nvidia-container-runtime.experimental instead of nvidia-container-runtime-experimental
• Install and configure mode-specific runtimes for cdi and legacy modes

Changes from libnvidia-container v1.13.0-rc.2

• Fix segfault on WSL2 systems. This was triggered in the v1.12.1 and v1.13.0-rc.1 releases.

Full Changelog: v1.13.0-rc.1...v1.13.0-rc.2

v1.12.1

• Don't fail chmod hook if paths are not injected. Fixes known issue in v1.12.0 release
• Fix possible blank nvidia-ctk path in generated CDI specifications
• Fix error in postun scriplet on RPM-based systems
• Fix missing NVML symbols when running nvidia-ctk on some platforms [#49]
• Discover all gsb*.bin GSP firmware files when generating CDI specification.
• Remove fedora35 packaging targets

Changes in toolkit-container

• Install nvidia-ctk from toolkit container
• Use installed nvidia-ctk path in NVIDIA Container Toolkit config
• Bump CUDA base images to 12.1.0

Changes from libnvidia-container v1.12.1

• Include all gsp*.bin firmware files if present

Full Changelog: v1.12.0...v1.12.1

v1.13.0-rc.1

• Include MIG-enabled devices as GPUs when generating CDI specification
• Fix missing NVML symbols when running nvidia-ctk on some platforms [#49]
• Add CDI spec generation for WSL2-based systems to nvidia-ctk cdi generate command
• Add auto mode to nvidia-ctk cdi generate command to automatically detect a WSL2-based system over a standard NVML-based system.
• Add mode-specific (.cdi and .legacy) NVIDIA Container Runtime binaries for use in the GPU Operator
• Discover all gsb*.bin GSP firmware files when generating CDI specification.
• Align .deb and .rpm release candidate package versions
• Remove fedora35 packaging targets

Changes in toolkit-container

• Install nvidia-container-toolkit-operator-extensions package for mode-specific executables.
• Allow nvidia-container-runtime.mode to be set when configuring the NVIDIA Container Toolkit

Changes from libnvidia-container v1.13.0-rc.1

• Include all gsp*.bin firmware files if present
• Align .deb and .rpm release candidate package versions
• Remove fedora35 packaging targets

Full Changelog: v1.12.0...v1.13.0-rc.1

Known Issues

Failure to run container due to missing /dev/dri and / or /dev/nvidia-caps paths in container

As of v1.12.0 using a CDI Specification generated with the nvidia-ctk cdi generate command may result in a failure to run a container if a device is selected which has no DRM device nodes (in /dev/dri) or NVIDIA Cap devices (in /dev/nvidia-caps) associated with it. The workaround is to remove the following createContainer hook:

  - args:
    - nvidia-ctk
    - hook
    - chmod
    - --mode
    - "755"
    - --path
    - /dev/dri
    hookName: createContainer
    path: /usr/bin/nvidia-ctk

from the generated CDI specification or select a device that includes associated DRM nodes at /dev/dri and / or NVIDIA Caps devices at /dev/nvidia-caps.

v1.12.0

This is a promotion of the v1.12.0-rc.5 release to GA.

This release of the NVIDIA Container Toolkit v1.12.0 adds the following features:

• Improved support for headless Vulkan applications in containerized environments.
• Tooling to generate Container Device Interface (CDI) specifications for GPU devices. The use of CDI is now the recommended mechanism for using GPUs in podman.

NOTE: This release is a unified release of the NVIDIA Container Toolkit that consists of the following packages:

• libnvidia-container 1.12.0
• nvidia-container-toolkit 1.12.0
• nvidia-container-runtime 3.12.0
• nvidia-docker2 2.12.0

The packages for this release are published to the libnvidia-container package repositories.

Full Changelog: v1.11.0...v1.12.0

v1.12.0

Changes for the container-toolkit container

• Update CUDA base images to 12.0.1

Changes from libnvidia-container v1.12.0

• Add nvcubins.bin to DriverStore components under WSL2

v1.12.0-rc.5

• Fix bug here the nvidia-ctk path was not properly resolved. This causes failures to run containers when the runtime is configured in csv mode or if the NVIDIA_DRIVER_CAPABILITIES includes graphics or display (e.g. all).

v1.12.0-rc.4

• Generate a minimum CDI spec version for improved compatibility.
• Add --device-name-strategy [index | uuid | type-index] options to the nvidia-ctk cdi generate command that can be used to control how device names are constructed.
• Set default for CDI device name generation to index to generate device names such as nvidia.com/gpu=0 or nvidia.com/gpu=1:0 by default. NOTE: This is a breaking change and will cause a v0.5.0 CDI specification to be generated. To keep the previous generate a v0.4.0 CDI specification with nvidia.com/gpu=gpu0 or nvidia.com/gpu=mig1:0 device names use the type-index option.
• Ensure that nivdia-container-toolkit package can be upgraded to from versions older than v1.11.0 on RPM-based systems.

v1.12.0-rc.3

• Don't fail if by-path symlinks for DRM devices do not exist
• Replace the --json flag with a --format [json|yaml] flag for the nvidia-ctk cdi generate command
• Ensure that the CDI output folder is created if required
• When generating a CDI specification use a blank host path for devices to ensure compatibility with the v0.4.0 CDI specification
• Add injection of Wayland JSON files
• Add GSP firmware paths to generated CDI specification
• Add --root flag to nvidia-ctk cdi generate command to allow for a non-standard driver root to be specified

v1.12.0-rc.2

• Update golang version to 1.18
• Inject Direct Rendering Manager (DRM) devices into a container using the NVIDIA Container Runtime
• Improve logging of errors from the NVIDIA Container Runtime
• Improve CDI specification generation to support rootless podman
• Use nvidia-ctk cdi generate to generate CDI specifications instead of nvidia-ctk info generate-cdi

Changes from libnvidia-container v1.12.0-rc.2

• Skip creation of existing files when mounting them from the host

v1.12.0-rc.1

• Improve injection of Vulkan configurations and libraries
• Add nvidia-ctk info generate-cdi command to generated CDI specification for available devices

Changes for the container-toolkit container

• Update CUDA base images to 11.8.0

Changes from libnvidia-container v1.12.0-rc.1

• Add NVVM Compiler Library (libnvidia-nvvm.so) to list of compute libraries

v1.12.0-rc.5

• Fix bug here the nvidia-ctk path was not properly resolved. This causes failures to run containers when the runtime is configured in csv mode or if the NVIDIA_DRIVER_CAPABILITIES includes graphics or display (e.g. all).

v1.12.0-rc.4

• Generate a minimum CDI spec version for improved compatibility.
• Add --device-name-strategy [index | uuid | type-index] options to the nvidia-ctk cdi generate command that can be used to control how device names are constructed.
• Set default for CDI device name generation to index to generate device names such as nvidia.com/gpu=0 or nvidia.com/gpu=1:0 by default. NOTE: This is a breaking change and will cause a v0.5.0 CDI specification to be generated. To keep the previous generate a v0.4.0 CDI specification with nvidia.com/gpu=gpu0 or nvidia.com/gpu=mig1:0 device names use the type-index option.
• Ensure that nivdia-container-toolkit package can be upgraded to from versions older than v1.11.0 on RPM-based systems.

v1.12.0-rc.3

• Don't fail if by-path symlinks for DRM devices do not exist
• Replace the --json flag with a --format [json|yaml] flag for the nvidia-ctk cdi generate command
• Ensure that the CDI output folder is created if required
• When generating a CDI specification use a blank host path for devices to ensure compatibility with the v0.4.0 CDI specification
• Add injection of Wayland JSON files
• Add GSP firmware paths to generated CDI specification
• Add --root flag to nvidia-ctk cdi generate command to allow for a non-standard driver root to be specified

v1.12.0-rc.2

• Update golang version to 1.18
• Inject Direct Rendering Manager (DRM) devices into a container using the NVIDIA Container Runtime
• Improve logging of errors from the NVIDIA Container Runtime
• Improve CDI specification generation to support rootless podman
• Use nvidia-ctk cdi generate to generate CDI specifications instead of nvidia-ctk info generate-cdi

Changes from libnvidia-container v1.12.0-rc.2

• Skip creation of existing files when mounting them from the host