Docker driver devices host_path accepts directory but fails to provide it to Docker container

[jinnatar]

Nomad version

Nomad v1.5.2
BuildDate 2023-03-21T22:54:38Z
Revision 9a2fdb5f53dce81edf2802f0b64962e07596fd03

Operating system and Environment details

%> cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
%> docker --version
Docker version 23.0.2, build 569dd73

Issue

The given config snippet is accepted but does not produce the same results as running directly with Docker, i.e. no devices are made available to the container:

devices = [                       
           {                               
             host_path = "/dev/bus/usb/002"
            }
          ]

Reproduction steps

These examples should be run on a node with usb devices connected, i.e. there's something under /dev/bus/usb

Plain Docker variant (works)

%> docker run --rm -it --device /dev/bus/usb alpine:latest ls /dev/bus/usb

Nomad variant (does not work)

job "devices-repro" {
  datacenters = ["dc1"]
  type        = "service"
 
  group "devices-repro" {
    count = 1
    task "devices-repro" {
      driver = "docker"
      config {
        image = "alpine:latest"
        entrypoint = ["ls", "/dev/bus/usb/"]
        devices = [
          {
            host_path = "/dev/bus/usb"
          }
        ]
      }
    }
  }
}    

Expected Result

Vanilla Docker run:

001  002  003  004

Actual Result

Nomad task log:

ls: /dev/bus/usb/: No such file or directory

[schmichael]

Hi @Artanicus, sorry you ran into this and thanks for reporting it! While the docker CLI defaults container_path to the host_path, Nomad does not. Explicitly setting container_path is required as of Nomad 1.5.3:

job "devices-repro" {
  type = "batch"

  group "devices-repro" {
    task "devices-repro" {
      driver = "docker"
      config {
        image      = "alpine:latest"
        entrypoint = ["ls", "/dev/bus/usb/"]
        devices = [
          {
            host_path      = "/dev/bus/usb"
            container_path = "/dev/bus/usb"
          }
        ]
      }
    }
  }
}

That being said we ought to default container_path if it's unset to match Docker's behavior (and I suspect the common use case is that the path's match).

I posted PR #16811 to fix this!