compose v2 ignores runtime field

[egyptianbman]

I'm trying to get a container that uses the nvidia runtime to work but I can't seem to be able to do so, using compose v2. I can run the following without issue:

➜ docker run --rm --runtime nvidia nvidia/cuda:11.0-base nvidia-smi    
Fri Oct  1 07:24:26 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      Off  | 00000000:00:04.0 Off |                    0 |
| N/A   32C    P0    50W / 350W |      0MiB / 40536MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Running the following test compose:

version: '3.9'
                                                                                                       
services:                                                                                              
  test:
    image: nvidia/cuda:11.0-base
    command: nvidia-smi
    runtime: nvidia  

Doesn't yield the expected output. I changed the command to bash and ran the container and noticed that upon running a docker inspect, the output isn't what I expected:

➜ docker inspect test-test-1 | jq '.[0].HostConfig.Runtime'
"runc"

As expected, the runtime is configured in /etc/docker/daemon.json:

➜ cat /etc/docker/daemon.json 
{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

[ghost]

I got same issue while trying gVisor(runsc).

I create minimum regenerating manifest:

version: '2.4'

services:
  docker:
    image: docker:dind
    privileged: true
    entrypoint: []
    command:
      - sh
      - -c
      - |
        set -euvx

        ## install compose v2.0.1
        wget -qO ./docker-compose-linux-x86_64 'https://github.com/docker/compose/releases/download/v2.0.1/docker-compose-linux-x86_64'
        wget -qO ./docker-compose-linux-x86_64.sha256 'https://github.com/docker/compose/releases/download/v2.0.1/docker-compose-linux-x86_64.sha256'
        sha256sum -c ./docker-compose-linux-x86_64.sha256
        mkdir -p /root/.docker/cli-plugins/
        mv ./docker-compose-linux-x86_64 /root/.docker/cli-plugins/docker-compose
        chmod +x /root/.docker/cli-plugins/docker-compose

        ## install compose v1.28.0-r1
        apk add -qU --no-cache docker-compose=1.28.0-r1

        ## configure dummy runtime
        mkdir -p /etc/docker/
        cat << __EOT__ > /etc/docker/daemon.json
        {
          "runtimes": {
            "anotherrunc": {
              "path": "/usr/local/bin/runc"
            }
          }
        }
        __EOT__

        ## start dockerd
        nohup dockerd-entrypoint.sh > /dockerd.log 2>&1 &

        ## wait dockerd start normaly
        until docker ps > /dev/null 2>&1; do echo 'wait until dockerd start...'; sleep 2s; done

        ## create docker-compose.yaml
        cat << '__EOT__' > docker-compose.yaml
        version: '2.4'
        services:
          nginx:
            image: nginx:latest
            runtime: anotherrunc
            container_name: nginx
        __EOT__

        ## up with compose v2
        docker compose version
        docker compose up -d --quiet-pull
        sleep 2s
        docker inspect nginx | grep Runtime

        sleep 2s
        docker compose down

        ## up with compose v1
        docker-compose version
        docker-compose up -d --quiet-pull
        sleep 2s
        docker inspect nginx | grep Runtime

what manifest do:

• run docker:dind container as docker service
• in the container, install compose V1 and V2 command
• configure anotherrunc runtime
  • it's just dummy runtime. call runc actually.
• create docker-compose.yaml manifest with runtime field
• run the manifest with compose V2/V1
  • inspect the container that compose V2 starts

Following result are shown:

$ docker-compose up
Creating network "runtime-issue_default" with the default driver
Creating runtime-issue_docker_1 ... done
Attaching to runtime-issue_docker_1
docker_1  | + wget -qO ./docker-compose-linux-x86_64 https://github.com/docker/compose/releases/download/v2.0.1/docker-compose-linux-x86_64
docker_1  | + wget -qO ./docker-compose-linux-x86_64.sha256 https://github.com/docker/compose/releases/download/v2.0.1/docker-compose-linux-x86_64.sha256
docker_1  | + sha256sum -c ./docker-compose-linux-x86_64.sha256
docker_1  | docker-compose-linux-x86_64: OK
docker_1  | + mkdir -p /root/.docker/cli-plugins/
docker_1  | + mv ./docker-compose-linux-x86_64 /root/.docker/cli-plugins/docker-compose
docker_1  | + chmod +x /root/.docker/cli-plugins/docker-compose
docker_1  | + apk add -qU --no-cache 'docker-compose=1.28.0-r1'
docker_1  | + mkdir -p /etc/docker/
docker_1  | + cat
docker_1  | + + docker ps
docker_1  | nohup dockerd-entrypoint.sh
docker_1  | + echo 'wait until dockerd start...'
docker_1  | + sleep 2s
docker_1  | wait until dockerd start...
docker_1  | + docker ps
docker_1  | + cat
docker_1  | + docker compose version
docker_1  | Docker Compose version v2.0.1
docker_1  | + docker compose up -d --quiet-pull
docker_1  | nginx Pulling
docker_1  | nginx Pulled
docker_1  | Network _default  Creating
docker_1  | Network _default  Created
docker_1  | Container nginx  Creating
docker_1  | Container nginx  Created
docker_1  | Container nginx  Starting
docker_1  | Container nginx  Started
docker_1  | + sleep 2s
docker_1  | + docker inspect nginx
docker_1  | + grep Runtime
docker_1  |             "Runtime": "runc",
docker_1  |             "CpuRealtimeRuntime": 0,
docker_1  | + sleep 2s
docker_1  | + docker compose down
docker_1  | Container nginx  Stopping
docker_1  | Container nginx  Stopping
docker_1  | Container nginx  Stopped
docker_1  | Container nginx  Removing
docker_1  | Container nginx  Removed
docker_1  | Network _default  Removing
docker_1  | Network _default  Removed
docker_1  | + docker-compose version
docker_1  | docker-compose version 1.28.0, build unknown
docker_1  | docker-py version: 5.0.0
docker_1  | CPython version: 3.9.5
docker_1  | OpenSSL version: OpenSSL 1.1.1l  24 Aug 2021
docker_1  | + docker-compose up -d --quiet-pull
docker_1  | Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
docker_1  | Creating network "default_default" with the default driver
docker_1  | Creating nginx ...
docker_1  | Creating nginx ... done
docker_1  | + sleep 2s
docker_1  | + docker inspect nginx
docker_1  | + grep Runtime
docker_1  |             "Runtime": "anotherrunc",
docker_1  |             "CpuRealtimeRuntime": 0,
runtime-issue_docker_1 exited with code 0

above result looks compose V2 ignores runtime field and run container by runc.

My host docker environment:

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc., v2.0.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 132
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 12.17GiB
 Name: DESKTOP-OE11EOF
 ID: 3IX3:ACCW:U7UI:C2IE:32V2:TMO3:V5EW:IC7P:R3PY:HVWM:EXYB:32HH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

$ docker-compose version
docker-compose version 1.25.0, build unknown
docker-py version: 4.1.0
CPython version: 3.8.10
OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

@egyptianbman
Hi. Thank you for your issue report.
I think that the title "compose v2 runtime issue" is too abstractive to get maintainer's attention.
Could you consider to change the title?
For example, "compose v2 ignores runtime field", or something.

[egyptianbman]

Updated title as requested.

Looking at the documentation again, I noticed that runtime is marked as legacy. I tested the provided example utilizing capabilities:

services:
  test:
    image: nvidia/cuda:10.2-base
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

With that, I am now finding that the nvidia-smi command exists, but yields an error:

❯ docker-compose up test   
[+] Running 1/0
 ⠿ Container test-test-1  Created                                                                                                                                                                         0.0s
Attaching to test-test-1
test-test-1  | Failed to initialize NVML: Unknown Error
test-test-1 exited with code 255

With runtime, the output is:

❯ docker-compose up test
[+] Running 1/0
 ⠿ Container test-test-1  Recreated                                                                                                                                                                       0.1s
Attaching to test-test-1
Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "nvidia-smi": executable file not found in $PATH: unknown

In both cases, the runtime is runc.