Docker CLI: "unable to upgrade to tcp, received 409"

[rgov]

Issue Description

I recreated my podman machine VM. Before, I could run containers with the Docker CLI. Now, I get an error from the Docker CLI.

$ podman machine init
Downloading VM image: fedora-coreos-38.20230918.2.0-qemu.aarch64.qcow2.xz: done  

$ podman machine start
Starting machine "podman-machine-default"
...
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully

$ docker run --rm -it ros:noetic bash
Unable to find image 'ros:noetic' locally
...
b1df96a9ed3c: Download complete 
unable to upgrade to tcp, received 409

podman info output

host:
  arch: arm64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 90.59
    systemPercent: 3.68
    userPercent: 5.73
  cpus: 1
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.4.15-200.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 275283968
  memTotal: 2048487424
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.aarch64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.aarch64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 5m 12.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 3781304320
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1694549246
  BuiltTime: Tue Sep 12 13:07:26 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/arm64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

$ docker  version
Client:
 Cloud integration: v1.0.33
 Version:           24.0.2
 API version:       1.41 (downgraded from 1.43)
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:16 2023
 OS/Arch:           darwin/arm64
 Context:           default

Server: linux/arm64/fedora-38
 Podman Engine:
  Version:          4.6.2
  APIVersion:       4.6.2
  Arch:             arm64
  BuildTime:        2023-09-12T13:07:26-07:00
  Experimental:     false
  GitCommit:        
  GoVersion:        go1.20.7
  KernelVersion:    6.4.15-200.fc38.aarch64
  MinAPIVersion:    4.0.0
  Os:               linux
 Conmon:
  Version:          conmon version 2.1.7, commit: 
  Package:          conmon-2.1.7-2.fc38.aarch64
 OCI Runtime (crun):
  Version:          crun version 1.9
commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
rundir: /run/user/501/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  Package:          crun-1.9-1.fc38.aarch64
 Engine:
  Version:          4.6.2
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.20.7
  Git commit:       
  Built:            Tue Sep 12 13:07:26 2023
  OS/Arch:          linux/arm64
  Experimental:     false

Additional information

podman 4.7.0 (released 3 hours ago) is not yet available via Homebrew. Perhaps the issue is related to running too-new a VM?

Potentially a duplicate of #16266.

[rgov]

podman run works OK. The same issue happens with podman 4.7.0 (although the server version is still listed as 4.6.2).

[rgov]

I interposed the Docker API traffic with

$ socat -v TCP-LISTEN:2375,reuseaddr,fork UNIX-CONNECT:/var/run/docker.sock
$ DOCKER_HOST=tcp://localhost:2375 docker run --rm -it ros:noetic bash

First there is a POST to /v1.41/containers/create to create the container. This succeeds.

Second there is POST to attach to the container:

POST /v1.41/containers/8515b138e41adf1baea3e2ead7a9871cea3b5c688d642eec061e1fc2e187a08c/attach?stderr=1&stdin=1&stdout=1&stream=1 HTTP/1.1\r
Host: localhost:2375\r
User-Agent: Docker-Client/24.0.2 (darwin)\r
Content-Length: 0\r
Connection: Upgrade\r
Content-Type: text/plain\r
Upgrade: tcp\r
\r

The server rejects it:

HTTP/1.1 409 Conflict\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 4.6.2\r
Server: Libpod/4.6.2 (linux)\r
X-Reference-Id: 0x4000696720\r
Date: Wed, 27 Sep 2023 22:55:38 GMT\r
Content-Length: 242\r
\r
{"cause":"OCI permission denied","message":"preparing container 8515b138e41adf1baea3e2ead7a9871cea3b5c688d642eec061e1fc2e187a08c for attach: crun: write to `/proc/self/oom_score_adj`: Permission denied: OCI permission denied","response":409}

[rgov]

I turned on rootful for the VM and restarted it, and that seemed to allow me to run containers again.

[rgov]

It also works to go back to an earlier version of the VM:

$ podman machine init --image-path stable
Downloading VM image: fedora-coreos-38.20230902.3.0-qemu.aarch64.qcow2.xz: done  

So I believe this is a regression between 38.20230902.3.0 and 38.20230918.2.0.

[Luap99]

Duplicate of #19930