API endpoint /images/create seems to ignore the tag parameter?

[fgimian]

Issue Description

Hey there. This issue originated when trying to use Podman as a backend for GitLab Runner. The issue presented itself when GitLab Runner was attempting to pull images for performing builds as can be seen by some of the log entries:

podman[400972]: Trying to pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66...
podman[400972]: @ - - [17/May/2023:10:50:23 +1000] "POST /v1.40/images/create?fromImage=registry.gitlab.com%2Fgitlab-org%2Fgitlab-runner%2Fgitlab-runner-helper&tag=x86_64-dcfb4b66 HTTP/1.1" 200 284 "" "Go-http-client/1.1"
podman[400972]: time="2023-05-17T10:50:23+10:00" level=info msg="Request Failed(Not Found): failed to find image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66: No such image"
podman[400972]: @ - - [17/May/2023:10:50:23 +1000] "GET /v1.40/images/registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66/json HTTP/1.1" 404 441 "" "Go-http-client/1.1"
gitlab-runner[400957]: WARNING: Failed to pull image with policy "if-not-present": Error response from daemon: failed to find image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66: No such image (manager.go:241:0s)  job=639118 project=39435 runner=r3KQ8vBa
gitlab-runner[400957]: WARNING: Preparation failed: failed to pull image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66" with specified policies [if-not-present]: Error response from daemon: failed to find image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-dcfb4b66: No such image (manager.go:241:0s)  job=639118 project=39435 runner=r3KQ8vBa

After investigating this further, I can actually reproduce a difference in behaviour between Docker and the Podman Docker API which I believe is resulting in the issue. When hitting the /images/create API endpoint with a repo and tag parameter, Docker creates an image with the chosen name and tag while Podman seems to only use the name and set the tag to latest.

More information is shown below with a repro. 😄

Steps to reproduce the issue

Steps to reproduce the issue

1. Ensure you have a system with both Podman and Docker available
2. Create an empty image using each and inspect the results as follows:

me@myserver:~$ # Create an empty image with Docker.
me@myserver:~$ curl -v -XPOST --unix-socket /run/docker.sock 'http:///api/images/create?fromSrc=-&repo=myimage&tag=mytag'
*   Trying /run/docker.sock:0...
* Connected to api (/run/docker.sock) port 80 (#0)
> POST /images/create?fromSrc=-&repo=myimage&tag=mytag HTTP/1.1
> Host: api
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Api-Version: 1.41
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/20.10.21 (linux)
< Date: Wed, 17 May 2023 06:51:16 GMT
< Transfer-Encoding: chunked
<
{"status":"sha256:bb944196d717548f774e7d64da0c3b1ab9b567dcef3577d5db11c712b0eb8df1"}
* Connection #0 to host api left intact
me@myserver:~$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
myimage      mytag     bb944196d717   6 seconds ago   0B
me@myserver:~$ curl -v --unix-socket /run/docker.sock 'http://api/images/myimage:mytag/json'
*   Trying /run/docker.sock:0...
* Connected to api (/run/docker.sock) port 80 (#0)
> GET /images/myimage:mytag/json HTTP/1.1
> Host: api
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Api-Version: 1.41
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/20.10.21 (linux)
< Date: Wed, 17 May 2023 06:51:26 GMT
< Content-Length: 1424
<
{"Id":"sha256:bb944196d717548f774e7d64da0c3b1ab9b567dcef3577d5db11c712b0eb8df1","RepoTags":["myimage:mytag"],"RepoDigests":[],"Parent":"","Comment":"Imported from -","Created":"2023-05-17T06:51:16.354773741Z","Container":"","ContainerConfig":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"DockerVersion":"20.10.21","Author":"","Config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"Architecture":"amd64","Os":"linux","Size":0,"VirtualSize":0,"GraphDriver":{"Data":{"MergedDir":"/var/lib/docker/overlay2/669ba8d7a9fcdf7b1b321ec814400ec4df2964cfdc11c65a0d320afb7e288114/merged","UpperDir":"/var/lib/docker/overlay2/669ba8d7a9fcdf7b1b321ec814400ec4df2964cfdc11c65a0d320afb7e288114/diff","WorkDir":"/var/lib/docker/overlay2/669ba8d7a9fcdf7b1b321ec814400ec4df2964cfdc11c65a0d320afb7e288114/work"},"Name":"overlay2"},"RootFS":{"Type":"layers","Layers":["sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"]},"Metadata":{"LastTagTime":"2023-05-17T16:51:16.357888207+10:00"}}
* Connection #0 to host api left intact
me@myserver:~$ # Create an empty image with Podman.
me@myserver:~$ curl -v -XPOST --unix-socket /run/user/1000/podman/podman.sock 'http:///api/images/create?fromSrc=-&repo=myimage&tag=mytag'
*   Trying /run/user/1000/podman/podman.sock:0...
* Connected to api (/run/user/1000/podman/podman.sock) port 80 (#0)
> POST /images/create?fromSrc=-&repo=myimage&tag=mytag HTTP/1.1
> Host: api
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Api-Version: 1.41
< Content-Type: application/json
< Libpod-Api-Version: 4.5.0
< Server: Libpod/4.5.0 (linux)
< X-Reference-Id: 0xc0000c0490
< Date: Wed, 17 May 2023 06:51:45 GMT
< Content-Length: 198
<
{"status":"sha256:b27eea98c26530caa6fbb4754d1dd08d401f5c839e0ae4068fff65be1668ddb1","progress":"","progressDetail":{},"id":"sha256:b27eea98c26530caa6fbb4754d1dd08d401f5c839e0ae4068fff65be1668ddb1"}
* Connection #0 to host api left intact
me@myserver:~$ podman images
REPOSITORY                 TAG         IMAGE ID      CREATED        SIZE
docker.io/library/myimage  latest      b27eea98c265  3 seconds ago  1.09 kB
me@myserver:~$ curl -v --unix-socket /run/user/1000/podman/podman.sock 'http://api/images/myimage:mytag/json'
*   Trying /run/user/1000/podman/podman.sock:0...
* Connected to api (/run/user/1000/podman/podman.sock) port 80 (#0)
> GET /images/myimage:mytag/json HTTP/1.1
> Host: api
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Api-Version: 1.41
< Content-Type: application/json
< Libpod-Api-Version: 4.5.0
< Server: Libpod/4.5.0 (linux)
< X-Reference-Id: 0xc0000c92a0
< Date: Wed, 17 May 2023 06:51:59 GMT
< Content-Length: 205
<
{"cause":"failed to find image myimage:mytag: docker.io/library/myimage:mytag: No such image","message":"failed to find image myimage:mytag: docker.io/library/myimage:mytag: No such image","response":404}
* Connection #0 to host api left intact

Describe the results you received

As you can see, Docker creates the image myimage:mytag while Podman creates the image docker.io/library/myimage:latest where the tag has not been respected by Podman.

Describe the results you expected

I expected Podman to create the image with the tag mytag as Docker does.

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_100:2.1.2~0_amd64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.2, commit: '
  cpuUtilization:
    idlePercent: 99.08
    systemPercent: 0.23
    userPercent: 0.69
  cpus: 4
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: myserver.mydomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-71-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 7083307008
  memTotal: 16753397760
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc_1.1.0-0ubuntu1_amd64
    path: /usr/sbin/runc
    version: |-
      runc version 1.1.0-0ubuntu1
      spec: 1.0.2-dev
      go: go1.17.3
      libseccomp: 2.5.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.0.1-2_amd64
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 314h 16m 0.00s (Approximately 13.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/me/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/me/.local/share/containers/storage
  graphRootAllocated: 103240073216
  graphRootUsed: 48463527936
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/me/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

I've replicated this problem on various versions of Podman ranging from 3.4.4 to the latest on both Ubuntu 22.04 and AlmaLinux 9.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[fgimian]

Just want to clarify that my GitLab Runner issue did turn out to be because the server running it was using Podman 3.x instead of 4.2+ as required. The tests shown above were performed on a fresh server running Podman 4.5. Thus this particular issue is not really affecting me anymore, but I'll leave it open in case the difference in API behaviour is worth looking at.

Cheers
Fotis