rm + build + up keeps using the previously existing image for a container (2.11.0)

[nocive]

Description

It seems that using a combination of docker compose rm <CONTAINER> + docker compose build <CONTAINER> + docker compose up <CONTAINER> does not tag a new image and therefore keeps using an old one when it exists.
Only after calling docker compose down --rmi local or docker rmi will it then cause a new image to be tagged and used.

From the output it's possible to gather that the "sending tarball" and "importing to docker" steps that should happen under normal circumstances, do not run when the problem occurs. Furthermore, passing the --build flag to docker compose up seems to circumvent the problem.

Steps to reproduce the issue:

1. Use this dockerfile:

FROM busybox

COPY ./test.txt /

CMD ["sh", "-c", "cat test.txt && tail -f /dev/null"]

2. Use this project file:

version: '3.8'
services:
  test:
    build:
      context: ./
      dockerfile: Dockerfile

3. Follow these steps

echo "NOK" > test.txt
docker compose up -d test
docker compose logs test ## shows NOK, so far so good
docker images --format '{{ .CreatedAt }}' <the image name> ## verify creation date

echo "OK" > test.txt
docker compose rm -fsv test
docker compose build test
docker compose up -d test ## if one uses --build here it works fine
docker compose logs test ## still shows NOK when it should be showing OK
docker images --format '{{ .CreatedAt }}' <the image name> ## verify it has the same creation date as the previous one

Describe the results you received:
A new image is not tagged despite the build running and the project keeps using an older image until the image is explicitly removed via docker compose down --rmi local or docker rmi.

output from the test described above

[+] Building 0.9s (8/8) FINISHED                                                                                                                                                     
 => [internal] load build definition from Dockerfile                                                                                                                            0.0s
 => => transferring dockerfile: 124B                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                               0.6s
 => [internal] load build context                                                                                                                                               0.0s
 => => transferring context: 39B                                                                                                                                                0.0s
 => [1/2] FROM docker.io/library/busybox@sha256:ad9bd57a3a57cc95515c537b89aaa69d83a6df54c4050fcf2b41ad367bec0cd5                                                                0.0s
 => => resolve docker.io/library/busybox@sha256:ad9bd57a3a57cc95515c537b89aaa69d83a6df54c4050fcf2b41ad367bec0cd5                                                                0.0s
 => CACHED [2/2] COPY ./test.txt /                                                                                                                                              0.0s
 => exporting to oci image format                                                                                                                                               0.3s
 => => exporting layers                                                                                                                                                         0.0s
 => => exporting manifest sha256:941ca5deef493a2a8e5afd32651eb7db8d65908839e5869b06e2290d79f9339c                                                                               0.0s
 => => exporting config sha256:af505d274ae946bf64669eec69d554af97c65921d35c47e5b4989d813d105c02                                                                                 0.0s
 => => sending tarball                                                                                                                                                          0.2s
 => importing to docker                                                                                                                                                         0.1s
[+] Running 2/2
 ⠿ Network compose-build-image-issue_default   Created                                                                                                                          0.1s
 ⠿ Container compose-build-image-issue-test-1  Started                                                                                                                          0.7s
compose-build-image-issue-test-1  | NOK
2022-09-18 09:20:25 +0200 CEST
[+] Running 1/1
 ⠿ Container compose-build-image-issue-test-1  Stopped                                                                                                                         10.3s
Going to remove compose-build-image-issue-test-1
[+] Running 1/0
 ⠿ Container compose-build-image-issue-test-1  Removed                                                                                                                          0.0s
[+] Building 0.8s (7/7) FINISHED                                                                                                                                                     
 => [internal] load .dockerignore                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                 0.0s
 => [internal] load build definition from Dockerfile                                                                                                                            0.0s
 => => transferring dockerfile: 124B                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                               0.5s
 => [internal] load build context                                                                                                                                               0.0s
 => => transferring context: 38B                                                                                                                                                0.0s
 => CACHED [1/2] FROM docker.io/library/busybox@sha256:ad9bd57a3a57cc95515c537b89aaa69d83a6df54c4050fcf2b41ad367bec0cd5                                                         0.0s
 => => resolve docker.io/library/busybox@sha256:ad9bd57a3a57cc95515c537b89aaa69d83a6df54c4050fcf2b41ad367bec0cd5                                                                0.0s
 => [2/2] COPY ./test.txt /                                                                                                                                                     0.1s
 => exporting to image                                                                                                                                                          0.1s
 => => exporting layers                                                                                                                                                         0.1s
 => => exporting manifest sha256:48a47ae7211259601b95bda08318c688509dfba1f60e03c1489439f75c877f74                                                                               0.0s
 => => exporting config sha256:3adee52fc6b1eb26beea505426b3702996ea2f1b7bc87d1539b28ef7abee57c0                                                                                 0.0s
[+] Running 1/1
 ⠿ Container compose-build-image-issue-test-1  Started                                                                                                                          0.5s
compose-build-image-issue-test-1  | NOK
2022-09-18 09:20:25 +0200 CEST

Describe the results you expected:
A new image to be tagged and used if the container changed.

Additional information you deem important (e.g. issue happens only occasionally):
This bug seems to have been introduced in 2.11.0, problem is not reproducible with earlier versions.

It seems that using the --build flag on the second docker compose up call (after docker compose build) does make the problem go away...

Output of docker compose version:

Docker Compose version v2.11.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc., v2.11.0)
  scan: Docker Scan (Docker Inc., v0.1.0-227-g061fe0a0c5)

Server:
 Containers: 79
  Running: 2
  Paused: 0
  Stopped: 77
 Images: 83
 Server Version: 20.10.18
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 96df0994faabc1944fc614e52b0b3c6feb609a57.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.18.1-arch1-1
 Operating System: xxx
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.49GiB
 Name: jps-x1
 ID: xxx
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: xxx
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[nocive]

Problem seems to have been introduced with 2.11.0 (perhaps #9819?).
The same test against 2.10.2 does not reproduce the issue.

[botanegg]

It seems like docker compose build is broken since v2.11.0 cause it dont save images

to reproduce -
repeat steps 1 and 2 from issue description
then

$ docker images
REPOSITORY      TAG               IMAGE ID       CREATED       SIZE
moby/buildkit   buildx-stable-1   3ca743fc1489   3 weeks ago   142MB

$ echo "NOK" > test.txt

$ docker compose build
[+] Building 1.3s (7/7) FINISHED                                                
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 123B                                       0.0s
 => [internal] load metadata for docker.io/library/busybox:latest          1.2s
 => [internal] load build context                                          0.0s
 => => transferring context: 39B                                           0.0s
 => [1/2] FROM docker.io/library/busybox@sha256:ad9bd57a3a57cc95515c537b8  0.0s
 => => resolve docker.io/library/busybox@sha256:ad9bd57a3a57cc95515c537b8  0.0s
 => CACHED [2/2] COPY ./test.txt /                                         0.0s
 => exporting to image                                                     0.1s
 => => exporting layers                                                    0.0s
 => => exporting manifest sha256:5f90d1f5c9c6ca1a0b7aae82d5cfd3989d219211  0.0s
 => => exporting config sha256:316b44986cb2457d98f7fa6a176ad49d7da2c6176a  0.0s

$ docker images
REPOSITORY      TAG               IMAGE ID       CREATED       SIZE
moby/buildkit   buildx-stable-1   3ca743fc1489   3 weeks ago   142MB

but if you run your container by docker compose up

$ docker images
REPOSITORY         TAG               IMAGE ID       CREATED         SIZE
compose_9856-test   latest            316b44986cb2   5 minutes ago   1.24MB
moby/buildkit      buildx-stable-1   3ca743fc1489   3 weeks ago     142MB

It seems like 2.11 require to use moby/buildkit and buildx and some new stuff, and lead to break some scenarios like (my GitLab CI loves it)

docker compose build
docker compose push

push command is fully unusable since 2.11 cause it says An image does not exist locally

I started to see moby/buildkit in my images list only since 2.11

[vgolynets]

I have similar issue regarding docker compose build command on 2.11 version. After command complete there are no any images created/tagged. Compose file looks like:

services:
  my-service1:
    image: <aws account id>.dkr.ecr.us-east-1.amazonaws.com/my-service1:1.2.3
    build:
...
  my-service2:
    image: <aws account id>.dkr.ecr.us-east-1.amazonaws.com/my-service2:1.2.3
    build:
...

After some experiments I found that following workaround may help (but not in every build!):

services:
  my-service1:
    image: my-service1:1.2.3
    build:
      tags:
        - <aws account id>.dkr.ecr.us-east-1.amazonaws.com/my-service1:1.2.3
...