[BUG] compose watch combined with depends_on can lead to failure of dependencies

[nikita-vanyasin]

Description

I've started to use 'watch' feature for one project, but it fails every time when I change files from state A to state B and then back to state A.

Steps To Reproduce

1. Compose yaml (truncated):

apiserver:
  build:
    context: ./backend
    dockerfile: api.Dockerfile
  container_name: apiserver
  pull_policy: build
  restart: always
  depends_on:
    - "db"
    - "rabbit"
  expose:
    - 3000
  volumes:
    - /etc/localtime:/etc/localtime:ro
  develop:
    watch:
      - path: ./backend/
        action: rebuild

analytics:
  build:
    context: ./backend
    dockerfile: analytics.Dockerfile
  container_name: analytics
  pull_policy: build
  restart: always
  depends_on:
    - "db"
    - "rabbit"
  volumes:
    - /etc/localtime:/etc/localtime:ro
  develop:
    watch:
      - path: ./backend/
        action: rebuild

# ...
# more services here. No more "depends_on" statements.

2. Run watch: docker compose watch.
3. Edit file under ./backend folder, e.g. add new line.
4. Watch detects changes and rebuilds both services as expected.
5. Edit same file under ./backend folder bringing it to the state it was before step 3 (remove new line).
6. Watch starts to rebuild both services, but fails:

...
service "apiserver" successfully built
Failed to recreate service after update. Error: Error response from daemon: Conflict. The container name "/eaad56074ed9_db" is already in use by container "30b1ad6dcf9135f90dcef865d7de2bfaf1e6a2faa4b3c1ac56351d97e18cf2fa". You have to remove (or rename) that container to be able to reuse that name.
WARN[0011] Error handling changed files for service apiserver: Error response from daemon: Conflict. The container name "/eaad56074ed9_db" is already in use by container "30b1ad6dcf9135f90dcef865d7de2bfaf1e6a2faa4b3c1ac56351d97e18cf2fa". You have to remove (or rename) that container to be able to reuse that name. 
Failed to recreate service after update. Error: Error response from daemon: No such container: 30b1ad6dcf9135f90dcef865d7de2bfaf1e6a2faa4b3c1ac56351d97e18cf2fa
WARN[0012] Error handling changed files for service analytics: Error response from daemon: No such container: 30b1ad6dcf9135f90dcef865d7de2bfaf1e6a2faa4b3c1ac56351d97e18cf2fa 

As you can see, the error mentions container db. This container dies.
Looks like compose is trying to restart dependencies after rebuilding apiserver but somehow conflict occurs.

The problem does not occur when I run watch only for one service (docker compose watch apiserver)

Compose Version

# docker compose version
Docker Compose version 2.26.1

# docker-compose version
Docker Compose version 2.26.1

Docker Environment

Client:
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.13.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.26.1
    Path:     /usr/lib/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.1.0-280-gc7fa31d4c4
    Path:     /usr/lib/docker/cli-plugins/docker-scan

Server:
 Containers: 7
  Running: 7
  Paused: 0
  Stopped: 0
 Images: 632
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dcf2847247e18caba8dce86522029642f60fe96b.m
 runc version:
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.2-arch2-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.08GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.168.128.0/20, Size: 24

Anything else?

This might be related to #9014

[nikita-vanyasin]

After some more time using watch feature, I can say that the bug in the topic starter does not seem related to "depends_on" field. I've removed all depends_on fields and still getting same error about container conflict

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hadasarik]

Hey @nikita-vanyasin I encountered this too - Any idea what's causing this? Did you find a fix?

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[nikita-vanyasin]

Hey @nikita-vanyasin I encountered this too - Any idea what's causing this? Did you find a fix?

Hello!
No idea for now. I'm also seeing this error even if changing from state A to B.
It is also reproducible on freshly installed MacOS + Docker Desktop.

[nikita-vanyasin]

I see that error is a bit different with new version and running under MacOS+DockerDesktop:

service "backend" successfully built
Failed to recreate service after update. Error: Error response from daemon: removal of container 48731f86401f10ce0def0ea4418edde701fbe95aafdf4d5930ef44760f593f55 is already in progress
WARN[0030] Error handling changed files for service backend: Error response from daemon: removal of container 48731f86401f10ce0def0ea4418edde701fbe95aafdf4d5930ef44760f593f55 is already in progress
Failed to recreate service after update. Error: Error response from daemon: No such container: 8d1229e84b13cff2f51505bbd84f28db564705bf3cd9b989830a25d9d5807473
WARN[0030] Error handling changed files for service analyticsupdater: Error response from daemon: No such container: 8d1229e84b13cff2f51505bbd84f28db564705bf3cd9b989830a25d9d5807473

[nikita-vanyasin]

Hey @nikita-vanyasin I encountered this too - Any idea what's causing this? Did you find a fix?

I've figured out that problem occurs only when services apiserver and analytics are watching for same folder while building images with shared base layers. This looks like a race condition.

I've ended up to use next workaround:

• Added a no-op service to build "base image" (let's call it 'backend')
• This 'backend' image contains all binaries for both apiserver and analytics services (this is actually makes sense for my use-case as they share most of source code)
• 'watch' is enabled only for "base image builder" service
• Added 'depends_on': ['baseimage'] to apiserver and analytics. These images just start different binaries from base image.

This works fine, although it is not quite a use-case for which docker compose was disigned. When code changes, both services are restarted. The no-op container exits immediately after start:

backend_build:
  image: "${DOCKER_NAMESPACE}/backend:${BACKEND_TAG}"
  restart: no
  build:
    context: ./backend
    dockerfile: Dockerfile
  develop:
    watch:
      - path: ./backend/
        action: rebuild
  ports: [] # expose nothing
  command: "sh -c 'echo \"all good!\" && exit 0'"

apiserver:
  image: "${DOCKER_NAMESPACE}/backend:${BACKEND_TAG}"
  command: ["/apiserver"]
  restart: always
  depends_on:
    - "backend_build"
  expose:
    - 3000
  volumes:
    - /etc/localtime:/etc/localtime:ro


analytics:
  image: "${DOCKER_NAMESPACE}/backend:${BACKEND_TAG}"
  command: ["/analytics"]
  restart: always
  depends_on:
    - "backend_build"
  volumes:
    - /etc/localtime:/etc/localtime:ro