[BUG] Multi-document YAML files does not extend for service in same config without explicitly stating file

[ToshY]

Description

I was trying the new multi-document YAML files feature and stumbled upon an issue regarding extends. Previously I've been using a common-services.yml and put in some services which my docker-compose.yml could extend from. Now using the multi-document setup, I've put those "base" services in the same docker-compose.yml file, removed file: common-services.yml from the extending services, and thought it would work.

When performing any kind of docker compose command for the current project, e.g. docker compose config or docker compose up, I encounter the following error:

cannot extend service "php-base" in /home/<user>/myproject/docker-compose.yml: service not found

But if I explictely specify file: docker-compose.yml in extend for those services it has no problem upping them.

---

So for now, I can workaround the issue by explictely stating the file to point to itself, but as the docs state that this should be optional makes me think this is a new bug (possibly not covered by new test cases for multi-document YAML files).

Steps To Reproduce

Not working example

1. Use the following reproduction case.

/home/<user>/myproject/docker-compose.yml

version: '3.9'

services:
  php-base:
    image: php:latest
---
services:
  php-worker:
    extends:
      service: php-base
    restart: always

2. Run docker compose config

cannot extend service "php-base" in /home/<user>/myproject/docker-compose.yml: service not found

---

Working example

1. Use the following reproduction case.

/home/<user>/myproject/docker-compose.yml

version: '3.9'

services:
  php-base:
    image: php:latest
---
services:
  php-worker:
    extends:
      file: docker-compose.yml
      service: php-base
    restart: always

2. Run docker compose config

name: myproject
services:
  php-base:
    image: php:latest
    networks:
      default: null
  php-worker:
    image: php:latest
    networks:
      default: null
    restart: always
networks:
  default:
    name: myproject_default

Compose Version

Docker Compose version v2.21.0

Docker Environment

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /home/<user>/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /home/<user>/.docker/cli-plugins/docker-compose
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-dev: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-extension" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-extension: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-sbom" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-sbom: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 54
  Running: 29
  Paused: 0
  Stopped: 25
 Images: 173
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.90.3-microsoft-standard-WSL2
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.761GiB
 Name: DESKTOP-TTP757E
 ID: 0e6a9e8e-5b9e-4049-baf9-1efaf7f3d139
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

extends used without a file require the referred service to be declared inside the same yaml document. With multi-document feature, a single file can have many, but there's no way to select a specific document inside such a file.

Can you give more details on your usage of multi-document yaml here, and why you prefer this approach over multiple files?

[ToshY]

Hey @ndeloof 👋

extends used without a file require the referred service to be declared inside the same yaml document. With multi-document feature, a single file can have many, but there's no way to select a specific document inside such a file.

As I'm fairly new to the whole multi-document YAML way to begin with, this is where I started to get confused, as the documentation currently states that it looks for the service configuration in the "current file", but as you've currently stated it, it looks inside the "same yaml document".

• In that case, do I understand it correctly that if the keyword file is omitted, it actually no longer refers to the current "file" but actually the current YAML "document"? And if so, would it be a good idea to change the wording in the extends documentation to refer to "current YAML document" instead of "current file" (and possibly add a link to refer to the multi-document section of the documentation)?

If the above is correct, I can close this issue and create a follow-up request in docker/docs.

Can you give more details on your usage of multi-document yaml here, and why you prefer this approach over multiple files?

I don't really have an actual preference (yet), it's more because I was curious testing the new feature.

While I stated that I use multiple files currently, I initially didn't realize that changing to a multi-document way actually changes the way it works, as it will up the "base" service alongside the other services now as well, while previously with extends from another file it wouldn't up the "base" service. In hindsight this makes sense as the feature is about multi-document files after all (🤦‍♂️), and this gives me the opportunity to do the following:

Multiple files (old way)

common-services.yml

version: '3.9'

services:
  php-base:
    image: php:latest

docker-compose.yml

version: '3.9'

services:
  php-main:
    extends:
      service: php-base
      file: common-services.yml

  php-worker:
    extends:
      service: php-base
      file: common-services.yml
    restart: always

Multi-document (new way)

docker-compose.yml

version: '3.9'

services:
  php-main:
    image: php:latest
---
services:
  php-worker:
    extends:
      service: php-main
      file: docker-compose.yml
    restart: always

---

Personally for me these are the pros:

• Multiple files:
  • Using additional file like common-services.yaml gives me a better idea on which services extend from the "base" service, making the docker-compose.yml itself a little bit cleaner/nicer to read.
• Multi-document:
  • Having every service I need in one single docker-compose.yml.
  • Less config needed to achieve the same.

[ndeloof]

wording indeed need to be updated. Compose has not been designed with multi-document yaml file in mind, this recent addition require some updates.

Still, I need to double check this specific scenario. As long as the extended service exists in same compose file EARLIER in the set of document, and as such has already been parsed, we SHOULD be able to retrieve it's definition and extend it.
Let's keep this issue open while I'm investigating this

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.