Difference in relative path handling for .env files between v1 / v2

[alexdev-wb]

Description

Suppose we have some common service definition:

$ cat deps/common/services.yaml:
services:

  service1:
    image: hello-world
    env_file: ./env/service1.env
    environment:
      - SERVICE1_VAR=s1v2

  service2:
    image: hello-world
    env_file: ./env/service2.env
    environment:
      - SERVICE2_VAR=s2v2

two env files:

$ cat deps/common/env/service[12].env
SERVICE1_VAR2=s1v2
SERVICE2_VAR2=s2v3

Lets try to use common services via extends:

$ cat docker-compose.yaml:

services:
  srv1:
    extends:
      file: ./deps/common/services.yaml
      service: service1
    environment:
      - SERVICE1_VAR=foobar
  srv2:
    extends:
      file: ./deps/common/services.yaml
      service: service2
    environment:
      - SERVICE2_VAR=foobar2

docker compose is looking for env files in path relative to docker-compose.yaml file location, whereas docker-compose is looking for files relative to yaml file location which contain path definition.:

$ pwd
/tmp/compose-replative-path

$ docker-compose config
services:
  srv1:
    environment:
      SERVICE1_VAR: foobar
      SERVICE1_VAR2: s1v2
    image: hello-world
  srv2:
    environment:
      SERVICE2_VAR: foobar2
      SERVICE2_VAR2: s2v3
    image: hello-world
version: '3.9'

$ docker compose config
Failed to load /tmp/compose-replative-path/env/service1.env: open /tmp/compose-replative-path/env/service1.env: no such file or directory

compose-replative-path.tar.gz

Steps To Reproduce

1. Untar config sample
2. docker-compose config
3. docker compose config

Compose Version

$ docker compose version
Docker Compose version v2.16.0

$ docker-compose version
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Docker Environment

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 11
  Running: 8
  Paused: 0
  Stopped: 3
 Images: 66
 Server Version: 23.0.1
 Storage Driver: btrfs
  Btrfs: 
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-6-amd64
 Operating System: Debian GNU/Linux bookworm/sid
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 62.77GiB
 Name: alexdev.pp.ua
 ID: 4CIL:2NHL:VDNH:SDW4:TKOE:KHOI:LB36:JLQE:EN4H:5543:4A5Z:ZAFU
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 10.100.0.0/16, Size: 24

Anything else?

No response

[ndeloof]

Relative path in compose is a nightmare, as for legacy reasons they are computed based on the first compose.yaml file configured. Seems docker compose v1 adds to the confusion by letting extend use a distinct rule to compute relative paths in the extended file in a different way 🤪
None of those behaviors seems correct to me...

[alexdev-wb]

@ndeloof I agree that it is hard to choose one of ways and call it "right". As for me, if both behaviors look like something useful, then you may want to introduce a way how compose file author may choose one of them. If you can't make it perfect, make it adjustable (C). If you need some services configured the same way so that they are available for use in different "end-user" builds, you need to copy everything. Besides the fact that this make it impossible to prepare self-contained infrastructure parts, it introduce some maintenance headaches: you need to update configuration everywhere you use that common parts.

[ndeloof]

@alexdev-wb I tend to agree, main issue is that changing anything in this area will break many existing usages.
A possible way yo fix this would be to introduce an explicit opt-in feature for users to select how they want compose to handle relative paths. This would require some edition in you compose config, but at least would give more control:

extends:
      file: ./deps/common/services.yaml
      resolve_relative_paths: true  # relative paths in extended file are resolved based on deps/common
      resolve_relative_paths: false  # relative paths in extended file are not resolved and included as-is in compose model
      service: service2

[ndeloof]

I tried to reproduce issue as reported but don't get any error, and env_file is loaded as expected from subfolder:

$ docker compose config
services:
  srv1:
    environment:
      SERVICE1_VAR: foobar
      SERVICE1_VAR2: s1v2
      SERVICE2_VAR2: s2v3
  srv2:
    environment:
      SERVICE1_VAR2: s1v2
      SERVICE2_VAR: foobar2
      SERVICE2_VAR2: s2v3

i.e, same as docker compose v1:

$ docker-compose-v1 config
services:
  srv1:
    environment:
      SERVICE1_VAR: foobar
      SERVICE1_VAR2: s1v2
      SERVICE2_VAR2: s2v3
    image: hello-world
  srv2:
    environment:
      SERVICE1_VAR2: s1v2
      SERVICE2_VAR: foobar2
      SERVICE2_VAR2: s2v3
    image: hello-world
version: '3.9'

AFAIK this codebase didn't changed recently, so I don't see reason I get a different behavior (I'm running 2.17)

Could you please checkout code from https://github.com/compose-spec/compose-go
then try docker compose -f loader/testdata/compose-test-extends.yaml config ?

[alexdev-wb]

@ndeloof

It was fixed here compose-spec/compose-go#351 , I believe.

[ndeloof]

oh yes, forgot about this one. Closing this issue as duplicate for #10258