[BUG] docker compose exec fails when projectdir is a symlink

[jnweiger]

Description

My docker-compose.yml lives inside some github checkout subdirectory.
As an admin, I make a symlink 'o' in my home, that brings me there.
changing directory to the real path, I run docker compose up -d
Then, when using the symlink, docker compose commands either silently do nothing, or produce an error.
When using the real path, the same docker compose commands work just fine.

(Also the other way around: You always have to use the symlink after starting with cd ~/o; docker compose up -d)

docker compose ls shows which variant works.

Expected behavior:

It should make no difference, how I navigated my shell to the project directory.

Steps To Reproduce

# cd
# mkdir -p tedious-long/path/for/testing
# ln -s tedious-long/path/for/testing o
# cat <<EOF > o/docker-compose.yml
---
services:
  hello:
    image: ubuntu:latest
    command: [ 'bash', '-c', 'while true; do date; sleep 1; done' ]
EOF
# cd tedious-long/path/for/testing
# ls -la
total 12
drwxr-xr-x  2 root root 4096 Jul 26 11:18 .
drwx------ 10 root root 4096 Jul 26 12:14 ..
-rw-r--r--  1 root root  117 Jul 26 11:18 docker-compose.yml
docker compose up -d
# cd ~/o
# ls -la
total 12
drwxr-xr-x  2 root root 4096 Jul 26 11:18 .
drwx------ 10 root root 4096 Jul 26 12:14 ..
-rw-r--r--  1 root root  117 Jul 26 11:18 docker-compose.yml
# docker compose logs
# cd ~/tedious-long/path/for/testing
# docker compose logs
hello-1  | Fri Jul 26 12:38:04 UTC 2024
hello-1  | Fri Jul 26 12:38:05 UTC 2024
hello-1  | Fri Jul 26 12:38:06 UTC 2024
hello-1  | Fri Jul 26 12:38:07 UTC 2024
hello-1  | Fri Jul 26 12:38:08 UTC 2024
hello-1  | Fri Jul 26 12:38:09 UTC 2024
hello-1  | Fri Jul 26 12:38:10 UTC 2024
hello-1  | Fri Jul 26 12:38:11 UTC 2024
hello-1  | Fri Jul 26 12:38:12 UTC 2024
hello-1  | Fri Jul 26 12:38:13 UTC 2024
hello-1  | Fri Jul 26 12:38:14 UTC 2024
# cd ~/o
# docker compose logs
# docker compose exec hello id
service "hello" is not running
# cd ~/tedious-long/path/for/testing
# docker compose exec hello id
uid=0(root) gid=0(root) groups=0(root)

Workaound

In bash, use set -o physical

Compose Version

Docker Compose version v2.29.1

Docker Environment

Client:
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 6
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-116-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.566GiB
 Name: jw-bt4
 ID: b1ad372b-c7d8-418f-bd86-d05d7e5f4091
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

I can see several usages of EvalSymlinks() in pkg/compose/watch.go, but none in e.g. cmd/compose/compose.go

Please note that filepath.Abs(".") happily constructs absolute paths with unresolved symlinks. That's a pity, as it is often used.
I tested the symlink behaviour with:

package main

import (
	"fmt"
	"path/filepath"
)

func main() {
	fmt.Printf("Dir: %q\n", filepath.Dir("index"))
	p, err := filepath.Abs(".")
	fmt.Printf("Abs: %q, %q\n", p, err)
	e, err := filepath.EvalSymlinks(p)
	fmt.Printf("EvalSymlinks: %q, %q\n", e, err)
}

My assumption is that golang's filepath.Abs() happily relies on the PWD environment variable from the shell, which is weird.

[teodorescuserban]

I am also using the symlink trick quite extensively... Any solution for docker compose v2 please? :\

Maybe I could live with a new COMPOSE_PROJECT_DIRECTORY env var for --project-directory option, because that seems to work pretty well.

[teodorescuserban]

For now, I settled with

export COMPOSE_FILE=$(readlink -f /path/to/symlink/dir/docker-compose.yml`

or

cd /path/to/symlink/dir
export COMPOSE_FILE=$(readlink -f ./docker-compose.yml)

since -f compose_file is working as well.

It would still be nice to have the COMPOSE_DIR env var available. Or just fix compose to take eval the symlinks.

[ndeloof]

The root cause for this behavior is that project name (which is used to filter resources) is inferred from parent folder for your compose file. If you use symlink o to access your compose folder, then project name is o and docker compose logs will search for resources with label com.docker.compose.project=o .. which doesn't match anything.

As a workaround, you could rely on --project-name so you don't even need to cd inside project directory:

$ docker compose -p testing logs