[Bug]: compose.dockerCompose.Up errors out with "failed to connect to reaper: dial tcp [::1]:49485: connect: connection refused: Connecting to Ryuk on localhost:49485 failed", although containers are up & running

[aggarwalpiyush01]

Testcontainers version

0.31.0

Using the latest Testcontainers version?

Yes

Host OS

Debian GNU/Linux 11 (bullseye)

Host arch

x86-64

Go version

1.22.0

Docker version

Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.15
 Git commit:        55c4c88
 Built:             Mon May 30 18:34:49 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.5+dfsg1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.15
  Git commit:       363e9a8
  Built:            Mon May 30 18:34:49 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.13~ds1
  GitCommit:        1.4.13~ds1-1~deb11u4
 runc:
  Version:          1.0.0~rc93+ds1
  GitCommit:        1.0.0~rc93+ds1-5+deb11u3
 docker-init:
  Version:          0.19.0
  GitCommit:

Docker info

Client:
 Context:    default
 Debug Mode: true
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.5)
  compose: Docker Compose (Docker Inc., v2.18.1)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 182
 Server Version: 20.10.5+dfsg1
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1.4.13~ds1-1~deb11u4
 runc version: 1.0.0~rc93+ds1-5+deb11u3
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.0-28-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 39.18GiB
 Name: vdi-bglddp-099
 ID: 3NOF:C5EQ:B3QX:HPR2:GADA:5I6N:Q33R:AIBP:K5QL:LBXN:ARCV:H6JK
 Docker Root Dir: /local-ssd/paggarwa/docker-root/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.168.0.0/16, Size: 24

WARNING: Support for cgroup v2 is experimental

What happened?

Below is my reproduction code. The code simply uses the methods available in the testcontainers-go/modules/compose package to run a docker-compose.yml file. The relevant output logs I have provided shows that the cmp.Up call errors out with failed to connect to reaper: dial tcp [::1]:49533: connect: connection refused: Connecting to Ryuk on localhost:49533 failed. Although the logs also show that all the containers were up & healthy, indicating that the compose file was started correctly.

The bug here is that cmp.Up API seems to be returning a non nil error unnecessarily, even though the compose file ran correctly. Expectation for correct result is a nil error from cmp.Up

func testMain(m *testing.M) int {
    cmp, e := compose.NewDockerComposeWith(compose.WithStackFiles(dockerComposeFilePath))
    if e != nil {
        log.Printf("error creating docker-compose: %v", e)
        return 1
    }
    
    e = cmp.Up(context.Background(), compose.Wait(true))
    if e != nil {
        log.Printf("error running docker-compose: %v", e)
        return 1
    }
    defer func(){
        if e := cmp.Down(context.Background(), compose.RemoveOrphans(true)); e != nil {
            log.Printf("error shutting docker-compose: %v", e)
        }
    }()
    
    return m.Run()
}

func TestMain(m *testing.M) {
    os.Exit(testMain(m))
}

// Test Methods to be run which use these microservice containers

Relevant log output

2024/06/01 19:51:59 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 20.10.5+dfsg1
  API Version: 1.41
  Operating System: Debian GNU/Linux 11 (bullseye)
  Total Memory: 40115 MB
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 4eb4cd082a793d05e11fe6254b279fc83ef372e938738c8f3b5909a8762604ed
  Test ProcessID: 3a6ccc38-eef3-45e3-8b6c-13f0b542c1f3
2024/06/01 19:51:59 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/06/01 19:51:59 ✅ Container created: cac4d3f9aed6
2024/06/01 19:51:59 🐳 Starting container: cac4d3f9aed6
2024/06/01 19:52:00 ✅ Container started: cac4d3f9aed6
2024/06/01 19:52:00 🚧 Waiting for container id cac4d3f9aed6 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/06/01 19:52:00 🔔 Container is ready: cac4d3f9aed6
 Network f1ced4ca-f582-47ea-8a19-e91de40d74e3_default  Creating
 Network f1ced4ca-f582-47ea-8a19-e91de40d74e3_default  Created
 Volume "f1ced4ca-f582-47ea-8a19-e91de40d74e3_vol1"  Creating
 Volume "f1ced4ca-f582-47ea-8a19-e91de40d74e3_vol1"  Created
 Volume "f1ced4ca-f582-47ea-8a19-e91de40d74e3_vol2"  Creating
 Volume "f1ced4ca-f582-47ea-8a19-e91de40d74e3_vol2"  Created
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service1  Creating
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service2  Creating
 Container service6  Creating
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service3  Creating
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service3  Created
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service1  Created
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service2  Created
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service4  Creating
 Container service6  Created
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service5  Creating
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service4  Created
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service5  Created
 Container service6  Starting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service3  Starting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service2  Starting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service1  Starting
 Container service6  Started
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service2  Started
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service4  Starting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service1  Started
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service5  Starting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service3  Started
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service4  Started
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service5  Started
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service4  Waiting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service5  Waiting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service2  Waiting
 Container service6  Waiting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service3  Waiting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service1  Waiting
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service3  Healthy
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service1  Healthy
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service5  Healthy
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service4  Healthy
 Container f1ced4ca-f582-47ea-8a19-e91de40d74e3-service2  Healthy
 Container service6  Healthy

error running docker-compose: failed to connect to reaper: dial tcp [::1]:49533: connect: connection refused: Connecting to Ryuk on localhost:49533 failed
FAIL	test	14.967s

Additional information

No response

[mdelapenya]

Hi @aggarwalpiyush01 thanks for opening the issue. What is the value for ryuk.reconnecion.timeout in your testcontainers.properties file?

I ask because I've experimented this error when ryuk is waiting more than that time (default is 10s) because the compose services take more time than that. My recommendation is to increase that number to, say, 30s?? and check again. I thought I had documented in #2485, but it seems not 😞 You can check that I do it for the CI here: https://github.com/testcontainers/testcontainers-go/pull/2485/files#diff-fcb3e1d60af8eeb3cd33971e4989185a1cbbcfff6cdc46c88605074bd6fa4e92R53

I'll open a PR updating the compose docs for it.

Thank you so much for your report 🙏

[maxmzkr]

Would it be reasonable for us to hold a ryuk connection during the duration of the call to Up? It seems like we know ryuk shouldn't exit while we're calling Up so it would be okay to hold the connection open? In the case I talked about over in #2618, increasing the reconnect timeout would help, but it would be nice to have our tests work out of the box without having to change config if possible

[mdelapenya]

@maxmzkr I think we can use your above comment as a feature request in a separate ticket, if you agree. Otherwise, let's reopen this one 🙏

[mdelapenya]

@aggarwalpiyush01 were you able to check and configure the Ryuk timeouts properly?