[Bug]: Error (501) Reason: connection reset by peer

[phoenix147]

Testcontainers version

0.23.0

Using the latest Testcontainers version?

Yes

Host OS

Ubuntu 22.04

Host arch

x86-64

Go version

1.19

Docker version

Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:44 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:44 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 19
  Running: 9
  Paused: 0
  Stopped: 10
 Images: 370
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-31-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.03GiB
 Name: gat-00399
 ID: 14c7b8fb-da80-4aa7-ace9-7138fc0e8efa
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

We recently upgraded from testcontainers v0.13.0 to v0.23.0 and ran into this problem when using testcontainers on Github Actions. We only have this with problem with the rabbitmq testcontainer. Postgres container for example seem to be fine.

Reverting to 0.13.0 fixes the issue, but of course we would like to stay on a recent version.

Sometimes it's working, but most of the time at least one of the test fails on Github Actions.
Locally (Ubuntu 22.04) everything runs smoothly.

This also seems to be related with #1248

We're starting the testcontainer like that:

	var env = map[string]string{
		"RABBITMQ_DEFAULT_USER": "***",
		"RABBITMQ_DEFAULT_PASS": "***",
	}

	exposedPort := "5672/tcp"

	request := testcontainers.ContainerRequest{
		Image:        "rabbitmq:3.12-management-alpine",
		ExposedPorts: []string{exposedPort},
		Env:          env,
		WaitingFor:   wait.ForListeningPort(nat.Port(exposedPort)),
		HostConfigModifier: func(hc *container.HostConfig) {
			hc.AutoRemove = true
		},
	}

	container, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: request,
		Started:          true,
		Logger:           log.GetLogger(ctx),
	})

Relevant log output

Exception (501) Reason: \"read tcp [::1]:54704-\u003e[::1]:32774: read: connection reset by peer\"\n"

Additional information

No response

[mdelapenya]

Hey @phoenix147 I'm not able to reproduce this error yet, although I have one question: why do you need the host config modifier to set the AutoRemove of the container? Are you using Ryuk BTW?

[mdelapenya]

Sometimes it's working, but most of the time at least one of the test fails on Github Actions. Locally (Ubuntu 22.04) everything runs smoothly.

I think this could be caused by the Ryuk reaper trying to remove the container and Docker auto-removing it. Could you also try without the host config modifier?

[phoenix147]

@mdelapenya we had the (now deprectated) Autoremove flag before, and i migrated it tho the host config modifier when upgrading to v.0.23.0
nevertheless i tried to run the tests also without Autoremove, because i had the same idea, but sadly the problem stays...

And yes we are using Ryuk and from the documentation i gathered, that Ryuk tries to kill the container after 10 seconds, but from starting the container until the error, it takes only about 5-7 seconds (i think rabbitmq takes that long to start)

[mdelapenya]

i think rabbitmq takes that long to start

I'm working on a RabbitMQ module, which by default uses:

wait.ForLog(".*Server startup complete.*").AsRegexp().WithStartupTimeout(60 * time.Second)

Depending if SSL is enabled, it should be extended to check for other log entries: see https://www.rabbitmq.com/ssl.html#enabling-tls-verify-configuration

[phoenix147]

I think the StartupTimeout is 60 seconds by default anyway, but i'll try this strategy, because i also assume that rabbitmq is not fully up even when the port is already exposed..i will let you know..

[phoenix147]

@mdelapenya Seems like it is working with the Log waiting strategy, the tests went through 5 times in a row.
Since the regexp part is not released yet, i'll wait before merging this into our main, but we can close this for now i think.

Thx for your help! :)