Can't connect to Ryuk when using when running in container and using Docker-Socket-Mounting on Docker Desktop

[jrussellsmyth]

When running testcontainer based test within another container (vscode devcontainer)

Tests that executed flawlessly on 1.15.3 now fail attempting to start the compose configuration

java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:58364
at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:227) ~[testcontainers-1.16.0.jar:?]
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:204) ~[testcontainers-1.16.0.jar:?]
at org.testcontainers.containers.DockerComposeContainer.(DockerComposeContainer.java:135) ~[testcontainers-1.16.0.jar:?]
at org.testcontainers.containers.DockerComposeContainer.(DockerComposeContainer.java:119) ~[testcontainers-1.16.0.jar:?]

We did attempt disabling Ryak, but that just results in the same error attempting to connect to the first container.

Docker Desktop 3.6.0
macOS Mojave version 10.14.6
intel mac

[jrussellsmyth]

Created a minimal test case for this here demonstrating expectations with 1.15.3 and failure with 1.16.0
https://github.com/jrussellsmyth/testcontainer-problem

[kiview]

Hi @jrussellsmyth,
thanks a lot for reporting this issue and providing us with a reproducer.

This problem also occurs on Windows in WSL2. In addition, it also happens with GenericContainer. Therefore, I assume this has nothing to do with DockerComposeContainer or Docker for Mac and it seems there is a regression in 1.16.0 for the Docker Wormhole Pattern (i.e., socket mounting) in general.

[Nayshins]

Is there a timeline for this to be fixed? I'm stuck in a bit of a bind here of needing to be on 1.16.+ for M1 support, but this issue is blocking me from using this version with docker for mac.

If anyone has a suggestion for a workaround that would be greatly appreciated!

[kiview]

No ETA yet, see the PR #4396 and the possible implications from upstream Docker here:
docker/for-mac#5588 (comment)

What is your use case for running the tests inside of Docker Compose in the first place? Maybe if we think about this, we can come up with a workaround or alternative setup.

[jonathan-meier]

Our use case is a development environment inside a docker container using the VS Code Remote Containers extension running on a local instance of Docker Desktop for Windows (WSL2). We have tests using testcontainers that need to be runnable in that environment. This works fine with testcontainers up to version 1.15.3, but we can't update to version 1.16.0 due to this issue.

Any advice on how we might work around this issue until PR #4396 is merged or a more general solution is available?

[kiview]

Hi @jonathan-meier, as you can see from the comment in the upstream issue, there might exist different sockets that need to be mounted for the docker-wormhole pattern to work again. However, I did not yet figure out how to make it work on Docker for Desktop with WSL2.

As a temporary workaround, you could try out to consume the PR as a Jitpack dependency:
https://www.testcontainers.org/jitpack_dependencies/

It would also be interesting to know, if this mitigates the issue for your setup as well.

[ggili]

hi all

I came across to this issue which I think its related to mine (opened last week). Doe I am. using also a mac , but it happens as well on linux CentOS
#4685
Does the PR on docker is providing a fix for both versions? (linux and mac)?

[jonathan-meier]

Hi @kiview, thanks a lot for your answer. I saw that there might be a workaround using a different socket, however, I'm not sure if we could use it, because our configuration needs to be platform independent, i.e. work on Docker Desktop with WSL2 as well as Docker on Linux.

Using jitpack as you suggest, I can confirm that PR #4396 solves the problem for our setup, so we would love to see it make its way into the next released version.

[kiview]

@ggili It is very unlikely that this issue appears on Linux as well since it is related to the VM setup of Docker Desktop.

@jonathan-meier Thanks for confirming that the PR solves the issue for you, we will look into merging it.

[ggili]

@kiview
Yes indeed. I was too fast on conclusion. On mac that might be a solution . Was investigating our gitlabrunner issue further on my mac.

However I just tried on our centos gitlabrunner. Indeed -p 1234:80 or -p 80 doesn't matter. The containers cannot talk to eachother via the gateway. Where on Mac they can.

`docker run -dp 1234:80 docker/getting-started

"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "423fde3a0e3896a69512aa3c8fa2016a1d5e2c976707b423e8accc075e3c17e7",
"EndpointID": "d46fd61a22e4fcceb9c6229405d5db747248981d8f56de4f1ad112c3adff3197",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.3",
`

`docker run -it maven:3.3-jdk-8 /bin/bash

Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "423fde3a0e3896a69512aa3c8fa2016a1d5e2c976707b423e8accc075e3c17e7",
"EndpointID": "ecdd25978e06d056ff62a51fc82910bd34cf638439aa3feafaed975b3460c067",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.5",
`

curl 172.17.0.1:1234 - works well from host, but not from maven:3.3-jdk-8 container. Only via curl 172.17.0.5:8080

I will try digging a bit later, but for me seems that is also a regression on docker. Or there might be something on our Centos which blows things up.

[kiview]

Since I remember a regression on Fedora from some time ago, it might be about the firewalld configuration on CentOS.

[jonathan-meier]

I just saw that version 1.16.3 was released, but sadly without PR #4396. I understand that there is some risk involved with merging PR #4396 as is, though what would you think about hiding the workaround behind an opt-in configuration flag?

[BoraxTheClean]

On Docker Desktop for Mac (M1) 4.9.0, I was able to work around this issue by mounting the docker socket into the container in this fashion: -v /var/run/docker.sock.raw:/var/run/docker.sock

[bozo-ivetic-cif]

On Docker Desktop for Mac (M1) 4.9.0, I was able to work around this issue by mounting the docker socket into the container in this fashion: -v /var/run/docker.sock.raw:/var/run/docker.sock

I run into the same problem and this workaround did the trick. macOS 13, testcontainers:1.17.5 and Docker Desktop 4.13.1 (90346)

[kiview]

I can confirm that -v /var/run/docker.sock.raw:/var/run/docker.sock also work in WSL2 now. Therefore, I recommend to extend the docs with this information and then close this issue.

[jxblum]

/var/run/docker.sock.raw is not even a file on macOS 13.3.1 with Docker Desktop 4.19.0 (106363) using Testcontainers 1.18.1:

$ ls -la /var/run/docker.sock.raw
ls: /var/run/docker.sock.raw: No such file or directory

I still see:

Caused by: java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:59391
	at org.testcontainers.utility.RyukResourceReaper.maybeStart(RyukResourceReaper.java:132)
	at org.testcontainers.utility.RyukResourceReaper.init(RyukResourceReaper.java:42)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:224)
	at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:104)
	at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:109)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:321)
	at example.app.crm.config.TestcontainersCassandraConfiguration.cassandraContainer(TestcontainersCassandraConfiguration.java:71)
...
..
.

I am attempting to run tests using Testcontainers to bootstrap a Apache Cassandra 3.11.15 container with:

$ docker run -it --mount type=bind,source="$(pwd)",target=/source-from-github -v /var/run/docker.sock:/var/run/docker.sock -v /usr/local/bin/docker:/usr/local/bin/docker eclipse-temurin:17.0.6_10-jdk-focal /bin/bash

Essentially, bootstrapping the Cassandra container with:

	private static final DockerImageName CASSANDRA_DOCKER_IMAGE_NAME = DockerImageName.parse("cassandra:3.11.15");

	private static final String LOCAL_DATACENTER_NAME = "datacenter1";

	private static final int CASSANDRA_DEFAULT_PORT = 9042;

	@Bean("CassandraContainer")
	GenericContainer<?> cassandraContainer(Environment environment) {

		GenericContainer<?> cassandraContainer = newEnvironmentOptimizedCassandraContainer();

71:		cassandraContainer.start();

		return withCassandraServer(cassandraContainer, environment);
	}

	private @NonNull GenericContainer<?> newCassandraContainer() {

		return new CassandraContainer<>(CASSANDRA_DOCKER_IMAGE_NAME)
			.withInitScript(CASSANDRA_SCHEMA_CQL)
			//.withInitScript(CASSANDRA_INIT_CQL)
			.withExposedPorts(CASSANDRA_DEFAULT_PORT)
			.withReuse(true);
	}

	private @NonNull GenericContainer<?> newEnvironmentOptimizedCassandraContainer() {

		return newCassandraContainer()
			.withEnv("CASSANDRA_SNITCH", "SimpleSnitch")
			.withEnv("HEAP_NEWSIZE", "128M")
			.withEnv("MAX_HEAP_SIZE", "1024M")
			.withEnv("JVM_OPTS", "-Dcassandra.skip_wait_for_gossip_to_settle=0 -Dcassandra.initial_token=0");
	}

	private @NonNull CqlSession newCqlSession(@NonNull GenericContainer<?> cassandraContainer) {

		return CqlSession.builder()
			.addContactPoint(resolveContactPoint(cassandraContainer))
			.withLocalDatacenter(LOCAL_DATACENTER_NAME)
			.build();
	}

	private @NonNull InetSocketAddress resolveContactPoint(@NonNull GenericContainer<?> cassandraContainer) {
		return new InetSocketAddress(cassandraContainer.getHost(),
			cassandraContainer.getMappedPort(CASSANDRA_DEFAULT_PORT));
	}