[Bug]: Test containers fails to pull image with podman

[fedinskiy]

Module

Core

Testcontainers version

1.17.5

Using the latest Testcontainers version?

Yes

Host OS

Linux, Fedora 36, 5.19.14-200.fc36.x86_64

Host Arch

x86

Docker version

podman version 4.2.1

What happened?

I have a code, which tries to start and stop container. If image, which I use, is not yet pulled by podman, then it fails with NPE, otherwise it succeeds.

How reproduce:

1. Have podman installed
2. Make podman use dockerhub by default(due to [1] ):
   a. Change line short-name-mode setting inside of /etc/containers/registries.conf into "disabled"
   b. Change order of unqualified-search-registries in the same file, so docker.io would be the first one
   c. In file /usr/share/containers/containers.conf uncomment compat_api_enforce_docker_hub = true
3. Clone the reproducer: [email protected]:fedinskiy/reproducer.git -b reproducer/podman-nopull
4. Use podman rmi to purge postgres:9.6.12 and nginx:latest
5. Force usage of podman socket: export DOCKER_HOST=unix:///run/user/${UID}/podman/podman.sock && export TESTCONTAINERS_RYUK_DISABLED=true
6. Inside reproducer folder run mvn clean verify. This will fail.
7. Download the images: podman pull postgress:9.6.12 && podman pull nginx:latest
8. Repeat step 6 again. Now everything runs as expected

[1] #5612

Relevant log output

org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:349)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at io.quarkus.qe.AppTest.testName(AppTest.java:18)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at junit.framework.TestCase.runTest(TestCase.java:177)
	at junit.framework.TestCase.runBare(TestCase.java:142)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:130)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:9.6.12, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@76c7beb3)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1371)
	at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:651)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:331)
	... 27 more
Caused by: java.lang.NullPointerException
	at com.github.dockerjava.api.command.PullImageResultCallback.checkForDockerSwarmResponse(PullImageResultCallback.java:48)
	at com.github.dockerjava.api.command.PullImageResultCallback.onNext(PullImageResultCallback.java:35)
	at org.testcontainers.images.LoggedPullImageResultCallback.onNext(LoggedPullImageResultCallback.java:48)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:73)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:24)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onNext(AbstrAsyncDockerCmdExec.java:41)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:315)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:298)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:275)
	at java.base/java.lang.Thread.run(Thread.java:829)

Additional Information

Similar bug[2] was reported a year before, but it was flaky, has different error output and used older version of podman and testcontainers.
This[3] bug in podman may be related.

[2] #4440
[3] containers/podman#15306

[kiview]

The workaround mentioned in #5612 (comment) is valid for this issue, correct?

[fedinskiy]

@kiview only after I fixed #5612 with the workaround I was able to experience this one :)

The workaround is a part of the reproducer, see point 2.c

[kiview]

@fedinskiy thanks for the clarification. In this case, I think the podman behavior with regards to #5275 is in play here. However, I am surprised by it since I would assume that what you mention as step 1. should also help.

Did you consider also raising the issue with the podman project? Because I believe, ideally podman would implement a compatibility setting, that makes it behave like Docker (which is clearly not the case here, hence the issue you observe).

[fedinskiy]

@kiview, yes, but could you describe, what are expected and actual responses of podman/Docker API in that case, or point me to the relevant class? NullPointerException in checkForDockerSwarmResponse is not a helpful message, I am afraid.

[kiview]

Good point, this is actually an issue in https://github.com/docker-java/docker-java in this case and not in Testcontainers. Since I am not a podman user or expert and we as Testcontainers are consumers of docker-java (although we also help in maintaining it), I can currently only add, that Testcontainers expects a Docker-API compatible container runtime. You can find a more elaborate explanation in our docs under General Docker requirements.

[fedinskiy]

@kiview from my point of view it is not really clear, where is an issue there.
Debugging shows, that PullImageResultCallback(from docker-java) receives ResponseItem object with unexpected null inside status field and that the same object has error field with the following text: initializing source docker://postgres:9.6.12: unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password. The origin of the error is unclear, since this is a pull from public repo and I do not set up any authorisation. Unfortunately, simultaneous use of inheritance, generic classes and callbacks in docker-java makes it almost impossible for me to understand, what is really happening under the hood.

I see a 3 possible options there:

1. Testcontainers use 'docker-java' incorrectly
2. docker-java uses some undocumented API of Docker socket
3. podman implementation of Docker socket doesn't follow the documentation

What do testcontainers/docker-java are trying to retrieve from the socket? Is there a way to see real requests/responses (except traffic sniffing)?

[fedinskiy]

After closer inspection, I found out, that if we use fully-qualified image names, then the error doesn't reproduce.
Example:

-        GenericContainer container = new GenericContainer<>("postgres:9.6.12");
+        GenericContainer container = new GenericContainer<>("docker.io/library/postgres:9.6.12");

Hope, that help.

[kiview]

Thanks, that's what is also mentioned in the linked issues, correct? Different behavior regarding docker.io and library/.

I would suggest bringing this issue to podman, since the Docker compatibility mode clearly behaves differently than the actual Docker.

Thanks a lot for your investigation into this.

[fedinskiy]

@kiview as I wrote before, first it would be nice to know, which one of three projects misbehaves there.

As far as I understand, testcontainers do a (HTTP?) request to socket and receive(in this case), some unexpected response. Could you describe the request(s) and response(s) which are used during run of generic docker image?

[kiview]

Since Testcontainers and docker-java work fine with Docker Desktop and Docker in general, Occam's razor would suggest Podman misbehaving there. This is in line with our findings from the experiments with Podman we did run in 2021: https://www.atomicjar.com/2021/10/docker-on-windows-and-macos/

We are happy to engage with the Podman community into helping them reach full Docker API compatibility, but we don't plan to currently invest more resources from our side. As mentioned above, this is also outlined in our Docker requirements.

Therefore, I am going to close this issue, but we really appreciate your collaboration on this.