[Bug]: Ryuk does not work with podman #2264

dfioravanti · 2024-02-23T17:00:57Z

Testcontainers version

0.28

Using the latest Testcontainers version?

Yes

Host OS

MacOs

Host arch

ARM

Go version

1.22

Docker version

Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.41 (downgraded from 1.44)
 Go version:        go1.21.6
 Git commit:        4debf411d1
 Built:             Tue Feb  6 20:42:40 2024
 OS/Arch:           darwin/arm64
 Context:           default

Server: linux/arm64/fedora-39
 Podman Engine:
  Version:          4.9.0
  APIVersion:       4.9.0
  Arch:             arm64
  BuildTime:        2024-01-24T11:07:09+01:00
  Experimental:     false
  GitCommit:
  GoVersion:        go1.21.6
  KernelVersion:    6.7.4-200.fc39.aarch64
  MinAPIVersion:    4.0.0
  Os:               linux
 Conmon:
  Version:          conmon version 2.1.8, commit:
  Package:          conmon-2.1.8-2.fc39.aarch64
 OCI Runtime (crun):
  Version:          crun version 1.14
commit: 667e6ebd4e2442d39512e63215e79d693d0780aa
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  Package:          crun-1.14-1.fc39.aarch64
 Engine:
  Version:          4.9.0
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:
  Built:            Wed Jan 24 11:07:09 2024
  OS/Arch:          linux/arm64
  Experimental:     false

Docker info

Client: Docker Engine - Community
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 4.9.0
 Storage Driver: overlay
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: false
  Using metacopy: true
  Supports shifting: true
  Supports volatile: true
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge macvlan ipvlan
  Log: k8s-file none passthrough journald
 Swarm: inactive
 Runtimes: kata krun runj youki crun crun-wasm ocijail runc runsc
 Default Runtime: crun
 Init Binary:
 containerd version:
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: default
  selinux
 Kernel Version: 6.7.4-200.fc39.aarch64
 Operating System: fedora
 OSType: linux
 Architecture: arm64
 CPUs: 4
 Total Memory: 1.904GiB
 Name: localhost.localdomain
 ID: c9c1ca7e-0a95-4d0d-b573-b82806d74d03
 Docker Root Dir: /var/lib/containers/storage
 Debug Mode: false
 Experimental: true
 Live Restore Enabled: false
 Product License: Apache-2.0

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: IPv4 forwarding is disabled
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

What happened?

I wanted to switch from docker-desktop to podman as docker provider on MacOs. Everything works but only if I disable ryuk. Otherwise testcontainer crashes at boot up with

getting request provider: Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed: failed to create container

Any idea how to fix this? I have tried everything that I can google, podman is run as rootful and the socket is correctly configured. Testcontainers with ryuk disabled works as expected but it would be nice if it would just work without requiring disabling it.

Relevant log output

No response

Additional information

I am using this project to trigger the issue in case you want to reproduce: https://github.com/dfioravanti/go-rest. Just run go test ./... in the root.

The text was updated successfully, but these errors were encountered:

godart · 2024-04-17T11:11:49Z

Running into the same issue. The test I used to recreate the issue looks like:

import (
	"context"
	"testing"

	"github.com/testcontainers/testcontainers-go"
)

func TestPodman(t *testing.T) {
	ctx := context.Background()
	_, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{}, // never evaluated ...
	})
	if err != nil {
		t.Fatalf("could not start container: %v", err)
	}
}

I'd expect it to fail with something hinting towards the image name being empty.

The output is

=== RUN   TestPodman
2024/04/17 12:04:54 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 5.0.1
  API Version: 1.41
  Operating System: fedora
  Total Memory: 1952 MB
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 748e590ad3a32566931082a26236b0bed5703256a2a60b218eb5f4bfd72a9b76
  Test ProcessID: 6986206b-fbed-4e04-8f4e-d2ba49e32112
2024/04/17 12:04:54 🐳 Creating container for image testcontainers/ryuk:0.7.0
    model_test.go:36: could not start container: Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed: failed to create container
--- FAIL: TestPodman (0.09s)
=== RUN   TestPodman
2024/04/17 12:04:54 🐳 Creating container for image testcontainers/ryuk:0.7.0
    podman_testcontainers_test.go:16: could not start container: Error response from daemon: container create: unable to find network with name or ID bridge: network not found: creating reaper failed: failed to create container
--- FAIL: TestPodman (0.01s)


FAIL

using Podman 5.0.1 here:

~> podman version
Client:       Podman Engine
Version:      5.0.1
API Version:  5.0.1
Go Version:   go1.22.1
Git Commit:   946d055df324e4ed6c1e806b561af4740db4fea9
Built:        Mon Apr  1 13:28:31 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.0.1
API Version:  5.0.1
Go Version:   go1.21.8
Built:        Mon Apr  1 01:00:00 2024
OS/Arch:      linux/arm64

godart · 2024-04-17T19:45:23Z

I debugged a bit:

The current auto-detection of Podman looks for "podman.sock" within the socket name. This works with Podman on Linux, but when using podman-mac-helper for MacOS, the socket name (on the host) is /var/run/docker.sock. (https://github.com/testcontainers/testcontainers-go/blob/main/provider.go#L106)

Setting ProviderType in the test code like this

	_, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{}, // never evaluated ...
		ProviderType:     testcontainers.ProviderPodman,
	})

fixes the network configuration issue. Also the PR #1990 addresses this with a global configuration option. However this has been stale for a bit.

Maybe the /info or /version calls could be scraped to detect the server is actually Podman instead.

MadsRC · 2024-05-04T20:01:37Z

I ran into this issue on a setup similar to the original poster.

@godart, I tried your fix, but I kept running into this issue:

2024/05/04 21:42:45 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 5.0.1
  API Version: 1.41
  Operating System: fedora
  Total Memory: 3622 MB
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: f9ea1ec477217d1ab405cd38e21e3a471f5360d6b852712ecd8c286ba9a5f9f2
  Test ProcessID: 9043d990-9cb6-40d3-adf7-ba8d0381a833
2024/05/04 21:42:45 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/05/04 21:42:45 ✅ Container created: 2b3afd5d2e60
2024/05/04 21:42:45 🐳 Starting container: 2b3afd5d2e60
2024/05/04 21:42:45 ✅ Container started: 2b3afd5d2e60
2024/05/04 21:42:45 🚧 Waiting for container id 2b3afd5d2e60 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/05/04 21:42:45 container logs (unexpected container status "stopped"):
2024/05/04 19:42:45 Pinging Docker...
panic: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

I even tried setting my DOCKER_HOST to point to my podman socket with DOCKER_HOST=unix:///Users/user/.local/share/containers/podman/machine/podman.sock which got rid of the permissions issue but still yielded an error:

2024/05/04 21:56:15 github.com/testcontainers/testcontainers-go - Connected to docker:
  Server Version: 5.0.1
  API Version: 1.41
  Operating System: fedora
  Total Memory: 3622 MB
  Resolved Docker Host: unix:///Users/user/.local/share/containers/podman/machine/podman.sock
  Resolved Docker Socket Path: /Users/user/.local/share/containers/podman/machine/podman.sock
  Test SessionID: 64a4a58cee7888739aebd93541d7a4b0b0200d84a9b10f396a5c4148984f1601
  Test ProcessID: 33aa8c36-5cdb-4b96-8725-b9539ef9c0d3
2024/05/04 21:56:15 🐳 Creating container for image testcontainers/ryuk:0.7.0
--- FAIL: TestWithRedis (0.07s)
    startup_test.go:23: Could not start redis: Error response from daemon: container create: statfs /Users/user/.local/share/containers/podman/machine/podman.sock: operation not supported: creating reaper failed: failed to create container

Did you not run into this issue?

mohsanabbas · 2024-05-09T19:24:17Z

We are having similar issues with colima and rancher-desktop

    2024/05/09 11:11:09 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 24.0.9
  API Version: 1.43
  Operating System: Ubuntu 23.10
  Total Memory: 1895 MB
  Resolved Docker Host: unix:///Users/mohsan.abbas/.colima/default/docker.sock
  Resolved Docker Socket Path: /Users/mohsan.abbas/.colima/default/docker.sock
  Test SessionID: c1729139638928699b678e3fdc7dfff4e56482cc2310ee2358a09f68931641e0
  Test ProcessID: e4f04ff7-5395-48a9-be14-51c54e9af8df
2024/05/09 11:11:09 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/05/09 11:11:09 ✅ Container created: 114d4680aa73
2024/05/09 11:11:09 🐳 Starting container: 114d4680aa73
2024/05/09 11:11:09 ✅ Container started: 114d4680aa73
2024/05/09 11:11:09 🚧 Waiting for container id 114d4680aa73 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/05/09 11:11:09 failed accessing container logs: Error response from daemon: No such container: 114d4680aa73b82030172ccfe23cb81d4a7364beb7217e6fb2472cd716b0d055
--- FAIL: TestKafkaIntegration (0.57s)
    kgo_test.go:182: Failed to start Kafka container: Error response from daemon: No such container: 114d4680aa73b82030172ccfe23cb81d4a7364beb7217e6fb2472cd716b0d055: could not start container: creating reaper failed: failed to create container
FAIL

nicanor-romero · 2024-05-24T16:14:30Z

Same here, tested with colima and rancher-desktop and ryuk fails to start (it exits as soon as it is started). Tested same code with Docker Desktop and it works OK.

It fails 95% of the times in my case.

Using testcontainers-go v0.31.0

2024/05/24 18:03:14 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 24.0.7
  API Version: 1.43
  Operating System: Alpine Linux v3.19
  Total Memory: 3917 MB
  Resolved Docker Host: unix:////Users/nicanor.romero/.rd/docker.sock
  Resolved Docker Socket Path: //Users/nicanor.romero/.rd/docker.sock
  Test SessionID: b20e559a2d29b918a349d7787d3b028fed79a1dae056e79c34e679e5e8532320
  Test ProcessID: 7cb791c0-9fce-4843-9459-877e9a87f0a7
2024/05/24 18:03:14 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/05/24 18:03:14 ✅ Container created: 9784e0df56ac
2024/05/24 18:03:14 🐳 Starting container: 9784e0df56ac
2024/05/24 18:03:14 ✅ Container started: 9784e0df56ac
2024/05/24 18:03:14 🚧 Waiting for container id 9784e0df56ac image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/05/24 18:03:14 failed accessing container logs: Error response from daemon: No such container: 9784e0df56ace89ed75bf6ff055fdef26c849382c5a69525f95a879f43ecd854
2024/05/24 18:03:14 Could not start redis: Error response from daemon: No such container: 9784e0df56ace89ed75bf6ff055fdef26c849382c5a69525f95a879f43ecd854: could not start container: creating reaper failed: failed to create container

I tried changing the version of ryuk but got the same results.

I debugged the process and found it fails here, between the CreateContainer and the Start.

Tested with the Quickstart example

nicanor-romero · 2024-05-25T12:07:40Z

I finally tracked my error to something between v0.13.0 (OK) and v0.14.0. Debugging how ryuk containers where created I saw this diff among others:

"Mounts": [
    {
        "Type": "bind",
-       "Source": "/var/run/docker.sock",
+       "Source": "/Users/nicanor.romero/.colima/docker.sock",
        "Destination": "/var/run/docker.sock",
        "Mode": "",
        "RW": true,
        "Propagation": "rprivate"
    }
],

I found 2 fixes (I prefer the first one):

Setting TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE env variable to /var/run/docker.sock. I previously tried setting it to colima's docker.sock and it still failed. And keep your DOCKER_HOST env variable as unix:////Users/$USER/.colima/docker.sock with your particular user.
Linking the /var/run/docker.sock to colima's docker.sock and removing the DOCKER_HOST env variable that was pointing to colima.

sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock

mdelapenya · 2024-08-21T11:40:42Z

I found 2 fixes (I prefer the first one):

Setting TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE env variable to /var/run/docker.sock. I previously tried setting it to colima's docker.sock and it still failed. And keep your DOCKER_HOST env variable as unix:////Users/$USER/.colima/docker.sock with your particular user.

Linking the /var/run/docker.sock to colima's docker.sock and removing the DOCKER_HOST env variable that was pointing to colima.
sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock

@nicanor-romero :

Option 1 is documented here https://golang.testcontainers.org/features/configuration/#docker-socket-path-detection, as part of the discovery mechanism.

And Option 2 is documented here: https://golang.testcontainers.org/system_requirements/using_colima/

@dfioravanti if you agree, I think we can close this one, but please keep it open if you see it's not resolved with the recent versions of the project.

Thanks!

dfioravanti · 2024-10-22T14:03:48Z

@mdelapenya

@dfioravanti if you agree, I think we can close this one, but please keep it open if you see it's not resolved with the recent versions of the project.

I tired again with 0.34 and it is still broken. I tired both 1 and 2 and it does not fix the problem. I still get

failed to start container: generic container: create container: reaper: new reaper: run container: container create: Error response from daemon: container create: unable to find network with name or ID bridge: network not found

mdelapenya · 2024-10-22T14:20:14Z

@dfioravanti in your tests, are you setting the Podman provider explicitly? This is required at the moment. With the work in #2810 that would be way simpler thanks to the support for docker contexts. But we are not there yet

mdelapenya · 2024-10-22T15:34:38Z

@dfioravanti I'm sorry, I responded very quick. It's Ryuk, not your own container the one you are interested in. I suggest you test it with that PR/branch, and verify it works as expected.

dfioravanti added the bug An issue with the library label Feb 23, 2024

bojand mentioned this issue Mar 13, 2024

redpanda: set entrypoint to the custom entrypoint file #2347

Merged

godart mentioned this issue Apr 18, 2024

feat: allow explicitly specifying provider type #1990

Closed

p-jahn mentioned this issue Apr 20, 2024

[Enhancement]: Use default docker host detection chain to auto-detect Podman #2503

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Ryuk does not work with podman #2264

[Bug]: Ryuk does not work with podman #2264

dfioravanti commented Feb 23, 2024 •

edited

Loading

godart commented Apr 17, 2024

godart commented Apr 17, 2024

MadsRC commented May 4, 2024

mohsanabbas commented May 9, 2024

nicanor-romero commented May 24, 2024 •

edited

Loading

nicanor-romero commented May 25, 2024

mdelapenya commented Aug 21, 2024

dfioravanti commented Oct 22, 2024

mdelapenya commented Oct 22, 2024

mdelapenya commented Oct 22, 2024 •

edited

Loading

[Bug]: Ryuk does not work with podman #2264

[Bug]: Ryuk does not work with podman #2264

Comments

dfioravanti commented Feb 23, 2024 • edited Loading

Testcontainers version

Using the latest Testcontainers version?

Host OS

Host arch

Go version

Docker version

Docker info

What happened?

Relevant log output

Additional information

godart commented Apr 17, 2024

godart commented Apr 17, 2024

MadsRC commented May 4, 2024

mohsanabbas commented May 9, 2024

nicanor-romero commented May 24, 2024 • edited Loading

nicanor-romero commented May 25, 2024

mdelapenya commented Aug 21, 2024

dfioravanti commented Oct 22, 2024

mdelapenya commented Oct 22, 2024

mdelapenya commented Oct 22, 2024 • edited Loading

dfioravanti commented Feb 23, 2024 •

edited

Loading

nicanor-romero commented May 24, 2024 •

edited

Loading

mdelapenya commented Oct 22, 2024 •

edited

Loading