-
-
Notifications
You must be signed in to change notification settings - Fork 512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Endpoint() returns "port not found" error, even though wait.ForListeningPort is used #605
Comments
I am seeing this as well on 0.17.0 with mysql image, test sometimes fails, more often on CI. My configuration is the same as presented except I am using normal Docker and don't skip reaper. For context, the build starts and terminates a database container about three times, perhaps having multiple start/stop in the same binary run (in this case |
@anuraaga @awagner-iq thanks for open this issue, and sorry for the radio silence, I went on paternity leave at the beginning of December and probably skipped this ticket. I'm investigating why it fails for your use case although, with the above snippet, I'm not able to reproduce it yet. Debugging... |
I'm noticing this as well while running on Docker 20.10 on Linux - it's resulting in a fair bit of flakiness, especially in our CI pipeline. It doesn't look like this is strictly related to Podman. |
It seems to be a bug in the Endpoint function. I'm using docker on linux and I can reproduce this easily on my machine and in ci. Now I create the URL myself for a specific port (not the first one, as Endpoint does it) and I did not see this issue yet. |
Could you please add a repro snippet, including which version of the library are you using? 🙏 If there are so many users seeing it, we could be in front of a real bug that we should fix 🐞 |
Yes would be great if this one can go up for priority because it is still occurring , and it is breaking a lot of CI pipelines. github.com/testcontainers/testcontainers-go v0.19.0
|
Hi, I did not have the time yet to create a repro snippet. The failing code I have is something I cannot share. I switched all my What I'd do given some time:
I used a GenericContainer with mongo:4.0, and testcontainers 0.19.0, but also switched to the latest version for a quick test and that did not fix the issue. Maybe this already helps and gives someone the opportunity to reproduce and analyze this bug. |
I'm able to reproduce this exact error in a VM using podman. Will take a look after my parental leave |
In the mean time, checking the state of the Docker types, I saw this difference when inspecting a container with podman Vs docker: Steps to reproduce
func (c *DockerContainer) Ports(ctx context.Context) (nat.PortMap, error) {
inspect, err := c.inspectContainer(ctx)
if err != nil {
return nil, err
}
+ fmt.Printf(">>> network.settings.ports: %+v\n", inspect.NetworkSettings.Ports)
+ fmt.Printf(">>> config.exposed.ports: %+v\n", inspect.Config.ExposedPorts)
return inspect.NetworkSettings.Ports, nil
}
go run gotest.tools/gotestsum --format short-verbose --packages="./..." -- -run "^TestContainerWithHostNetworkOptions_UseExposePortsFromImageConfigs" -timeout 600s -count=1 -v With Podman (Ubuntu VM)
With Docker (on Mac)
So for some reason, the docker types in Podman are inconsistent according Docker 🤷 . Pinging @kiview @cristianrgreco @eddumelendez @HofmeisterAn for awareness while I'm out |
Could be related to containers/podman#17780 🤔 |
I'm seeing an intermittent
and a wait strategy that passes:
If folks have any workarounds or tips that would be helpful. EDIT: After investigating this issue further, it looks like |
Retrying func getMappedPorts(containerID string) (nat.PortMap, error) {
cmd := exec.Command("docker", "inspect", "--format", "{{json .NetworkSettings.Ports}}", containerID)
output, err := cmd.CombinedOutput()
if err != nil {
return nil, fmt.Errorf("failed to execute docker inspect: %s, %v", output, err)
}
var ports nat.PortMap
if err = json.Unmarshal(output, &ports); err != nil {
return nil, fmt.Errorf("failed to parse output: %v", err)
}
return ports, nil
} |
This issue started occurring for me only after installing However adding a manual delay and watching that the port is open seems to resolve this. I assume there is a timing issue here... This occurs on Docker as well as podman |
Also seeing this error with the
adding
helps resolve the issue. This also seems to have helped
|
Some fixes around container inspect and available ports were added in v0.32.0 and v0.33.0. Could you please check if the issue described here is gone? Thanks! |
Testcontainers version
0.15.0
Using the latest Testcontainers version?
Yes
Host OS
Linux
Host Arch
x86_64
Go Version
1.19
Docker version
mero@dibbler ~$ podman version Version: 3.4.7 API Version: 3.4.7 Go Version: go1.16.15 Built: Thu Apr 21 19:38:09 2022 OS/Arch: linux/amd64
Docker info
What happened?
We are seeing test flakes with the error message
mysql.Endpoint() = port not found
. The code we use to create the container is attached below. In particular, we use bothwait.ForListeningPort("3306/tcp")
andwait.ForLog("port: 3306")
to make sure we wait until the container is running and available. However, it seems even with that,GenericContainer
will occasionally return without being ready, as evidenced by the fact thatEndpoint
returns an error about the port not being found.Relevant log output
No response
Additional Information
The text was updated successfully, but these errors were encountered: