ResourceReaperException: Initialization has been cancelled #449
Comments
|
Hi @HofmeisterAn, the image was not updated since December. To debug this issue, it would be nice if we could log some information about the current environment.
A possible cause of this issue might be an occasionally slow download of the ryuk image. |
… function: ResourceReaper'
{Add diagnostics information to ResourceReaperException.}
…ourceReaper'
{Add diagnostic information to ResourceReaperException.}
|
@HofmeisterAn did you get new reports regarding this issue recently? |
|
No, not lately. Looks stable for my builds, but this is an interesting comment: #443 (reply in thread). |
|
Here we go: 2252. |
|
@HofmeisterAn on first sight it seems odd that there was only a single connection attempt. The resource reaper session id looks wrong, too: |
|
Yes, I was wondering about the connection attempt as well. I didn't notice the label. The container name looks fine:
|
|
The session ID ( |
|
I dont know if this helps.. but the same things was happening to me, trying to use a mariadb database (Logs just like yours.) Interesting.. while debugging it was working fine, and failing only on some computers. What seems to fix it was adding ".WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(3306))"
|
|
@kron0s19 Do you still have the logs? The resource reaper exception incl. its diagnostic information might be helpful. |
|
error.txt I can confirm it has not happened again after adding "WithWaitStrategy" |
|
Interesting detail about the new error log: {
[...]
"ExpectedHost": "localhost",
"ExpectedPort": 49153,
"ContainerInspection": {
[...]
"NetworkSettings": {
[...]
"Ports": {
"8080/tcp": [
{
"HostIP": "0.0.0.0",
"HostPort": "63127"
}
]
},
[...]
}
}
}The expected port does not match with the actual host port binding of the ryuk container. This is different than in the previous error log we have. Edit: Another detail which might be important is that the host has IPv6 support (also loopback), but there is only a IPv4 binding for that port on the host. So a connection to |
Make sense 👍 |
|
Same issue here on Windows. Sometimes it works, sometimes it fails with this exception docker ps At first look, it doesn't seem related to reserved ports as commented here #443 (reply in thread) |
|
I wrote a quick "test" to make it easily reproductible public class ResourceReaperTest
{
private readonly ITestOutputHelper _testOutputHelper;
public ResourceReaperTest(ITestOutputHelper testOutputHelper)
{
_testOutputHelper = testOutputHelper;
}
[Fact]
public async Task ResourceReaperStressTest()
{
var expectedPortRegex = new Regex("\"ExpectedPort\": (\\d+)");
var hostPortRegex = new Regex("\"HostPort\": \"(\\d+)\"");
var i = 0;
while (i++ < 100)
{
try
{
await using var reaper = await ResourceReaper.GetAndStartNewAsync(Guid.NewGuid(), initTimeout: TimeSpan.FromSeconds(2));
_testOutputHelper.WriteLine($"{i} : success");
}
catch (ResourceReaperException ex)
{
try
{
var message = ex.Message;
var expectedPort = int.Parse(expectedPortRegex.Match(message).Groups[1].Value,
CultureInfo.InvariantCulture);
var hostPort = int.Parse(hostPortRegex.Matches(message)[1].Groups[1].Value,
CultureInfo.InvariantCulture);
_testOutputHelper.WriteLine($"{i} : fail - expectedPort: {expectedPort}, hostPort: {hostPort}");
}
catch (Exception)
{
//NetworkSettings HostPort is missing usually
_testOutputHelper.WriteLine($"{i} : fail");
}
}
}
}
}As you can see, it fails a lot!
|
That's nice - thanks. That should help to fix the issue finally. The ports don't match somehow. |
|
Also, I checked on a Linux computer. I have 100% success rate. |
100% success that it fails 😂? I tried something similar in the past. I was not able to break it on my machine... Even now. I attached the log file (xUnit.log) from your test (Docker Desktop 4.9.0 (80466)). Can you maybe add some more logging to (or debug into it): It would be interesting how the |
|
Your proposition here is a good idea #443 (reply in thread) Or maybe a way to define it for the default reaper. For example in Also, I tested this naive approach and it works well in my case (but may probably not be suitable for everyone) I replaced by private static int GetFreePort()
{
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
listener.Stop();
return port;
} |
|
It would be also interesting if something like this improves it: diff --git a/src/DotNet.Testcontainers/Containers/ResourceReaper.cs b/src/DotNet.Testcontainers/Containers/ResourceReaper.cs
index a36bf88..31e3181 100644
--- a/src/DotNet.Testcontainers/Containers/ResourceReaper.cs
+++ b/src/DotNet.Testcontainers/Containers/ResourceReaper.cs
@@ -48,6 +48,7 @@ namespace DotNet.Testcontainers.Containers
.WithExposedPort(RyukPort)
.WithPortBinding(RyukPort, true)
.WithBindMount("/var/run/docker.sock", "/var/run/docker.sock", AccessMode.ReadOnly)
+ .WithStartupCallback((resourceReaper, ct) => Task.Delay(TimeSpan.FromSeconds(5), ct))
.Build();
this.SessionId = sessionId;Maybe the container field (value) is not up-to-date, and we return a value that's obsolete in the meantime. You need to remove the timeout in your test then.
I prefer to find the root cause. The resource reaper must run reliable. |
|
I will try to add the callback and check if it improves And for the container state: This one was working this one not What I have seen is that when the PublicPort is in the 60K range, it works. When it is in the 40K range it fails. (and |
|
You are right! :) |
|
As soon as I finished the left tasks regarding the repository transfer, I can add a fix. Even, I'm not sure what the fix will be 😀. |
|
Great! Let me know if I can help. |
|
Refreshing TestcontainersContainer.container before getting the mapped port, may also work. diff --git a/src/Testcontainers/Containers/TestcontainersContainer.cs b/src/Testcontainers/Containers/TestcontainersContainer.cs
--- src/Testcontainers/Containers/TestcontainersContainer.cs
+++ src/Testcontainers/Containers/TestcontainersContainer.cs
@@ -145,8 +145,11 @@
public ushort GetMappedPublicPort(string privatePort)
{
this.ThrowIfContainerHasNotBeenCreated();
+ this.container = await this.client.GetContainer(this.Id);
+ .ConfigureAwait(false);
+
var mappedPort = this.container.Ports.FirstOrDefault(port => Convert.ToString(port.PrivatePort, CultureInfo.InvariantCulture).Equals(privatePort, StringComparison.Ordinal));
if (mappedPort != null)
{
|
Yes, that will work too. It looks like that the state after the wait strategy is done is not final ( Probably it's already enough to assign it afterwards (or return |
|
It's included in |
|
@HofmeisterAn The problem remains on |
|
Can you share the test (log)? |
|
|
This time the log file looks good. The resource reaper gets the right port ( var waitOS = EnvironmentTools.IsWindows()
? Wait.ForWindowsContainer()
: Wait.ForUnixContainer();Doesn't make sense. Even if you're running on Windows, it's still a Linux container. It should be Maybe you can figure out with Configure the logger and set the level to debug. That should help to identify the issue. |
You are totally correct. Unfortunately, it does not solve the issue 😉
I will try to look at it this week. |
|
For all of my colleagues, everything is working fine. It only fails in my environment. I will look at it once I have a little more time. You can assign this issue to me if you want. However, it may be possible to fix as I do not have any problems using |
|
I tested with the Develop branch. It seems it fixes the issue with port mismatches. But now I have the issue with the reserved ports in Windows 😢. Note: I updated to Docker Desktop 4.9.1 (81317), and performed a factory reset In Powershell : As you can see, the chosen port is within the first excluded range. This is an issue with Docker for Windows and you probably cannot do anything. @pellared If you still have failures, maybe check the port exclusion as well |
|
Thanks for the feedback. At least we were able to fix the mismatching ports. I think we can add something like this as a workaround? I hope this won't produce race-conditions. @srollinet , you already tried something like this, right? diff --git a/src/Testcontainers/Containers/ResourceReaper.cs b/src/Testcontainers/Containers/ResourceReaper.cs
index 48b0da5..27baae6 100644
--- a/src/Testcontainers/Containers/ResourceReaper.cs
+++ b/src/Testcontainers/Containers/ResourceReaper.cs
@@ -2,6 +2,7 @@ namespace DotNet.Testcontainers.Containers
{
using System;
using System.IO;
+ using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
@@ -44,7 +45,7 @@ namespace DotNet.Testcontainers.Containers
.WithAutoRemove(true)
.WithCleanUp(false)
.WithExposedPort(RyukPort)
- .WithPortBinding(RyukPort, true)
+ .WithPortBinding(RyukPort, GetNextTcpPort())
.WithBindMount("/var/run/docker.sock", "/var/run/docker.sock", AccessMode.ReadOnly)
.Build();
@@ -378,5 +379,14 @@ namespace DotNet.Testcontainers.Containers
ryukInitializedTaskSource.SetException(new ResourceReaperException("Initialization failed.", this.diagnostics));
}
}
+
+ private static ushort GetNextTcpPort()
+ {
+ using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
+ {
+ socket.Bind(new IPEndPoint(IPAddress.Loopback, 0));
+ return (ushort)((IPEndPoint)socket.LocalEndPoint).Port;
+ }
+ }
}
} |
|
@HofmeisterAn Yes, this is the method I used until now with https://github.com/isen-ng/testcontainers-dotnet, and I also tested it with your lib. But you have to inverse the parameters order 😉 Maybe you could add something to opt in/out this workaround? or to use it only on Windows? |
|
Even sequentially, it fails sometimes, but not very frequently... But in practice I never had any problem with this method, on multiple projects, whereas I had frequent failures with the port exclusion issue and the mismatching ports. Unfortunately, I think there is no magic solution that will work 100% for everyone |
|
Published in |
Using
@srollinet If it occurs very problematic, you can consider making some thread-safe dictionary to "reserve" the ports returned by and "release" them once the docker container is disposed. Of course, it may still race with any other application (and even code in the same process) that will try to bind a new TCP port. |
I was thinking about a similar idea. You could return a port from your own excluded port range, but dunno if you run into the same issue again if you explicit assign an excluded port 😅. |
It could. Therefore, it would be better if the caller just "retry" if |
Did you try it? I prefer fixing the root cause instead of adding another workaround for Docker Desktop for Windows. |
|
I do not say "explicitly assign an excluded port" but some other thread may already have concurrently reserved the same port in parallel. See:
You are just trying to get a free port but someone can already take it in the meanwhile. For instance we are using similar approach in our integration tests and it is possible to have a race if they would be run in parallel.
Still, I do not say we should do any more workarounds in |
@pellared It was just a stress test to make the problem appear quickly. So I don't have real issues about that. But thanks for the tip 😉 |
Yes, it's not just the tests that may run into a race condition. Other processes can use the port in the meantime too.
I'll ask. Let's see how others deal with it. |
|
Not sure if this helps but I also encountered this problem. I investigated and found that I had the following, potential problems:
I solved the problem by:
HTH! |
Describe the bug
It looks like, that in some rare cases .NET Testcontainers can't connect to Ryuk. I got a few times the following error:
According the log messages, .NET Testcontainers tries to connect a couple of times. It looks like Ryuk is not ready to accept a connection. @PSanetra Do you have any idea? May this relate to the recent changes in the image?
To Reproduce
-
Expected behavior
.NET Testcontainers can connect to Ryuk.
Screenshots
-
Desktop (please complete the following information):
Additional context
Build 2193. May relate to: #423, #441, #443.
Workaround
Disable the ResourceReaper (
TestcontainersSettings.ResourceReaperEnabled = false).The text was updated successfully, but these errors were encountered: