Bitbucket pipelines throws an error - authorization denied by plugin pipelines: --mounts is not allowed

[FaredoonIrani]

Hi,

Getting this in bitbucket pipelines:

Docker.DotNet.DockerApiException : Docker API responded with status code=Forbidden, response={"message":"authorization denied by plugin pipelines: --mounts is not allowed"}

I have tried your suggestion with WithDockerEndpoint() in a related issue, but it doesn't fix this.

The java testcontainers package seems to have addressed this by providing TESTCONTAINERS_RYUK_DISABLED:

testcontainers/testcontainers-java#700

[HofmeisterAn]

If your Docker host can't access the domain socket (or IPC socket) /var/run/docker.sock, disable the Resource Reaper (TestcontainersSettings.ResourceReaperEnabled = false) before building any container configuration.

WithDockerEndpoint isn't necessary anymore. The latest version will read the env variable automatic.

[FaredoonIrani]

Thanks @HofmeisterAn . This worked in bitbucket pipelines.

But, seems to fail in Jenkins with this:

System.Net.Sockets.SocketException : Cannot assign requested address

Could this be a case for using WithNetwork() and maintaining the same network in Jenkins?

[FaredoonIrani]

On the above: it's not specific to Jenkins. This is the same error if I run via Docker locally.

[HofmeisterAn]

I need more information. Can you add your builder configuration and at least the full stack trace?

[FaredoonIrani]

Yup, sure.

Here's the config code:

TestcontainersSettings.ResourceReaperEnabled = false;
var builder = new TestcontainersBuilder<MsSqlTestcontainer>()
.WithDatabase(new DbContainerConfiguration(_initialDatabaseName, _databasePassword))
.WithNetwork(new TestcontainersNetworkBuilder().Build())
.WithName($"sql-db_{Guid.NewGuid()}");

Container = builder.Build();
await Container.StartAsync();

The class DbContainerConfiguration is inherited from TestcontainerDatabaseConfiguration with overrides for password, username and ACCEPT_EULA.

For the full stacktrace, I suppose I will need to set a LogLevel for Serilog somewhere?

[FaredoonIrani]

Anyway, the console output I have from docker is this (which may not be super helpful):

SearchMicroservice-integration-test | Starting test execution, please wait...
MyMicroservice-integration-test | A total of 1 test files matched the specified pattern.
MyMicroservice-integration-test | TearDown failed for test fixture MyMicroservice.Application.IntegrationTests.Persistence.DbSourceTests
MyMicroservice-integration-test | System.Net.Http.HttpRequestException : Connection failed
MyMicroservice-integration-test | ----> System.Net.Sockets.SocketException : Cannot assign requested address
MyMicroservice-integration-test | TearDown : System.InvalidOperationException : Testcontainer has not been created.
MyMicroservice-integration-test | Failed EnsureRecordsAreFetchedFromDb [236 ms]
MyMicroservice-integration-test | Error Message:
MyMicroservice-integration-test | OneTimeSetUp: System.Net.Http.HttpRequestException : Connection failed
MyMicroservice-integration-test | ----> System.Net.Sockets.SocketException : Cannot assign requested address
MyMicroservice-integration-test |
MyMicroservice-integration-test | Results File: /app/inttestout/MyMicroservice.Application.IntegrationTests.xml
MyMicroservice-integration-test |
MyMicroservice-integration-test | Failed! - Failed: 1, Passed: 0, Skipped: 0, Total: 1, Duration: 236 ms - /app/tests/Application.IntegrationTests/bin/Debug/net6.0/MyMicroservice.Application.IntegrationTests.dll (net6.0)

[HofmeisterAn]

You never create the network. You just build its configuration. Create it first, like:

testcontainers-dotnet/tests/Testcontainers.Tests/Fixtures/Containers/Unix/NetworkFixture.cs

Line 20 in 124f007

return this.Network.CreateAsync();

This examples uses a volume, but creating one is equal to a network: #438

[FaredoonIrani]

Yeah, apologies for that. I had recently added that. If that line is removed, the same issue pops up anyway.

[FaredoonIrani]

I suppose the problem here is that Testcontainers is running in a docker container within the outer docker container that Jenkins runs in. I could try your bridge network example.

[Xitric]

Hey, just ran into this as well yesterday. I wonder if it would make sense to support the TESTCONTAINERS_RYUK_DISABLED environment variable similarly to the Java library?

I know it is possible to change the setting in code, but for our use case I think it makes sense to enable Ryuk locally, and only disable it in our pipelines. Unless of course that could have adverse side effects with some clean up features of testcontainers.

Currently, our workaround involves a base test fixture that looks for the TESTCONTAINERS_RYUK_DISABLED environment variable and sets TestcontainersSettings.ResourceReaperEnabled accordingly.

[HofmeisterAn]

@FaredoonIrani It's looks like Testcontainers can't connect to the Docker endpoint. Where is your Docker endpoint? Is the Docker service running? I did two tests with a jenkins/jenkins:2.356 base image. One with a sibling container (mounting the domain socket) and another one with DinD. Both run fine.

Sibling container

env
_=/usr/bin/env
COPY_REFERENCE_FILE_LOG=/var/jenkins_home/copy_reference_file.log
DEBIAN_FRONTEND=noninteractive
HOME=/root
HOSTNAME=fc5248e6aa28
JAVA_HOME=/opt/java/openjdk
JENKINS_HOME=/var/jenkins_home
JENKINS_INCREMENTALS_REPO_MIRROR=https://repo.jenkins-ci.org/incrementals
JENKINS_SLAVE_AGENT_PORT=50000
JENKINS_UC_EXPERIMENTAL=https://updates.jenkins.io/experimental
JENKINS_UC=https://updates.jenkins.io
JENKINS_VERSION=2.356
LANG=C.UTF-8
PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/workdir
REF=/usr/share/jenkins/ref
SHLVL=1
TERM=xterm

df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/loop0        357768    357768         0 100% /usr/bin/docker
/dev/sdc       263174212  23147820 226588236  10% /var/jenkins_home
C:\            494465020 198242160 296222860  41% /workdir
none             6382388        16   6382372   1% /run/docker.sock
overlay        263174212  23147820 226588236  10% /
shm                65536         0     65536   0% /dev/shm
tmpfs              65536         0     65536   0% /dev
tmpfs            6382388         0   6382388   0% /proc/acpi
tmpfs            6382388         0   6382388   0% /sys/firmware
tmpfs            6382388         0   6382388   0% /sys/fs/cgroup

docker context ls
NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT   ORCHESTRATOR
default *           moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                         swarm

dotnet test --logger "console;verbosity=detailed" Testcontainers.Tests.csproj
  Determining projects to restore...
  Restored /workdir/Testcontainers.Tests.csproj (in 9.55 sec).
  Testcontainers.Tests -> /workdir/bin/Debug/net6.0/Testcontainers.Tests.dll
Test run for /workdir/bin/Debug/net6.0/Testcontainers.Tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
/workdir/bin/Debug/net6.0/Testcontainers.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET 6.0.6)
[xUnit.net 00:00:00.57]   Discovering: Testcontainers.Tests
[xUnit.net 00:00:00.61]   Discovered:  Testcontainers.Tests
[xUnit.net 00:00:00.61]   Starting:    Testcontainers.Tests
[xUnit.net 00:00:05.68]   Finished:    Testcontainers.Tests
    Passed Testcontainers.Tests.Sample.PingContainer [16 ms]
    Standard Output Messages:
  Docker container f56f77d80aaeb49fd398d83273413fd4996d33091358caa69208771feff9c485 created
  Start Docker container f56f77d80aaeb49fd398d83273413fd4996d33091358caa69208771feff9c485



Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 6.2538 Seconds

DinD

env
_=/usr/bin/env
COPY_REFERENCE_FILE_LOG=/var/jenkins_home/copy_reference_file.log
DEBIAN_FRONTEND=noninteractive
HOME=/root
HOSTNAME=e1f8e1058784
JAVA_HOME=/opt/java/openjdk
JENKINS_HOME=/var/jenkins_home
JENKINS_INCREMENTALS_REPO_MIRROR=https://repo.jenkins-ci.org/incrementals
JENKINS_SLAVE_AGENT_PORT=50000
JENKINS_UC_EXPERIMENTAL=https://updates.jenkins.io/experimental
JENKINS_UC=https://updates.jenkins.io
JENKINS_VERSION=2.356
LANG=C.UTF-8
PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/workdir
REF=/usr/share/jenkins/ref
SHLVL=1
TERM=xterm

df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sdc       263174212  23147816 226588240  10% /var/jenkins_home
C:\            494465020 198319868 296145152  41% /workdir
overlay        263174212  23147816 226588240  10% /
shm                65536         0     65536   0% /dev/shm
tmpfs              65536         0     65536   0% /dev
tmpfs            6382388         0   6382388   0% /sys/fs/cgroup

docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT   ORCHESTRATOR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                         swarm

dotnet test --logger "console;verbosity=detailed" Testcontainers.Tests.csproj
  Determining projects to restore...
  All projects are up-to-date for restore.
  Testcontainers.Tests -> /workdir/bin/Debug/net6.0/Testcontainers.Tests.dll
Test run for /workdir/bin/Debug/net6.0/Testcontainers.Tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
/workdir/bin/Debug/net6.0/Testcontainers.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET 6.0.6)
[xUnit.net 00:00:00.56]   Discovering: Testcontainers.Tests
[xUnit.net 00:00:00.59]   Discovered:  Testcontainers.Tests
[xUnit.net 00:00:00.59]   Starting:    Testcontainers.Tests
[xUnit.net 00:00:02.19]   Finished:    Testcontainers.Tests
    Passed Testcontainers.Tests.Sample.PingContainer [11 ms]
    Standard Output Messages:
  Docker container 559e50f5d1bc08ed721629e9c31490354e61cbacaf9305a412cd1ddddd0d84d4 created
  Start Docker container 559e50f5d1bc08ed721629e9c31490354e61cbacaf9305a412cd1ddddd0d84d4



Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 2.8233 Seconds

@Xitric

I wonder if it would make sense to support the TESTCONTAINERS_RYUK_DISABLED environment variable similarly to the Java library?

OC that make sense. With that, I'd like to support the .testcontainers.properties file as well. I just haven't had time yet.

I know it is possible to change the setting in code, but for our use case I think it makes sense to enable Ryuk locally, and only disable it in our pipelines. Unless of course that could have adverse side effects with some clean up features of testcontainers.

I would keep it enabled in the pipeline. Anyway, all you need is !bool.TryParse(Environment.GetEnvironmentVariable("TESTCONTAINERS_RYUK_DISABLED"), out var isRyukDisabled) || !isRyukDisabled 🙃.

[FaredoonIrani]

@HofmeisterAn Thanks for this. I will try to reply with some info soon.

[Xitric]

@HofmeisterAn It was my understanding, based on the issue reported here, that using Ryuk in BitBucket pipelines in not supported if it tries to mount the domain socket for Docker. After all, in BitBucket pipelines the docker daemon is accessible only via a TCP endpoint. That is why we have disabled Ryuk permanently in our pipelines on BitBucket. I just want to make sure that there is not a better alternative that we can try to pursue.

And yes, we are doing something along the lines of the code you posted, albeit not as a one-liner 😉

[FaredoonIrani]

@HofmeisterAn I don't know about the Docker host locally, let alone the actual Jenkins pipeline. If you have an idea about doing some console logs to get some info, I'll be glad to do that.

In the meantime, I tried to replicate my repo structure (see attached).

Ran a simple docker-compose up unit-test. Getting the exact same error immediately on my local machine.
TestingTestContainer.zip

[HofmeisterAn]

based on the issue reported here, that using Ryuk in BitBucket pipelines in not supported if it tries to mount the domain socket for Docker.

@Xitric I meant not in general, but yes, you're right. It won't work with Bitbucket. Sorry for the misconception.

---

@FaredoonIrani Your example can't work. Your Docker image or container does not contain Docker (access to a daemon). It's not mounted nor installed. That's what I'm mentioning above. Testcontainers can't connect to the Docker endpoint (probably it's missing in Jenkins too). Either mount the socket or add Docker to the image:

• Sibling container (WSL2): docker run -it -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):$(which docker) -v $(which com.docker.cli):$(which com.docker.cli) -v $(pwd):/workdir --rm mcr.microsoft.com/dotnet/sdk:6.0 /bin/bash
• CUSTOM_IMAGE that contains Docker: docker run -it -v $(pwd):/workdir --privileged --rm ${CUSTOM_IMAGE} /bin/bash

This is the Dockerfile that I'm using in the example above (WSL2):

FROM curlimages/curl:7.83.1 AS curl

RUN echo "Download dependencies..." \
  && curl https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -o /tmp/packages-microsoft-prod.deb \
  && curl https://get.docker.com -o /tmp/get-docker.sh

FROM jenkins/jenkins:2.356

# Don't do this. This is just a multi-stage test.
COPY --from=curl /tmp/* /tmp

USER root

ENV DEBIAN_FRONTEND=noninteractive

RUN echo "Install dependencies..." \
  && echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes \
  && dpkg -i /tmp/packages-microsoft-prod.deb \
  && apt update \
  && apt upgrade \
  && apt install --no-install-recommends \
    apt-transport-https \
    dotnet-sdk-6.0 \
  && sh /tmp/get-docker.sh \
  && touch /etc/fstab \
  && update-alternatives --set iptables /usr/sbin/iptables-legacy \
  && update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /workdir

ENTRYPOINT ["/bin/bash", "-c"]

CMD ["echo -e \"127.0.0.254\thost.docker.internal\" >> /etc/hosts && service docker start && dotnet test --logger \"console;verbosity=detailed\" Testcontainers.Tests.csproj"]

docker build --tag dind-example:0.1.0 .
docker run -it -v $(pwd):/workdir --privileged --rm dind-example:0.1.0

I hope this helps. Otherwise, you need to do some research and study how to set up Docker (especially inside a container).

[FaredoonIrani]

@HofmeisterAn Thanks very much. Yes, work needs to be done on my side - apologies for the display of dumbness here.
Thanks for the great work on the library. Closing this for now.

[HofmeisterAn]

No worries. It’s not trivial. You can also ask on Slack if you need help to set-up Docker.

[FaredoonIrani]

Thanks @HofmeisterAn.

As a followup, just FYI: the docker daemon location turns out to be /run/containerd/containerd.sock.

So, surprise - no docker. I used .WithDockerEndpoint(@"unix:///run/containerd/containerd.sock") and exposed it via docker-compose volume mapping, but no luck.

Thanks, as always.