Not working in Bitbucket pipeline: "authorization denied by plugin pipelines: Command not supported."

[4ntoine]

It seems to be both docker and testcontainers issue:

Running name.antonsmirnov.apptogether.dao.DeveloperDaoTest
08:08:56.040 [main] INFO  EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
08:08:56.060 [main] INFO  DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://localhost:2375
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://localhost:2375,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

08:08:56.066 [main] INFO  DockerClientFactory - Docker host IP address is localhost
08:08:56.227 [main] INFO  DockerClientFactory - Connected to docker: 
  Server Version: 17.12.0-ce
  API Version: 1.35
  Operating System: Alpine Linux v3.7 (containerized)
  Total Memory: 32171 MB
08:08:57.182 [testcontainers-netty-1-6] ERROR ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.DockerException: {"message":"authorization denied by plugin pipelines: Command not supported."}

	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:111) [testcontainers-1.7.2.jar:na]
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [testcontainers-1.7.2.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.211 sec <<< FAILURE!

Fabric docker plugin had the same issue and they were able to fix it with relatively small changes in plugin. See issues:

• https://bitbucket.org/site/master/issues/15844/bug-in-authorization-plugin#comment-45553072
• https://bitbucket.org/site/master/issues/15505/maven-fabric8-plugin-fails-to-run-docker

This is blocker issue and can be easily fixed.

[4ntoine]

testcontainers 1.7.2 used

[kiview]

Hey @4ntoine, thank for the feedback regarding Bitbucket pipeline.
I'm not sure yet it it's easy to fix, our aux container (Ryuk) needs to mount the docker socket and from a quick look at the linked discussion, this is forbidden in Bitbucket pipelines.

[nathanburrell]

Hey @kiview test-containers works in pipelines, the issue above (as I described here: https://bitbucket.org/site/master/issues/15844/bug-in-authorization-plugin#comment-45561539) is not an issue in the auth plugin interaction with the custom java docker client nor an issue with how the library talks to docker (its working fine over tcp) its in how the MySqlContainer class by default mounts the mysql.conf file.

Users will (in order for this to work on pipelines), need to reconfigure the default (hostPath) directory for this file as we limit what paths can be mounted into docker containers due to the layout of how the containers and filesystems interact, and than the library works fine :)

[kiview]

Hi @nathanburrell, thanks a lot for the clarification.
So how should we ideally tackle this? Change our MySqlContainer implementation, or mention it in the docs as a workaround for pipelines compatibility?

[4ntoine]

I believe paths should be adjusted to be in allowed by Bitbucket, see debug on local machine:

[kiview]

Hey @4ntoine, would you like to submit your fix as a PR? We can look into it together.
Whoops sorry, you've already submitted the PR in #704

[4ntoine]

sorry i did not have time to fix, but i was able to use https://github.com/palantir/docker-compose-rule instead (it works)

[dstepanov]

Can you provide a way to disable Ryuk?
I have found workaround:

public abstract class TestBase {
    static {
        try {
            DockerClientFactory.instance().client();
        } catch (DockerException e) {
            if (!e.getMessage().contains("BITBUCKET_CLONE_DIR")) {
                throw new IllegalStateException(e);
            }
            // Ignore exception related to reach outside of BITBUCKET_CLONE_DIR in ResourceReaper
        }
    }
}

It's ignoring first exception caused by Ryuk trying to connect to docker.sock. Is there a way to not use sock interactions?

[bsideup]

Hi @dstepanov,

DIsabling Ryuk doesn't fix the issue, rather delays it. Does it work if you disable the startup checks? See here how to do that: https://www.testcontainers.org/usage/properties.html

[dstepanov]

It doesn’t, you cannot mount docker.sock which is outside bitbucket directory. Bitbucket pipeline doesn’t need Ryuk anyway, it will do cleaning anyway. Is there a way at least to let Ryuk use tcp? 6. 8. 2018 v 16:23, Sergei Egorov <[email protected]>:

…

Hi @dstepanov, DIsabling Ryuk doesn't fix the issue, rather delays it. Does it work if you disable the startup checks? See here how to do that: https://www.testcontainers.org/usage/properties.html — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nathanburrell]

Hi,

Im a developer on bitbucket pipelines, since ryuk can only communicate with a docker daemon over its unix socket currently (and the fact that pipelines only allows communication with the steps docker daemon over TCP), we have since relaxed the mounts allowed in pipelines to allow mounting in the unix socket for the steps docker daemon, which now allows ryuk to run correctly :)

Kind Regards,
Nathan Burrell

[erikvanzijst]

now allows ryuk to run correctly :)

Note that the MySqlContainer probably still needs work if it uses a mount point for the mysql-default-conf file (the PostgreSQLContainer doesn't do this and already works correctly in Pipelines).

[bsideup]

@nathanburrell
That is... Amazing! 😍 Thanks for listening and relaxing it from your side, I'm sure there are other projects that will also benefit from it 👍

[bsideup]

@erikvanzijst We're working on mount-less file sharing (with copyFileToContainer with InputStream on container creation), it should fix the issue you mentioned. Will be released very soon :)

[4ntoine]

@nathanburrell Awesome, thanks!

[dstepanov]

@nathanburrell Great news! Thanks!

[dstepanov]

@nathanburrell As of today's morning versions testcontainers fail on Bitbucket pipelines with:

containers-ryuk] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at localhost:32768
    java.net.ConnectException: Connection refused (Connection refused)

[dstepanov]

@nathanburrell Older versions fails with old error:

    com.github.dockerjava.api.exception.DockerException: {"message":"authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"}

[bsideup]

@dstepanov is it with 1.9.1?

[dstepanov]

@bsideup 1.9.0 and 1.9.1

[nathanburrell]

Hi,

We rolled out a change yesterday to enable user namespace remapping on the docker daemons we provide to users for security reasons.

Thats an odd failure but, with this change I would expect to see

• network=host is no longer allowed (we have added customers to an exclusion list so that they wont have user namespace remapping yet, we plan on reaching out to them over the coming weeks to have them modify their builds to either use a custom network or simple port mappings, as there is no real need for network=host).
• permission issues with certain mounted files, for the repository we set it to already be fully open to cater for build containers using different uids to the clone and docker in docker containers, so I expect this to occur rarely.

Have you got a link to a reproducible test case showing this, or alternatively can you raise a support ticket at https://support.atlassian.com/contact, so I can investigate further.

Kind Regards,
Nathan Burrell

[dstepanov]

@nathanburrell Why wouldn't you include a simple testcontainers project to test your changes? Now all our builds are down because of this.

[nathanburrell]

I understand your frustration when we make changes like this and I apoligise, but unfortunately security fixes may sometimes break some builds for the good of all the users of pipelines.

For test containers we don't test it specifically as we have lots of users that use 1000s of different libraries from a broad range of different languages that perform similar functionality and we can't test all the things unfortunately.

I haven't had a chance to look into why test containers is failing yet due to other urgencies, and as I listed above I'm surprised a connection to local host is failing as I expect only the two cases above to fail.

For now I have added your account to the exclusion list to allow you to work, next time can you raise a support ticket with Atlassian directly so we can back to you in a more timely manner as I also don't monitor GitHub issues alot only in my spare time :)

Kind Regards
Nathan Burrell

[bsideup]

Hi @nathanburrell! Thanks for checking it at all 👍
Let me know if I can help anyhow to debug what happened, I hope that's not something big and we can solve it either on our or your side 🤞

[remmeier]

being able to disable ryuk would be great. For many cases this should be sufficient since the cloud provider will cleanup by itself. Even if it does not solve the problem completely, it would at least allow to work around it.

[remmeier]

is there an interest in a PR for such a property? For our use cases we don't really need ryuk.

[rnorth]

I've been trying to get up to speed on this issue, as seen here and in #712. Sorry for only recently looking into this. To summarise my understanding so far:

• User namespace remapping will always prevent access to the Docker daemon unix socket. Among other things that's part of the security benefits of the remapping.
• As such, even a workaround of making Ryuk (and other things) talk to a TCP socket for the docker daemon is moot - even if possible it would be sidestepping the security model. Is this accurate?
• Right now Bitbucket pipelines has an exception to allow Testcontainers to work, but this won't be for much longer.
• For other teams using namespace remapping there is no such exception, so we have a problem.
• Both of these cases are the same problem, it's just that we have a temporary reprieve for Bitbucket pipelines.

I'd like to think carefully about the options; disabling Ryuk might be one way of avoiding access from TC-spawned to the docker daemon. However, it's less desirable because it is not the only feature that will suffer (Docker Compose is another). Also, it creates yet another permutation for Testcontainers to operate in, one which lacks a lot of the safeguards for cleanup that we think are important. I don't feel comfortable yet with the implications of that.

Still, this is probably something that @kiview, @bsideup and I need to have a discussion about soon.

[nathanburrell]

Hey :)

Any updates on this, as we are planning on removing the exclusion list in the next month from pipelines.

Kind Regards,
Nathan Burrell

[raulgomis]

Hey @rnorth ,

Any updates on this? Have you guys discussed it internally (cc: @kiview @bsideup)?

Having an opt-out parameter to disable Ryuk would work for all CI/CD use cases (like Bitbucket Pipelines). At the moment we have two sets of customers impacted:

• Existing users using testcontainers: they will be impacted when we remove them from the exclusion list in February.
• New users: currently being impacted as they can't use testcontainers with Bitbucket Pipelines.

I would appreciate your help here, so that we reduce the impact to our customers and they can keep using testcontainers happily in Bitbucket Pipelines.

Cheers,
Raul

[rnorth]

Hi Raul, Nathan,
I’m very sorry for not replying sooner. Yes, we have had internal discussions and agree that disabling the Ryuk component via an environment variable is the best path forward.

@bsideup is working on it and we’d like to get this released very soon and communicated out to users.

Thank you for your help, and I apologise again.

Richard

_{Sent with GitHawk}

[kiview]

Hi @raulgomis,
we've discussed this together in our call yesterday and we decided to solve this with #1023.

This will be a quite non invasive hack/fix and most probably won't impact the Testcontainers UX on Bitbucket Pipelines negatively, since the containers should be removed by CI environment automatically after the build.

[raulgomis]

Great news!

Thanks a lot team 🙌, our users will love that you guys are planning to fix it.

Cheers,
Raul

[bsideup]

Hi @raulgomis & @nathanburrell,

I just pushed a PR:
#1181

If you set TESTCONTAINERS_RYUK_DISABLED environment variable to true, we will not start Ryuk:
https://5c4c37a5560ab600080292aa--testcontainers.netlify.com/features/configuration/#disabling-ryuk

Please try it with Jitpack (revision 03a5c368ae) if you have time:
https://www.testcontainers.org/jitpack_dependencies/

[schrieveslaach]

@bsideup, I tried to use your PR and now I can start containers with a docker setup providing container isolation with a user namespace mapping (see #712).

[bsideup]

@schrieveslaach thanks for trying!

FYI the userns parameter will be supported once we integrate #1007

[schrieveslaach]

@bsideup, thanks for pointing out this PR. I was going to ask you exactly that question. 😉

[rnorth]

We've now released this in Testcontainers 1.10.6!

Please upgrade as soon as you can and follow the steps here for disabling Ryuk: https://www.testcontainers.org/features/configuration/#disabling-ryuk

I'll close this ticket now, but please keep us posted if there are any new issues with this.

[modye]

Thanks a lot, this is working like a charm in Bitbucket 😄

[nathanburrell]

Thanks for actioning this quickly :)

We are organising some communicuations to customers in the coming weeks to tell them to update and set that variable, and I can also continue to start using test-containers more in the pipelines codebase itself ;)

[andreppires]

Hey, @nathanburrell is test-containers fully working on the bitbucket pipeline at this point? I'm, guessing that it is, but I'm having some problems using it with Cassandra. Basically, locally the tests passed but on bitbucket pipeline that seems to be a problem to communicate with the Cassandra container.

[czunker]

Hi,

Thanks a lot. The issues #700 and #1135 helped me, to get testcontainers running in Jenkins with dind-rootless inside Kubernetes.
In hope, this might help someone else, a stripped down example for a Jenkins build job pod definition:

apiVersion: v1
kind: Pod
spec:
  securityContext:
    fsGroup: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: dind
      image: docker:20.10-dind-rootless
      imagePullPolicy: Always
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      securityContext:
        # still needed: https://docs.docker.com/engine/security/rootless/#rootless-docker-in-docker
        privileged: true
        readOnlyRootFilesystem: false
    - name: gradle
      image: openjdk:11
      imagePullPolicy: Always
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
        # needed to get it working with dind-rootless
        # more details: https://www.testcontainers.org/features/configuration/#disabling-ryuk
        - name: TESTCONTAINERS_RYUK_DISABLED
          value: "true"
      command:
        - cat
      tty: true
      volumeMounts:
        - name: docker-auth-cfg
          mountPath: /home/ci/.docker
      securityContext:
        capabilities:
          drop:
            - ALL
        allowPrivilegeEscalation: false
        privileged: false
        readOnlyRootFilesystem: false
  volumes:
    - name: docker-auth-cfg
      secret:
        secretName: docker-auth

[Maciag312]

It seems upgrading test containers version to 1.16.0 solved the issue.