Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Failing with "protocol not available, failed to create Docker provider" on Windows #1249

Closed
danielorbach opened this issue May 28, 2023 · 10 comments · Fixed by #1294
Closed
Labels
bug An issue with the library

Comments

@danielorbach
Copy link
Contributor

danielorbach commented May 28, 2023

Testcontainers version

0.20.1

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host arch

x86

Go version

1.20

Docker version

Client:
 Cloud integration: v1.0.31
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:20:14 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.19.0 (106363)
 Engine:
  Version:          23.0.5
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       94d3ad6
  Built:            Wed Apr 26 16:17:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.10.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 167
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.48GiB
 Name: docker-desktop
 ID: 833c289d-cefa-4019-a820-b1b496cc06f6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

I upgraded from v0.19.0 using go get -u and got the latest v0.20.1.

To my surprise running the same tests that had just passed now fails consistently with the message:

protocol not available, failed to create Docker provider

Looking deeper into the changelog of this release I see the error comes from sockets.ConfigureTransport(transport, c.proto, c.addr) at github.com/docker/[email protected]+incompatible/client/options.go:71.

This option function (client.WithHost) is not called by the previous version because of the introduction of ReadConfig().

Previously, this option is only appended by NewDockerClient() only if the user-environment contains a .testcontainers.properties file that explicitly sets the docker host.

Nowadays, this option is always appended by NewDockerClient() because ReadConfig() always returns constant host, even when not explicitly set by the user-environment.

Relevant log output

I've summarized the difference between the two tags (ignoring comments and whitespaces):

1,3c1,2
< func NewDockerClient() (cli *client.Client, host string, tcConfig TestContainersConfig, err error) {
< 	tcConfig = configureTC()
< 	host = tcConfig.Host
---
> func NewDockerClient() (cli *client.Client, err error) {
> 	tcConfig := ReadConfig()
6,7c5,6
< 	if host != "" {
< 		opts = append(opts, client.WithHost(host))
---
> 	if tcConfig.Host != "" {
> 		opts = append(opts, client.WithHost(tcConfig.Host))
17,20d15
< 	} else if dockerHostEnv := os.Getenv("DOCKER_HOST"); dockerHostEnv != "" {
< 		host = dockerHostEnv
< 	} else {
< 		host = "unix:///var/run/docker.sock"

Additional information

Eventually, on Windows without any custom configuration, v0.19.0 used to access the host npipe:////./pipe/docker_engine.
Without the client.WithHost() option, the DefaultDockerHost is used correctly for Windows and never overridden.

@danielorbach danielorbach added the bug An issue with the library label May 28, 2023
@danielorbach
Copy link
Contributor Author

@mdelapenya I see the breaking change had been introduced with #941.
I am happy to contribute a fix, could you hint me towards the appropriate resolution?

I fear I don't understand the big picture entirely; I see from the commit message that the break in API was intentional:

break: do not return the Docker Host when retrieving a Docker client (e754c24)
fix: default for docker host (560672a)

@mdelapenya
Copy link
Member

@danielorbach indeed, we had to break that method signature in order to reach to a cleaner state. In #1161 we are doing great progress towards having a consistent strategy to bootstrap the Docker client, discovering the Docker host and docker socket path in a single and deterministic manner. Hopefully it will resolve inconsistencies regarding docker alternatives and different setups in the environment/configuration.

Could you give it a try? It was merged past Friday, so I think you could use that commit in your go.mod file

@danielorbach
Copy link
Contributor Author

danielorbach commented May 29, 2023

Thank you for the quick reply!

Could you give it a try? It was merged past Friday, so I think you could use that commit in your go.mod file

Tests still fail with the same error as before 😢

I've taken the commit from the pull request:

$ go get github.com/testcontainers/testcontainers-go@08a56c9c2999d58db13dacd9b7e445acf3272fdc
go: downloading github.com/testcontainers/testcontainers-go v0.20.2-0.20230526123138-08a56c9c2999
go: downloading golang.org/x/exp v0.0.0-20230510235704-dd950f8aeaea
go: upgraded github.com/imdario/mergo v0.3.13 => v0.3.15
go: upgraded github.com/testcontainers/testcontainers-go v0.20.1 => v0.20.2-0.20230526123138-08a56c9c2999
go: upgraded golang.org/x/exp v0.0.0-20230124195608-d38c7dcee874 => v0.0.0-20230510235704-dd950f8aeaea
-       github.com/testcontainers/testcontainers-go v0.19.0
+       github.com/testcontainers/testcontainers-go v0.20.2-0.20230526123138-08a56c9c2999

@sebastianbuechler
Copy link
Contributor

Can reproduce on Windows 11.

If I add tcp://localhost:2375 as a system env for the name DOCKER_HOST the setup works again.

I think the protocol part (unix) of the default value for docker host unix:///var/run/docker.sock is not recognized as valid when the config is read and thus the exception is thrown.

@meysamhadeli
Copy link

I still this error protocol not available in version v0.20.1.
Do you have any update for fixing this problem?

@danielorbach
Copy link
Contributor Author

@mdelapenya I tried the latest main this morning to no avail.
Looking at the latest PRs, I see you guys have explicitly commented to ignore Windows containers and default to the unix socket when no other docker-host has been provided by the environment.

Would you like me to propose a pull-request addressing this?
The latest refactor seems to collate code-paths that need a docker configuration, I think it can be as small as a few lines of code.

@mdelapenya
Copy link
Member

Hi @danielorbach I'm back from a week on PTO, so I'm basically catching up today.

Indeed, but IIRC we were ignoring windows containers, not windows usage... although I could probably have committed a mistake on it. I'm lacking a testing windows instance to perform consistent and repeatable tests for Windows at the moment, but this is something we are working on fixing it asap.

Would you like me to propose a pull-request addressing this?

That would be super!

The latest refactor seems to collate code-paths that need a docker configuration, I think it can be as small as a few lines of code.

Indeed, we tried to have the docker socket resolution as simple as possible, so you could probably figure it out in a very easy manner

@sebastianbuechler
Copy link
Contributor

sebastianbuechler commented Jun 14, 2023

Hi @danielorbach I'm back from a week on PTO, so I'm basically catching up today.

Indeed, but IIRC we were ignoring windows containers, not windows usage... although I could probably have committed a mistake on it. I'm lacking a testing windows instance to perform consistent and repeatable tests for Windows at the moment, but this is something we are working on fixing it asap.

Would you like me to propose a pull-request addressing this?

That would be super!

The latest refactor seems to collate code-paths that need a docker configuration, I think it can be as small as a few lines of code.

Indeed, we tried to have the docker socket resolution as simple as possible, so you could probably figure it out in a very easy manner

Wouldn't it make sense in general to expand the GitHub Actions to be run on more than just Ubuntu? For example adding something like this to the test workflows?

jobs:
  test-postgres:
    strategy:
      matrix:
        go-version: [1.19.x, 1.x]
        os: [ubuntu-latest, windows-latest, macos-latest]
    runs-on: ${{ matrix.os }}

This would have detected this bug I think: https://github.com/sebastianbuechler/testcontainers-go/actions/runs/5270951017/jobs/9531389308

Btw: I think the pipeline is using the wrong path for the TestSummary action:

      - name: Test Summary
        uses: test-summary/action@4ee9ece4bca777a38f05c8fc578ac2007fe266f7
        with:
          paths: "**/TEST-postgres*.xml"
        if: always()

Where in the gotestsum this is logged out: --junitfile TEST-unit.xml So the TestSummary says it has not found any tests.

@mdelapenya
Copy link
Member

@sebastianbuechler

This would have detected this bug I think: https://github.com/sebastianbuechler/testcontainers-go/actions/runs/5270951017/jobs/9531389308

Indeed, a Windows worker would have caught it, although GH Windows workers does not support container operations: actions/runner#904 (comment)

We are exploring Azure for that, and we will have a working Windows pipeline hopefully soon

Btw: I think the pipeline is using the wrong path for the TestSummary action:

Would you mind opening a separate ticket, and if able send a PR with the fix 🙏 ? I'd appreciate that!

@sebastianbuechler
Copy link
Contributor

@sebastianbuechler

This would have detected this bug I think: https://github.com/sebastianbuechler/testcontainers-go/actions/runs/5270951017/jobs/9531389308

Indeed, a Windows worker would have caught it, although GH Windows workers does not support container operations: actions/runner#904 (comment)

We are exploring Azure for that, and we will have a working Windows pipeline hopefully soon

Btw: I think the pipeline is using the wrong path for the TestSummary action:

Would you mind opening a separate ticket, and if able send a PR with the fix 🙏 ? I'd appreciate that!

Thanks for the information. Hopefully it will work with the Azure approach 👍

I opened another bug report #1288 and also a PR #1287.

@mdelapenya mdelapenya changed the title [Bug]: Upgraded from v0.19.0 to v0.20.1 failing with "protocol not available, failed to create Docker provider" [Bug]: Failing with "protocol not available, failed to create Docker provider" on Windows Jun 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug An issue with the library
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants