[Bug]: Cannot import testcontainers-go without locking to specific versions of indirect dependencies #634

cyberbeast · 2022-11-22T14:44:45Z

Testcontainers version

0.16.0

Using the latest Testcontainers version?

Yes

Host OS

Macos

Host arch

x86

Go version

1.19

Docker version

Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:35 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.13.1 (90346)
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:35 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.12.1)
  dev: Docker Dev Environments (Docker Inc., v0.0.3)
  extension: Manages Docker extensions (Docker Inc., v0.2.13)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 24
 Server Version: 20.10.20
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.772GiB
 Name: docker-desktop
 ID: UT6I:Z3MH:BVO7:4GHT:YRHP:MGEJ:734U:DELR:HXWX:AEIS:XLYK:VATP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

Cannot import testcontainers-go without a bunch of indirect dependencies failing.

I have to add the following the replace directives to make it work.

replace (
	github.com/docker/cli => github.com/docker/cli v20.10.3-0.20221013132413-1d6c6e2367e2+incompatible // 22.06 master branch
	github.com/docker/docker => github.com/docker/docker v20.10.3-0.20221013203545-33ab36d6b304+incompatible // 22.06 branch
	github.com/moby/buildkit => github.com/moby/buildkit v0.10.1-0.20220816171719-55ba9d14360a // same as buildx

	github.com/opencontainers/runc => github.com/opencontainers/runc v1.1.2 // Can be removed on next bump of containerd to > 1.6.4

	// For k8s dependencies, we use a replace directive, to prevent them being
	// upgraded to the version specified in containerd, which is not relevant to the
	// version needed.
	// See https://github.com/docker/buildx/pull/948 for details.
	// https://github.com/docker/buildx/blob/v0.8.1/go.mod#L62-L64
	k8s.io/api => k8s.io/api v0.22.4
	k8s.io/apimachinery => k8s.io/apimachinery v0.22.4
	k8s.io/client-go => k8s.io/client-go v0.22.4
)

If I don't lock these indirect dependencies to these versions (and use latest) things break.

Relevant log output

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

mdelapenya · 2022-11-22T14:50:50Z

Hi @cyberbeast is this related to #629?

We discussed there how to bypass it, sending PRs to update the docs and explaining why the replace directives.

mdelapenya · 2022-11-22T15:08:42Z

Did not mention in my previous comment, are the docs and the explanation in the release notes enough for you to consider this issue as closed? Other than that, please let us know if you have any suggestion on how to fix the transitive dependency with Docker Compose

cyberbeast · 2022-11-22T15:15:12Z

Reviewing #629 now

cyberbeast · 2022-11-22T15:25:57Z

@mdelapenya I reviewed #629, yes looks like it is related but the solution proposed there isn't sufficient for the errors I am seeing. I tried the suggested replace directive you mentioned. While the single replace directive,

replace github.com/docker/docker => github.com/docker/docker v20.10.3-0.20221013203545-33ab36d6b304+incompatible // 22.06 branch

resolves my go.mod issues, however, when I run my tests using go test ./... -cover, I get the following errors

# github.com/moby/buildkit/session/filesync
../../../go/pkg/mod/github.com/moby/[email protected]/session/filesync/filesync.go:112:20: cannot use dir.Map (variable of type func(string, *types.Stat) bool) as type fsutil.MapFunc in struct literal
# github.com/docker/cli/cli/registry/client
../../../go/pkg/mod/github.com/docker/[email protected]+incompatible/cli/registry/client/endpoint.go:93:40: assignment mismatch: 3 variables but registry.PingV2Registry returns 2 values
# github.com/docker/buildx/driver/kubernetes/manifest
../../../go/pkg/mod/github.com/docker/[email protected]/driver/kubernetes/manifest/manifest.go:92:9: unknown field 'Handler' in struct literal of type "k8s.io/api/core/v1".Probe
../../../go/pkg/mod/github.com/docker/[email protected]/driver/kubernetes/manifest/manifest.go:92:25: undefined: v1.Handler

The only way I can resolve this is by using ALL of the directives from here

cyberbeast · 2022-11-22T15:28:57Z

I see that the release notes mention adding the single replace directive for github.com/docker/docker, but the way I am seeing it, we may need to include ALL the replace directives (see link in my previous comment or the issue description) to truly resolve build issues when importing 0.16.0

mdelapenya · 2022-11-22T15:42:21Z

Indeed, you are right. I did a test in a sample Go repo and because one single test passed (debugged it with VSCode) I thought it was enough.

Will immediately update the release notes, and the docs site.

Would you mind reviewing the upcoming PR for that?

cyberbeast · 2022-11-22T15:43:05Z

Absolutely @mdelapenya :)
Thanks for looking into this.

mdelapenya · 2022-11-22T15:46:44Z

Changelog and PR updated: #632

ouamer-dahmani · 2022-11-29T14:18:38Z

Hello,
I had to go through similar hoops before importing testcontainers-go passed the go mod download and go mod tidy -v -e steps.

But now when I try to run a test attempting to use testcontainers.NewDockerCompose and I get the following errors:

Build Error: go build -o /home/user/work/git/clever-commit/2/cmd/clever_triage/local/__debug_bin -gcflags all=-N -l ./main.go
# github.com/aws/aws-sdk-go-v2/ec2imds
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/api_client.go:55:3: cannot use awsmiddleware.AddUserAgentKey("ec2imds") (value of type func(*"github.com/aws/smithy-go/middleware".Stack) error) as type func(*"github.com/awslabs/smithy-go/middleware".Stack) error in argument to append
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/api_client.go:64:24: invalid operation: options.Retryer == nil (operator == not defined on untyped nil)
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/api_client.go:100:57: cannot use cfg.APIOptions (variable of type []func(*"github.com/aws/smithy-go/middleware".Stack) error) as type []func(*"github.com/awslabs/smithy-go/middleware".Stack) error in argument to append
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/api_client.go:131:16: undefined: retry.Retryer
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/api_client.go:229:16: undefined: aws.NewBuildableHTTPClient
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/api_client.go:232:27: undefined: aws.BuildableHTTPClient
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/api_client.go:242:20: undefined: aws.BuildableHTTPClient
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/request_middleware.go:29:10: undefined: retry.AttemptMiddleware
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/request_middleware.go:50:49: cannot use stack (variable of type *"github.com/awslabs/smithy-go/middleware".Stack) as type *"github.com/aws/smithy-go/middleware".Stack in argument to awsmiddleware.AddUserAgentKey(ServiceID)
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/request_middleware.go:89:28: cannot use stack (variable of type *"github.com/awslabs/smithy-go/middleware".Stack) as type *"github.com/aws/smithy-go/middleware".Stack in argument to retry.AddRetryMiddlewares
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/[email protected]/request_middleware.go:89:28: too many errors
# github.com/aws/aws-sdk-go-v2/service/ssm
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:84:16: undefined: retry.Retryer
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:140:16: cannot use cfg.APIOptions (variable of type []func(*"github.com/aws/smithy-go/middleware".Stack) error) as type []func(*"github.com/awslabs/smithy-go/middleware".Stack) error in struct literal
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:150:21: undefined: aws.NewBuildableHTTPClient
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:154:18: invalid operation: o.Retryer != nil (operator != not defined on untyped nil)
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:168:39: cannot use stack (variable of type *"github.com/awslabs/smithy-go/middleware".Stack) as type *"github.com/aws/smithy-go/middleware".Stack in argument to awsmiddleware.AddUserAgentKey("ssm")
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:172:21: cannot use v4.NewSignHTTPRequestMiddleware(o.Credentials, o.HTTPSignerV4) (value of type *"github.com/aws/aws-sdk-go-v2/aws/signer/v4".SignHTTPRequestMiddleware) as type "github.com/awslabs/smithy-go/middleware".FinalizeMiddleware in argument to stack.Finalize.Add:
	*"github.com/aws/aws-sdk-go-v2/aws/signer/v4".SignHTTPRequestMiddleware does not implement "github.com/awslabs/smithy-go/middleware".FinalizeMiddleware (wrong type for HandleFinalize method)
		have HandleFinalize(ctx context.Context, in "github.com/aws/smithy-go/middleware".FinalizeInput, next "github.com/aws/smithy-go/middleware".FinalizeHandler) (out "github.com/aws/smithy-go/middleware".FinalizeOutput, metadata "github.com/aws/smithy-go/middleware".Metadata, err error)
		want HandleFinalize(ctx context.Context, in "github.com/awslabs/smithy-go/middleware".FinalizeInput, next "github.com/awslabs/smithy-go/middleware".FinalizeHandler) (out "github.com/awslabs/smithy-go/middleware".FinalizeOutput, metadata "github.com/awslabs/smithy-go/middleware".Metadata, err error)
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:172:68: too many arguments in call to v4.NewSignHTTPRequestMiddleware
	have (aws.CredentialsProvider, HTTPSignerV4)
	want ("github.com/aws/aws-sdk-go-v2/aws/signer/v4".SignHTTPRequestMiddlewareOptions)
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:186:19: cannot use v4.NewSigner(configureSignerV4) (value of type *"github.com/aws/aws-sdk-go-v2/aws/signer/v4".Signer) as type HTTPSignerV4 in assignment:
	*"github.com/aws/aws-sdk-go-v2/aws/signer/v4".Signer does not implement HTTPSignerV4 (wrong type for SignHTTP method)
		have SignHTTP(ctx context.Context, credentials aws.Credentials, r *"net/http".Request, payloadHash string, service string, region string, signingTime "time".Time, optFns ...func(options *"github.com/aws/aws-sdk-go-v2/aws/signer/v4".SignerOptions)) error
		want SignHTTP(ctx context.Context, credentials aws.Credentials, r *"net/http".Request, payloadHash string, service string, region string, signingTime "time".Time) error
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:187:3: cannot use configureSignerV4 (value of type func(s *"github.com/aws/aws-sdk-go-v2/aws/signer/v4".Signer)) as type func(signer *"github.com/aws/aws-sdk-go-v2/aws/signer/v4".SignerOptions) in argument to v4.NewSigner
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:202:35: cannot use stack (variable of type *"github.com/awslabs/smithy-go/middleware".Stack) as type *"github.com/aws/smithy-go/middleware".Stack in argument to retry.AddRetryMiddlewares
/home/user/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:202:35: too many errors (exit status 2)

mdelapenya · 2022-11-29T14:30:22Z

@ouamer-dahmani can you share a repo or snippet with the failure, and what version of Go are you using?

ouamer-dahmani · 2022-11-29T15:50:32Z

Hello,

I do not have simplified reproduction and the code I am working on cannot be shared.

I believe I am hitting a problem caused by another dependency needing a different version of the aws-sdk-go-v2 package. My question now would be why does testcontainers need the aws sdk? Or does it come from a transitive dependency?

mdelapenya · 2022-11-29T20:39:53Z

We are not using AWS deps directly, probably an indirect one

johanbrandhorst · 2022-12-06T23:31:48Z

Could we reopen this issue to track when we'll be able to remove these replace directives? It's all fine to say "we hope to be able to remove it in the next release" but this issue is still a problem until that is resolved, as far as I can tell. Thanks!

mdelapenya · 2022-12-07T20:13:20Z

Could we reopen this issue to track when we'll be able to remove these replace directives? It's all fine to say "we hope to be able to remove it in the next release" but this issue is still a problem until that is resolved, as far as I can tell. Thanks!

Indeed, makes sense. I'm reopening it right now

mdelapenya · 2022-12-23T15:49:30Z

Both #706 and #650 were merged and released in 0.17.0, please let us know if the dependency management is clear enough 🙏

PTAL at the release notes there, as we only need to replace Docker, removing a big amount of replacements (still needed if you use the compose module).

ravilushqa · 2023-01-26T08:57:09Z

I faced an issue with another one external library
../../../go/pkg/mod/oras.land/[email protected]/pkg/auth/docker/login_tls.go:181:30: assignment mismatch: 3 variables but registry.PingV2Registry returns 2 values

Looks like the issue because of replacing docker/docker

mdelapenya · 2023-01-26T09:06:12Z

Looks like the issue because of replacing docker/docker

@ravilushqa can you share more on your environment: Go version, Docker version? We are testing the library against Go 1.18 and 1.x without errors on CI, so I'd like to identity your issues with a reproducible environment

mdelapenya · 2023-02-17T07:04:30Z

As a result of the new 0.18.0 release, which includes a bump in Docker Compose code that removes the replace directive for Docker, I'm going to close this issue.

Please reopen if you consider it's not done or still have issues with it.

Thanks for your patience

cyberbeast added the bug An issue with the library label Nov 22, 2022

cyberbeast changed the title ~~[Bug]: Cannot import testcontainers-go~~ [Bug]: Cannot import testcontainers-go without locking to specific versions of indirect dependencies Nov 22, 2022

mdelapenya mentioned this issue Nov 22, 2022

docs: document replace directive for Docker Compose #632

Merged

mdelapenya closed this as completed in #632 Nov 22, 2022

mdelapenya reopened this Dec 7, 2022

This was referenced Dec 15, 2022

[Bug]: cannot install testcontainers #669

Closed

[Bug]: Can't update this dependency #687

Closed

[Bug]: Go get fails on panic: internal error: can't find reason for requirement on github.com/moby/sys/[email protected] #705

Closed

mdelapenya closed this as not planned Won't fix, can't repro, duplicate, stale Feb 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Cannot import testcontainers-go without locking to specific versions of indirect dependencies #634

[Bug]: Cannot import testcontainers-go without locking to specific versions of indirect dependencies #634

cyberbeast commented Nov 22, 2022

mdelapenya commented Nov 22, 2022

mdelapenya commented Nov 22, 2022

cyberbeast commented Nov 22, 2022

cyberbeast commented Nov 22, 2022 •

edited

Loading

cyberbeast commented Nov 22, 2022 •

edited

Loading

mdelapenya commented Nov 22, 2022

cyberbeast commented Nov 22, 2022

mdelapenya commented Nov 22, 2022

ouamer-dahmani commented Nov 29, 2022 •

edited

Loading

mdelapenya commented Nov 29, 2022

ouamer-dahmani commented Nov 29, 2022

mdelapenya commented Nov 29, 2022

johanbrandhorst commented Dec 6, 2022

mdelapenya commented Dec 7, 2022

mdelapenya commented Dec 23, 2022

ravilushqa commented Jan 26, 2023

mdelapenya commented Jan 26, 2023

mdelapenya commented Feb 17, 2023

[Bug]: Cannot import testcontainers-go without locking to specific versions of indirect dependencies #634

[Bug]: Cannot import testcontainers-go without locking to specific versions of indirect dependencies #634

Comments

cyberbeast commented Nov 22, 2022

Testcontainers version

Using the latest Testcontainers version?

Host OS

Host arch

Go version

Docker version

Docker info

What happened?

Relevant log output

Additional information

mdelapenya commented Nov 22, 2022

mdelapenya commented Nov 22, 2022

cyberbeast commented Nov 22, 2022

cyberbeast commented Nov 22, 2022 • edited Loading

cyberbeast commented Nov 22, 2022 • edited Loading

mdelapenya commented Nov 22, 2022

cyberbeast commented Nov 22, 2022

mdelapenya commented Nov 22, 2022

ouamer-dahmani commented Nov 29, 2022 • edited Loading

mdelapenya commented Nov 29, 2022

ouamer-dahmani commented Nov 29, 2022

mdelapenya commented Nov 29, 2022

johanbrandhorst commented Dec 6, 2022

mdelapenya commented Dec 7, 2022

mdelapenya commented Dec 23, 2022

ravilushqa commented Jan 26, 2023

mdelapenya commented Jan 26, 2023

mdelapenya commented Feb 17, 2023

cyberbeast commented Nov 22, 2022 •

edited

Loading

cyberbeast commented Nov 22, 2022 •

edited

Loading

ouamer-dahmani commented Nov 29, 2022 •

edited

Loading