secretmanager: New version upgrade of cloud.google.com/go/secretmanager to v1.14.0 is breaking while fetching the secret with below error

[raghvendra-dixit]

Client

secret manager client
issue with - cloud.google.com/go/secretmanager v1.14.0, getting below error:
Error: level=error msg="GCP Secret Manager: failed to create secretManager's NewClient: open /dev/null/.config/gcloud/certificate_config.json: not a directory"

Environment

cbl mariner distroless image
$ go version - go 1.22.6

Code and Dependencies

func (smc *GCPSMCClient) InitializeClient() (err error) {
	smc.traceLog.Debugf("GCP Secret Manager: In InitializeClient()")

	// Create the client.
	ctx := context.Background()
	var smClient *secretmanager.Client
	smClient, err = secretmanager.NewClient(ctx)
	if err != nil {
		smc.traceLog.Errorf("GCP Secret Manager: failed to create secretManager's NewClient: %v", err)
		return err
	}
	smc.client = smClient

	return err
}

go.mod

module modname

go 1.22.6

require (
	"context"
	"fmt"
	"hash/crc32"

	secretmanager "cloud.google.com/go/secretmanager/apiv1"
	"cloud.google.com/go/secretmanager/apiv1/secretmanagerpb"
)

Expected behavior

issue with - cloud.google.com/go/secretmanager v1.14.0, getting below error:
Error: level=error msg="GCP Secret Manager: failed to create secretManager's NewClient: open /dev/null/.config/gcloud/certificate_config.json: not a directory"
client works just fine with - cloud.google.com/go/secretmanager v1.12.0
As of now had to downgrade to v1.12.0 to make it work. with below changes in direct and indirect dependencies in go.mod
Please note that below go.mod is the revert from v1.14.0 to v1.12.0 in order to make secret fetching work. - denotes removal and + denotes addition, pasted the git diff of the reverted change.

go 1.22.6

require (
- cloud.google.com/go/secretmanager v1.14.0
+ cloud.google.com/go/secretmanager v1.12.0

-cloud.google.com/go/iam v1.2.0 // indirect
+cloud.google.com/go/iam v1.1.8 // indirect



-github.com/googleapis/gax-go/v2 v2.13.0 // indirect
+github.com/googleapis/gax-go/v2 v2.12.4 // indirect


-google.golang.org/api v0.194.0 // indirect
-google.golang.org/genproto v0.0.0-20240826202546-f6391c0de4c7 // indirect
+google.golang.org/api v0.182.0 // indirect
+google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda // indirect

)

Actual behavior

with cloud.google.com/go/secretmanager v1.12.0 to cloud.google.com/go/secretmanager v1.14.0, it should not break with error -
Error: level=error msg="GCP Secret Manager: failed to create secretManager's NewClient: open /dev/null/.config/gcloud/certificate_config.json: not a directory"

Screenshots

Let me know if any other details is required, will be happy to provide.

Additional context

No additional code in the client was touched, the only diff is upgrade of cloud.google.com/go/secretmanager v1.12.0 to cloud.google.com/go/secretmanager v1.14.0, which result in this break.

[codyoss]

Can you run go get cloud.google.com/go/auth@latest to see if that fixes the issue? I believe this weeks release should fix the issue here. Related: #10696

[raghvendra-dixit]

running:go get cloud.google.com/go/auth@latest

go get cloud.google.com/go/auth@latest
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.3
go: upgraded github.com/googleapis/enterprise-certificate-proxy v0.3.2 => v0.3.3: error finding sum for github.com/googleapis/[email protected]: github.com/googleapis/[email protected]: verifying module: checksum mismatch
downloaded: h1:G6q7VHBoU74wQHXFsZSLMPl0rFw0ZDrlZ3rt6/aTBII=
sum.golang.org: h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

go help module-auth
When the go command downloads a module zip file or go.mod file into the
module cache, it computes a cryptographic hash and compares it with a known
value to verify the file hasn't changed since it was first downloaded. Known
hashes are stored in a file in the module root directory named go.sum. Hashes
may also be downloaded from the checksum database depending on the values of
GOSUMDB, GOPRIVATE, and GONOSUMDB.

For details, see https://golang.org/ref/mod#authenticating.

Do I need to clear cache in order to make it work?

[codyoss]

I would try to run that command again and/or clear your cache. I am not able to reproduce that error though. Maybe you had a corrupted during download. I see the following in my .sum

github.com/googleapis/enterprise-certificate-proxy v0.3.3 h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=
github.com/googleapis/enterprise-certificate-proxy v0.3.3/go.mod h1:YKe7cfqYXjKGpGvmSg28/fFvhNzinZQm8DGnaburhGA=

[raghvendra-dixit]

not sure, i did a fresh install, cleaned cache, restarted the system
go clean --cache
followed by
go clean --modcache
After restart i ran go mod tidy to get the required dependencies and then ran the above command to pull latest go auth
still getting the same error:
go get cloud.google.com/go/auth@latest
go: downloading cloud.google.com/go/auth v0.9.4
go: downloading golang.org/x/sys v0.25.0
go: downloading google.golang.org/grpc v1.66.0
go: downloading golang.org/x/net v0.29.0
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.3
go: downloading golang.org/x/crypto v0.27.0
go: downloading golang.org/x/text v0.18.0
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1
go: upgraded github.com/googleapis/enterprise-certificate-proxy v0.3.2 => v0.3.3: error finding sum for github.com/googleapis/[email protected]: github.com/googleapis/[email protected]: verifying module: checksum mismatch
downloaded: h1:G6q7VHBoU74wQHXFsZSLMPl0rFw0ZDrlZ3rt6/aTBII=
sum.golang.org: h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.
And i see :

github.com/googleapis/enterprise-certificate-proxy v0.3.2 h1:Vie5ybvEvT75RniqhfFxPRy3Bf7vr3h0cechB90XaQs=
github.com/googleapis/enterprise-certificate-proxy v0.3.2/go.mod h1:VLSiSSBs/ksPL8kq3OBOQ6WRI2QnaFynd1DCjZ62+V0=

Let me try in another machine/colleague's machine and update in next thread if I am able to go through pas this upgrade.

[raghvendra-dixit]

Thanks @codyoss for looking into this..

Update:
I tested in my colleague's machine, and go get worked fine and i am able to fetch the latest version of cloud.google.com/go/auth@latest
gosum also shows v0.3.3

github.com/googleapis/enterprise-certificate-proxy v0.3.3 h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=
github.com/googleapis/enterprise-certificate-proxy v0.3.3/go.mod h1:YKe7cfqYXjKGpGvmSg28/fFvhNzinZQm8DGnaburhGA=

After upgrading, I verified, still throws the same error for not directory

time="2024-09-11T20:49:13.419841209Z" level=error msg="token rotation: getSMCLicenseFromCloud failed with
err=InitializeClient failed in getSMCLicenseFromCloud() with error open
 /dev/null/.config/gcloud/certificate_config.json: not a directory" 

Could this be related to the recent PR merged : https://github.com/googleapis/google-cloud-go/pull/10697/files
cc: @jba

[codyoss]

That is the change I would have suspected would fix issue, yes. That is why I was wondering if pulling cloud.google.com/go/auth @ latest would fix it. Can you share your go.mod from your example

[raghvendra-dixit]

sure, I am shairing only the impacted libs rather than entire go.mod and go.sum
go.mod

go 1.22.6
require (
	cloud.google.com/go/secretmanager v1.14.0
	golang.org/x/net v0.29.0
)
require (
	cloud.google.com/go/auth v0.9.4 // indirect
	cloud.google.com/go/iam v1.1.13 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.3 // indirect
	github.com/googleapis/gax-go/v2 v2.13.0 // indirect
	golang.org/x/crypto v0.27.0 // indirect
	google.golang.org/api v0.193.0 // indirect
	google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
	google.golang.org/grpc v1.66.0 // indirect
)
require (
	golang.org/x/sys v0.25.0 // indirect
	golang.org/x/text v0.18.0 // indirect
)

go.sum

cloud.google.com/go v0.115.1 h1:Jo0SM9cQnSkYfp44+v+NQXHpcHqlnRJk2qxh6yvxxxQ=
cloud.google.com/go v0.115.1/go.mod h1:DuujITeaufu3gL68/lOFIirVNJwQeyf5UXyi+Wbgknc=
cloud.google.com/go/auth v0.9.4 h1:DxF7imbEbiFu9+zdKC6cKBko1e8XeJnipNqIbWZ+kDI=
cloud.google.com/go/auth v0.9.4/go.mod h1:SHia8n6//Ya940F1rLimhJCjjx7KE17t0ctFEci3HkA=
cloud.google.com/go/iam v1.1.13 h1:7zWBXG9ERbMLrzQBRhFliAV+kjcRToDTgQT3CTwYyv4=
cloud.google.com/go/iam v1.1.13/go.mod h1:K8mY0uSXwEXS30KrnVb+j54LB/ntfZu1dr+4zFMNbus=
cloud.google.com/go/secretmanager v1.14.0 h1:P2RRu2NEsQyOjplhUPvWKqzDXUKzwejHLuSUBHI8c4w=
cloud.google.com/go/secretmanager v1.14.0/go.mod h1:q0hSFHzoW7eRgyYFH8trqEFavgrMeiJI4FETNN78vhM=
github.com/googleapis/enterprise-certificate-proxy v0.3.3 h1:QRje2j5GZimBzlbhGA2V2QlGNgL8G6e+wGo/+/2bWI0=
github.com/googleapis/enterprise-certificate-proxy v0.3.3/go.mod h1:YKe7cfqYXjKGpGvmSg28/fFvhNzinZQm8DGnaburhGA=
github.com/googleapis/gax-go/v2 v2.13.0 h1:yitjD5f7jQHhyDsnhKEBU52NdvvdSeGzlAnDPT0hH1s=
github.com/googleapis/gax-go/v2 v2.13.0/go.mod h1:Z/fvTZXF8/uw7Xu5GuslPw+bplx6SS338j1Is2S+B7A=
go.opentelemetry.io/otel/sdk v1.28.0 h1:b9d7hIry8yZsgtbmM0DKyPWMMUMlK9NEKuIG4aBqWyE=
go.opentelemetry.io/otel/sdk v1.28.0/go.mod h1:oYj7ClPUA7Iw3m+r7GeEjz0qckQRJK2B8zjcZEfu7Pg=
golang.org/x/crypto v0.27.0 h1:GXm2NjJrPaiv/h1tb2UH8QfgC/hOf/+z0p6PT8o1w7A=
golang.org/x/crypto v0.27.0/go.mod h1:1Xngt8kV6Dvbssa53Ziq6Eqn0HqbZi5Z6R0ZpwQzt70=
golang.org/x/net v0.29.0 h1:5ORfpBpCs4HzDYoodCDBbwHzdR5UrLBZ3sOnUJmFoHo=
golang.org/x/net v0.29.0/go.mod h1:gLkgy8jTGERgjzMic6DS9+SP0ajcu6Xu3Orq/SpETg0=
golang.org/x/sys v0.25.0 h1:r+8e+loiHxRqhXVl6ML1nO3l1+oFoWbnlu2Ehimmi34=
golang.org/x/sys v0.25.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224=
golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
google.golang.org/api v0.193.0 h1:eOGDoJFsLU+HpCBaDJex2fWiYujAw9KbXgpOAMePoUs=
google.golang.org/api v0.193.0/go.mod h1:Po3YMV1XZx+mTku3cfJrlIYR03wiGrCOsdpC67hjZvw=
google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142 h1:oLiyxGgE+rt22duwci1+TG7bg2/L1LQsXwfjPlmuJA0=
google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142/go.mod h1:G11eXq53iI5Q+kyNOmCvnzBaxEA2Q/Ik5Tj7nqBE8j4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 h1:pPJltXNxVzT4pK9yD8vR9X75DaWYYmLGMsEvBfFQZzQ=
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1/go.mod h1:UqMtugtsSgubUsoxbuAoiCXvqvErP7Gf0so0mK9tHxU=
google.golang.org/grpc v1.66.0 h1:DibZuoBznOxbDQxRINckZcUvnCEvrW9pcWIE2yF9r1c=
google.golang.org/grpc v1.66.0/go.mod h1:s3/l6xSSCURdVfAnL+TqCNMyTDAGN6+lZeVxnZR128Y=

[raghvendra-dixit]

hi @codyoss @quartzmo Wanted to check if we have noticed this issue, or there are any prospects of fixing this in coming releases. I am thinking this is broken in current release of secret manager APIs.

[codyoss]

Would you mind upgrading to https://github.com/googleapis/google-cloud-go/releases/tag/auth/v0.9.9. I believe this should fix the issue.