auth: universe/universe_domain calls honors HTTP_PROXY envvar where it did not before

[oschwald]

Client

trace

Environment

GCE

Go Environment

$ go version

go version go1.22.4 linux/amd64

$ go env

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/goschwald_maxmind_com/.cache/go-build'
GOENV='/home/goschwald_maxmind_com/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/goschwald_maxmind_com/go/pkg/mod'
GONOPROXY=''
GONOSUMDB='github.maxmind.com/*'
GOOS='linux'
GOPATH='/home/goschwald_maxmind_com/go'
GOPRIVATE=''
GOPROXY='https://artifactory.maxmind.com/artifactory/api/go/go/'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3338796299=/tmp/go-build -gno-record-gcc-switches'

Code

e.g.

package main

import (
	"context"

	trace "cloud.google.com/go/trace/apiv2"
	"cloud.google.com/go/trace/apiv2/tracepb"
	"google.golang.org/api/option"
	"google.golang.org/grpc"
)

func main() {
	ctx := context.Background()

	c, err := trace.NewClient(ctx, option.WithGRPCDialOption(grpc.WithNoProxy()))
	if err != nil {
		panic(err)
	}
	defer c.Close()

	req := &tracepb.BatchWriteSpansRequest{
		Name:  "projects/mm-lab-1234",
		Spans: []*tracepb.Span{},
	}
	err = c.BatchWriteSpans(ctx, req)
	if err != nil {
		panic(err)
	}
}

Expected behavior

With cloud.google.com/go/[email protected] and earlier, this code completes without error when the http_proxy environment variable is set. It does not try to connect to the proxy as grpc.WithNoProxy() is set.

Actual behavior

As of 1.10.8, this code now tries to connect to a proxy set via http_proxy. For instance, if I set it to an invalid value, e.g., 255.255.255.255, I get an error like:

panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: Get "http://169.254.169.254/computeMetadata/v1/universe/universe_domain": proxyconnect tcp: dial tcp 255.255.255.255:80: connect: network is unreachable

goroutine 1 [running]:
main.main()
	trace/main.go:33 +0x15e

Additional context

As mentioned, this started after upgrading to 1.10.8 or newer. It appears to be related to the new auth.

In our use case, we have http_proxy and https_proxy set. All external requests go through our proxy server. For requests that we consider "internal", such as those to the GCP metadata server, we explicitly disable the proxy for clients making these requests. Although setting no_proxy=169.254.169.254 would be a workaround, we would prefer to disable the proxy on a case-by-case basis.

[codyoss]

Hey thanks for the report. I think you are right this is is likely related to using our new auth library but it looks like the options are passed through from my quick testing. And I am unable to reproduce this locally when using service account credentials. Here is another example that does not relate to proxies that shows the option is being passed through:

package main

import (
	"context"
	"fmt"
	"log"
	"log/slog"
	"os"

	trace "cloud.google.com/go/trace/apiv2"
	"cloud.google.com/go/trace/apiv2/tracepb"
	"google.golang.org/api/option"
	"google.golang.org/grpc"
	"google.golang.org/grpc/metadata"
	"google.golang.org/protobuf/encoding/protojson"
	"google.golang.org/protobuf/reflect/protoreflect"
)

func main() {
	if err := run(); err != nil {
		slog.Error("shutting down", "err", err)
		os.Exit(1)
	}
}

func run() error {
	ctx := context.Background()
	projectID := "project-id"
	c, err := trace.NewClient(ctx,
		option.WithGRPCDialOption(grpc.WithUnaryInterceptor(loggingUnaryInterceptor())),
	)
	if err != nil {
		return err
	}
	defer c.Close()

	req := &tracepb.BatchWriteSpansRequest{
		Name:  fmt.Sprintf("projects/%s", projectID),
		Spans: []*tracepb.Span{},
	}
	err = c.BatchWriteSpans(ctx, req)
	if err != nil {
		panic(err)
	}

	return nil
}

func loggingUnaryInterceptor() grpc.UnaryClientInterceptor {
	return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
		err := invoker(ctx, method, req, reply, cc, opts...)
		log.Printf("Invoked method: %v", method)
		md, ok := metadata.FromOutgoingContext(ctx)
		if ok {
			log.Println("Metadata:")
			for k, v := range md {
				log.Printf("Key: %v, Value: %v", k, v)
			}
		}
		reqb, merr := protojson.Marshal(req.(protoreflect.ProtoMessage))
		if merr == nil {
			log.Printf("Request: %s", reqb)
		}
		return err
	}
}

Looking at the error it is a failed standard HTTP client call that failed. That option for gRPC does not affect the underlying HTTP client that is used to talk to the metadata service. I will have to look some more but perhaps the client being used now is honoring the environment variable where it was not before.

[codyoss]

I think this is the case the old code path would have used the default client in the metadata package that creates a transport:

google-cloud-go/compute/metadata/metadata.go

Lines 67 to 75 in 978d4a1

	Transport: &http.Transport{
	Dial: (&net.Dialer{
	Timeout: 2 * time.Second,
	KeepAlive: 30 * time.Second,
	}).Dial,
	IdleConnTimeout: 60 * time.Second,
	},
	Timeout: 5 * time.Second,
	}

The new codepath creates a client for this check and does not declare a transport so it falls back to Go's default:

google-cloud-go/auth/internal/internal.go

Lines 184 to 185 in 978d4a1

	client := metadata.NewClient(&http.Client{Timeout: time.Second})
	return client.GetWithContext(ctx, "universe/universe_domain")

Will have to investigate this more though.

[oschwald]

When running your example, I see:

$ http_proxy=255.255.255.255 ./trace2
2024/07/12 22:10:45 Invoked method: /google.devtools.cloudtrace.v2.TraceService/BatchWriteSpans
2024/07/12 22:10:45 Metadata:
2024/07/12 22:10:45 Key: x-goog-api-client, Value: [gl-go/1.22.4 gapic/1.10.9 gax/2.12.5 grpc/1.64.0]
2024/07/12 22:10:45 Key: x-goog-request-params, Value: [name=projects%2Fproject-id]
2024/07/12 22:10:45 Request: {"name":"projects/project-id"}
panic: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: Get "http://169.254.169.254/computeMetadata/v1/universe/universe_domain": proxyconnect tcp: dial tcp 255.255.255.255:80: connect: network is unreachable

goroutine 1 [running]:
main.run()
	trace2/main.go:43 +0x25b
main.main()
	trace2/main.go:20 +0x17

Please let me know if there is any other useful information that I can provide.

[codyoss]

Yeah thanks. My main point there is that the options are being passed through fine, sorry for the confusion. I think we can "fix" this, restore to the old behaviour. But I am not sure the old behaviour is actually a great one of not honoring HTTP_PROXY.

[oschwald]

Thanks. Yes, I understood your comment. I think the ideal behavior would be a consistent way to disable the use of the proxy for the client, but I recognize that is not a simple ask. The change in behavior between the new and old version was certainly unexpected in a patch release.

[codyoss]

I did create a PR to restore the previous behavior, at least for now. This may change in a future release though as I don't think it is intentional that the metadata package is ignoring standard proxy envvars.