cmd/go 'go mod download <package name>' fetches all modules if module cache is empty/missing

[mikemoate]

What version of Go are you using (go version)?

$ go version

go version go1.14.5 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/hello/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build922582841=/tmp/go-build -gno-record-gcc-switches"

What did you do?

https://golang.org/ref/mod#tmp_28 describes how go mod download can be used to pre-fill the module cache, and how the module paths/patterns to download can be specified. In practice, we find executing this command with an empty/missing module cache causes the download of all modules in go.mod and their dependencies, rather than just the specified module path.

A minimal example to reproduce this (using Docker to ensure a clean Go environment each time) is at https://github.com/mikemoate/go-mod-download-issue the output of this is:

docker build Output

$ docker build .
Sending build context to Docker daemon  71.68kB

Step 1/6 : FROM golang:1.14

---> 2b88645b406c

Step 2/6 : WORKDIR /hello

---> Using cache

---> 59e415a309ac

Step 3/6 : COPY . .

---> f6315f0f3481

Step 4/6 : RUN go mod download -x rsc.io/quote

---> Running in dcf267779b36

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.mod

# get https://proxy.golang.org/github.com/google/uuid/@v/v1.1.1.mod

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.mod: 200 OK (0.127s)

# get https://proxy.golang.org/rsc.io/sampler/@v/v1.3.0.mod

# get https://proxy.golang.org/github.com/google/uuid/@v/v1.1.1.mod: 200 OK (0.141s)

# get https://proxy.golang.org/rsc.io/sampler/@v/v1.3.0.mod: 200 OK (0.013s)

# get https://proxy.golang.org/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.mod

# get https://proxy.golang.org/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.mod: 200 OK (0.012s)

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.info

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.info: 200 OK (0.012s)

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.zip

# get https://proxy.golang.org/rsc.io/quote/@v/v1.5.2.zip: 200 OK (0.013s)

Removing intermediate container dcf267779b36

---> 0a038686c447

Step 5/6 : RUN go version

---> Running in 0d7f569a56eb

go version go1.14.5 linux/amd64

Removing intermediate container 0d7f569a56eb

---> 50c1d1104909

Step 6/6 : RUN go env

---> Running in b613c094cd1e

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/root/.cache/go-build"

GOENV="/root/.config/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOINSECURE=""

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD="/hello/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build704505364=/tmp/go-build -gno-record-gcc-switches"

Removing intermediate container b613c094cd1e

---> fcdae266e02e

Successfully built fcdae266e02e

go.mod Content

module main
go 1.14
require (

github.com/google/uuid v1.1.1

rsc.io/quote v1.5.2

)

You can clearly see the get for module github.com/google/uuid, despite this not being requested and not being a dependency of the requested module rsc.io/quote

Our use case here is to pre-fetch private modules individually, specifying appropriate credentials for each, ahead of performing a build. This fails because the attempt to fetch an individual private module, tries to download all the private modules in go.mod.

What did you expect to see?

Only the requested module is downloaded.

What did you see instead?

All modules were downloaded.

[andybons]

@jayconrod @matloob @bcmills

[bcmills]

If you go mod download a module without a specific version, the go command uses the version implied by the main module's go.mod file. In order to compute that version, it reads the go.mod file and the go.mod files of the transitive dependencies, because the go.mod file could be in an inconsistent or misleading state and the actual selected version may be higher.

If you were to specify an explicit version for the module, then in theory we would not have to load the dependencies, but we do anyway today. (That is #36812.)

With the proposed lazy loading semantics (#36460), we would make stronger consistency assumptions and load fewer go.mod files (in this case, just the main module's go.mod file, since it specifies the version of the requested module directly). However, in general we may still have to load go.mod files for one layer of transitive dependencies to find the appropriate version.

[bcmills]

Duplicate of #36812