cmd/go: module cache can cache inappropriate commits

[rogpeppe]

What version of Go are you using (go version)?

$ go version
go version devel +22dbc96d8a Tue Nov 27 09:26:16 2018 +0000 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/rog/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/rog/src/go"
GOPROXY="http://localhost:3000"
GORACE=""
GOROOT="/home/rog/go"
GOTMPDIR=""
GOTOOLDIR="/home/rog/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/rog/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build723184334=/tmp/go-build -gno-record-gcc-switches"
% rm go.mod
% go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/rog/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/rog/src/go"
GOPROXY="http://localhost:3000"
GORACE=""
GOROOT="/home/rog/go"
GOTMPDIR=""
GOTOOLDIR="/home/rog/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build238480985=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I was building a module with a dependency that had been forked.
The go.mod file specified this dependency.

github.com/altoros/gosigma v0.0.0-20170523021020-a27b59fe2be9 // indirect

Note that commit a27b59fe2be9 does not appear in the github.com/altoros/gosigma repository, although it does appear in a PR on that repository. The dependency does not contain a go.mod file.

The commit is also in a fork of the repository at its HEAD (github.com/juju/gosigma).

My build succeeds locally, and I can see that somehow that commit has made its way into $GOPATH/pkg/mod:

% cd $GOPATH/pkg/mod
% find . -name '*sigma*'
./cache/download/github.com/juju/gosigma
./cache/download/github.com/altoros/gosigma
./github.com/juju/[email protected]/provider/cloudsigma
./github.com/juju/[email protected]
./github.com/juju/[email protected]/gosigma.go
./github.com/juju/[email protected]/gosigma_test.go
./github.com/juju/[email protected]/provider/cloudsigma
./github.com/juju/[email protected]/provider/cloudsigma
./github.com/juju/[email protected]/patches/gosigma-race-fix.diff
./github.com/juju/[email protected]/provider/cloudsigma
./github.com/juju/[email protected]/patches/gosigma-race-fix.diff
./github.com/juju/[email protected]/provider/cloudsigma
./github.com/altoros/[email protected]
./github.com/altoros/[email protected]/gosigma.go
./github.com/altoros/[email protected]/gosigma_test.go
./github.com/altoros/[email protected]
./github.com/altoros/[email protected]/gosigma.go
./github.com/altoros/[email protected]/gosigma_test.go

When I pushed my package to be tested with CI, the CI failed because it could not find github.com/altoros/[email protected].

Unfortunately, I don't know how this happened, but I'm recording the issue here as a placeholder in case other people see the same thing. When I tried to reproduce the issue, I failed. I tried with a fresh GOPATH and go get github.com/juju/[email protected], in case that would poison the cache, but it didn't.

What did you expect to see?

I expected to get a build failure locally.

What did you see instead?

The build succeeded locally but failed with a fresh $GOPATH.

[bcmills]

Is it possible that your module cache was very old?

https://golang.org/cl/126956 (released in go1.11) is the last change to that logic I'm aware of, but if you happened to have a larger git history already fetched into your module cache it could include otherwise-unreachable commits.

[rogpeppe]

I'm fairly sure that I'd removed the cache directory relatively recently. FWIW the modification date on the erroneous cache entry was in the last couple of days.

[bcmills]

@rogpeppe, have you seen this again recently? Without steps to reproduce I'm not sure there's anything we can do to move forward on this. (My best guess is maybe a difference in the way git fetch works across git versions.)

[rogpeppe]

No, I haven't seen this issue again since.

[bcmills]

Ok. I'm going to close this issue for now, but please do reopen if you see it again.