cmd/go: global go get does not reference main package's module's go.sum

[myitcv]

What version of Go are you using (go version)?

$ go version
go version devel +b7689f5aa3 Fri Jan 31 06:02:00 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/tmp/tmp.lzY0uMO3kp"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build624481406=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The current advice for globally installing a tool is some variant of:

(cd $(mktemp -d); GO111MODULE=on go get example.com/tool)

The following command sequence succeeds:

export GOPATH=$(mktemp -d)
cd $(mktemp -d)
GO111MODULE=on go get github.com/myitcvscratch/[email protected]

However the following command sequence does not:

cd $(mktemp -d)
git clone -q https://github.com/myitcvscratch/badmain
cd badmain/
git checkout d46c956
go install

It fails with:

verifying golang.org/x/[email protected]: checksum mismatch
        downloaded: h1:DPqS0AlgYBVHhG5jnEVScBXXIS+xjgn7O8s1E3sDqxc=
        go.sum:     h1:DPqS0AlgYBVHhG5jnEVScBXXIS+xjgn7O8s1E3sD=

(an error I have deliberately introduced)

What did you expect to see?

Given we are installing a tool outside of a module context, have no go.{mod,sum} and hence are relying on a temporary module, part of me is not surprised by this behaviour.

However I think it's problematic. It feels like the default advice for installing tools globally should use some go.sum reference when installing said tool, rather than blindly accepting whatever it sees. In this case it could use the go.sum that is part of the github.com/myitcvscratch/badmain module.

Arguably go get's of this sort should fail if either the go.mod or go.sum is incomplete, but that's perhaps another issue.

What did you see instead?

As above.

[myitcv]

cc @bcmills @jayconrod

[bcmills]

This is closely related to #28802, which was withdrawn.

The default go.sum reference, as of Go 1.13, is the global checksum database. (The only modules that are not verified today during a global install are those that intentionally bypass the database due to GOPRIVATE or GONOSUMDB.)

[bcmills]

(CC @FiloSottile @katiehockman @matloob)

[myitcv]

Thanks @bcmills

The default go.sum reference, as of Go 1.13, is the global checksum database

Indeed. My description above was inaccurate ("blindly accepting whatever it sees") - thanks for the correction.

My asking this question is in the context of whether a go get of this sort should fail if the main package's module's go.mod is incomplete. Because if it is then we're not guaranteed to have reproducible builds.

Hence it seemed logical (to me at least!) to use the main package's module's go.sum for authentication, and similarly fail if it was incomplete/didn't verify.

The extent to which either of these points are a problem, though, is not totally clear to me. Hence I raised this issue more for discussion.

[bcmills]

My asking this question is in the context of whether a go get of this sort should fail if the main package's module's go.mod is incomplete.

Ah, that is a different (and also interesting) question!

go get explicitly updates module dependencies, so in general (within or outside of a module) it does resolve missing imports. But if we're in an explicit “global install” mode (#30515) — and especially if that “global install” mode has a well-defined notion of “main module” — then it might make sense to use the main module's go.mod and go.sum file as well.

(But I think that observation probably belongs on #30515, since absent an explicit “global install” flag we would like go get to have the same behavior both within and outside of a main module.)

[myitcv]

Per Slack, I don't even know what name to give this mode:

(cd $(mktemp -d); GO111MODULE=on go get example.com/tool)

So for the purposes of what follows, I will refer to it as "mode Y" of go get 😄

go get explicitly updates module dependencies, so in general (within or outside of a module) it does resolve missing imports

I guess my point is that in "mode Y" the user is blind to the fact that a temporary module is being created, used and then thrown away. So they are also blind to the fact that the step is potentially not reproducible.

But if we're in an explicit “global install” mode (#30515)

Indeed, there's definitely overlap with that discussion. Per my comment above, I suppose I'm not really clear what "mode Y" is useful for, or whether we should be advising people to rely on/use it.

... then it might make sense to use the main module's go.mod and go.sum file as well

We could even use the main module's non-directory replace directives as well 🏃

But I think that observation probably belongs on #30515,

Agreed. This issue was about the go.sum point. As I said above, happy for you to close this as "working as intended" if we don't think there's an issue here.

On that very point, is there some documentation somewhere that describes what happens in "mode Y"? Per my Slack message, I couldn't find any.

[bcmills]

We've been using the term “module mode outside of a main module” for “mode Y”, but that is admittedly pretty verbose.

The behavior outside of a module is not very thoroughly documented at this point because it has been in flux (#24250 in 1.12, #32027 in 1.14, perhaps #30515 in 1.15 if we can straighten out the semantics, maybe #36513 at some point). It probably still needs some fine-tuning, so we certainly appreciate the input!

[FiloSottile]

For what it's worth, there is a degree of per-machine reproducibility because the tree head of the checksum database which is stored in $GOPATH/pkg/sumdb/sum.golang.org/latest can only ratchet forward on a certain machine, and it can't include disagreeing values for the same module version without detection by auditors.

“Module mode outside of a main module” can't get any better than that I suspect, because there is no way to verify the go.sum of the go geted module itself, so relying on it would be fairly moot.

[seankhliao]

I believe with the introduction of go install pkg@version and rejection of go get pkg@version outside of a module, the semantics have been clarified to only use SUMDB's checksums (no main module).