cmd/go: building requires otherwise-unused .mod files to be present

[stapelberg]

What version of Go are you using (go version)?

$ go version
go version go1.12beta1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/michael/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/gp-repro"
GOPROXY=""
GORACE=""
GOROOT="/home/michael/sdk/go1.12beta1"
GOTMPDIR=""
GOTOOLDIR="/home/michael/sdk/go1.12beta1/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/repro/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build500484946=/tmp/go-build -gno-record-gcc-switches"

What did you do?

% export GOPATH=/tmp/gp-repro
% mkdir /tmp/repro
% cat > repro.go <<'EOT'
package main

import (
	"fmt"

	"golang.org/x/exp/mmap"
	"gonum.org/v1/gonum/unit"
)

func main() {
	fmt.Printf("%v, %v", mmap.ReaderAt{}, unit.Meter)
}
EOT
% go mod init repro
% go build
go: finding gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e
go: finding golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: finding golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
go: finding golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b
go: downloading gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e
go: downloading golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: extracting golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: extracting gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e

As you can see, two versions of golang.org/x/exp are involved here: the repro module uses the latest (v0.0.0-20181221233300-b68661188fbf), whereas the gonum.org/v1/gonum module specifies an older version (v0.0.0-20180321215751-8460e604b9de).

In the build itself, the latest version is picked:

% go list -m -json all
{
	"Path": "repro",
	"Main": true,
	"Dir": "/tmp/repro",
	"GoMod": "/tmp/repro/go.mod",
	"GoVersion": "1.12"
}
{
	"Path": "golang.org/x/exp",
	"Version": "v0.0.0-20181221233300-b68661188fbf",
	"Time": "2018-12-21T23:33:00Z",
	"Dir": "/tmp/gp-repro/pkg/mod/golang.org/x/[email protected]",
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20181221233300-b68661188fbf.mod"
}
{
	"Path": "golang.org/x/tools",
	"Version": "v0.0.0-20180525024113-a5b4c53f6e8b",
	"Time": "2018-05-25T02:41:13Z",
	"Indirect": true,
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/golang.org/x/tools/@v/v0.0.0-20180525024113-a5b4c53f6e8b.mod"
}
{
	"Path": "gonum.org/v1/gonum",
	"Version": "v0.0.0-20181223131727-b545e3e77e9e",
	"Time": "2018-12-23T13:17:27Z",
	"Dir": "/tmp/gp-repro/pkg/mod/gonum.org/v1/[email protected]",
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/gonum.org/v1/gonum/@v/v0.0.0-20181223131727-b545e3e77e9e.mod"
}

However, when preventing internet access by pointing GOPROXY to an unused port, I can see that the old version of golang.org/x/exp needs to be present for the build to succeed:

% rm repro && GOPROXY=http://localhost:1111/ go build
# works
% rm /tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.*
% rm repro && GOPROXY=http://localhost:1111/ go build
go: finding golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
go: golang.org/x/[email protected]: Get http://localhost:1111/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.info: dial tcp [::1]:1111: connect: connection refused
go: error loading module requirements
% echo 'module golang.org/x/exp' > /tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.mod
% GOPROXY=http://localhost:1111/ go build                                                                                         
# works

This came as a surprise to me, so I wanted to check in and see if this was intentional or an oversight. Given that the .mod file only contains “module golang.org/x/exp”, it seems to me that the file could easily be synthesized?

I discovered this issue because I was experimenting with packaging Go modules in Linux distributions (e.g. Debian). It would be a significant downside if we needed to provide (and therefore maintain!) all versions that a build ever references, instead of just the versions which are selected in the end.

Thanks,

[mvdan]

It might take a while for this to get a proper reply, as most engineers will be on holiday this week :)

/cc @bcmills @myitcv @thepudds

[thepudds]

@stapelberg two quick comments:

1. you might want to set GOPROXY=off (rather than using a non-existent port). That means go commands in module mode are disallowed from using the network for dependencies. I don't know if that explains your symptoms (and probably it does not), but you might get more standard behavior in general out of GOPROXY=off.

2. you could try with the go 1.12 beta. I think there was at least one fix in 1.12 for avoiding hitting the network unnecessarily (though it might have had to do with replace, which might not help here).

edit: sorry, I see you are using the 1.12 beta.

[thepudds]

Setting GOPROXY=off and attempting to reproduce gives a slightly different error, but still fails:

$ go1.12beta1 build
go: golang.org/x/[email protected]: module lookup disabled by GOPROXY=off
go: error loading module requirements

Someone else might need to comment if this is a bug, vs. necessary behavior, vs. perhaps a not-yet-implemented feature could avoid this.

@stapelberg you mentioned:

It would be a significant downside if we needed to provide (and therefore maintain!) all versions that a build ever references, instead of just the versions which are selected in the end.

I am just looking at this quickly, but is it just meta data about the old version 20180321215751-8460e604b9de that currently seems to be required? E.g., after a clean build (with GOPROXY unset):

$ ls $GOPATH/pkg/mod/cache/download/golang.org/x/exp/@v                      
list
v0.0.0-20180321215751-8460e604b9de.info
v0.0.0-20180321215751-8460e604b9de.mod
v0.0.0-20181221233300-b68661188fbf.info
v0.0.0-20181221233300-b68661188fbf.mod
v0.0.0-20181221233300-b68661188fbf.zip
v0.0.0-20181221233300-b68661188fbf.ziphash

and:

$ ls $GOPATH/pkg/mod/golang.org/x
[email protected]

or:

$ cd $GOPATH/pkg/mod
$ find -name "*8460e604b9de*"
./cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.info
./cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.mod

[stapelberg]

I am just looking at this quickly, but is it just meta data about the old version 20180321215751-8460e604b9de that currently seems to be required? E.g., after a clean build (with GOPROXY unset):

Yes, it’s just the .mod file. Hence my suspicion that this is unintended.

[bcmills]

This is working as designed. Every go.mod file in the transitive requirement graph — even for versions that we know will not be selected — impacts the constraints on which versions we can ultimately choose, so we have to load those files in order to compute the final versions.

The good news is, go.mod files are small and cacheable, so the impact on build times should usually be negligible.

See the cross test-cases for the MVS package for examples:

go/src/cmd/go/internal/mvs/mvs_test.go

Lines 42 to 116 in 006a5e7

	# Cross-dependency between D and E.
	# No matter how it arises, should get result of merging all build lists via max,
	# which leads to including both D2 and E2.
	name: cross1
	A: B C
	B: D1
	C: D2
	D1: E2
	D2: E1
	build A: A B C D2 E2
	name: cross1V
	A: B2 C D2 E1
	B1:
	B2: D1
	C: D2
	D1: E2
	D2: E1
	build A: A B2 C D2 E2
	name: cross1U
	A: B1 C
	B1:
	B2: D1
	C: D2
	D1: E2
	D2: E1
	build A: A B1 C D2 E1
	upgrade A B2: A B2 C D2 E2
	name: cross1R
	A: B C
	B: D2
	C: D1
	D1: E2
	D2: E1
	build A: A B C D2 E2
	name: cross1X
	A: B C
	B: D1 E2
	C: D2
	D1: E2
	D2: E1
	build A: A B C D2 E2
	name: cross2
	A: B D2
	B: D1
	D1: E2
	D2: E1
	build A: A B D2 E2
	name: cross2X
	A: B D2
	B: D1 E2
	C: D2
	D1: E2
	D2: E1
	build A: A B D2 E2
	name: cross3
	A: B D2 E1
	B: D1
	D1: E2
	D2: E1
	build A: A B D2 E2
	name: cross3X
	A: B D2 E1
	B: D1 E2
	D1: E2
	D2: E1
	build A: A B D2 E2

[stapelberg]

This is working as designed. Every go.mod file in the transitive requirement graph — even for versions that we know will not be selected — impacts the constraints on which versions we can ultimately choose, so we have to load those files in order to compute the final versions.

Can you elaborate on the ways in which these files can impact the version selection? Specifically, are you saying it is not sufficient to synthesize these files?

The good news is, go.mod files are small and cacheable, so the impact on build times should usually be negligible.

That’s not my concern, though. In Linux distributions such as Debian, we strive to have precisely one version of a specific module packaged. We can’t keep old packages around just for their .mod files.

If this is really working as intended with no wiggle room, we’ll need to work around that and either modify .mod files at package-time (seems more labor-intensive, but haven’t thought it through entirely) or synthesizing .mod files at build-time (ugh).

[bcmills]

In Linux distributions such as Debian, we strive to have precisely one version of a specific module packaged. We can’t keep old packages around just for their .mod files.

Sorry, but Go's version selection algorithm intentionally prioritizes reproducibility over recency: when you build a module-enabled binary, you use the specific versions of the modules it was tested against, not whatever newer version happens to be installed on your system.

Now that #27859 is fixed, you can probably make something work using wildcard replace directives, but the resulting builds will not be as reproducible as ordinary module-enabled builds.

[thepudds]

@stapelberg Is this in the context of what I think in the past might have been described as "more or less one giant GOPATH" for Debian packaging of Go libraries, or is this more in the context of trying to use Go modules to break out of that (or perhaps nothing to do with any of that)?

I am not sure how much flexibility you have at package time vs. build time, how that changes for a package vs. a binary, the level of control around setting env variables, or your ability to use an auxiliary directory that contains auxiliary data like go.mod files...

...but depending on your constraints, you might be able to do some variety of using GOPROXY=file:/// and/or temporarily setting GOPATH (which controls the location of GOPATH/pkg/mod cache location). That might not help, but here is a concrete example (from @myitcv):

https://github.com/go-modules-by-example/index/tree/master/012_modvendor

That particular example is showing an alternative to traditional vendoring, and I suspect that is not exactly what you'd want, but it serves as a concrete example of the types of things that are enabled via the flexibility of GOPROXY=file:///.

In any event, good chance GOPROXY=file:/// does not help based whatever your constraints are, but wanted to at least mention it in case it gives you an idea, etc.

[stapelberg]

resulting builds will not be as reproducible as ordinary module-enabled builds.

Debian has its own reproducibility efforts, see https://reproducible-builds.org/. The binaries we build are reproducible, and have been before Go modules :)

@stapelberg Is this in the context of what I think in the past might have been described as "more or less one giant GOPATH" for Debian packaging of Go libraries

Yes. This is the current model and we have no plans to change that.

I am not sure how much flexibility you have at package time vs. build time,

Changes at packaging-time take a long time to trickle down to all our volunteers. Build-time changes are preferable for this particular situation.

how that changes for a package vs. a binary,

No change.

the level of control around setting env variables

We can set arbitrary env variables.

or your ability to use an auxiliary directory that contains auxiliary data like go.mod files...

We can do that, but every step we need to set up introduces additional complexity, which makes builds slower and harder to understand/debug.

...but depending on your constraints, you might be able to do some variety of using GOPROXY=file:/// and/or temporarily setting GOPATH (which controls the location of GOPATH/pkg/mod cache location). That might not help, but here is a concrete example (from @myitcv):

Yeah, we currently already synthesize a GOPATH. From this discussion it seems like we need to synthesize all go.mod files ever referenced by the transitive closure as well. I have not yet looked into how easy or hard this would be to do.

[thepudds]

@stapelberg I am curious what your latest thinking here is, at least briefly? Part of the reason for me bouncing this is curiosity, but also some similar questions are coming up elsewhere.

Also, one specific area I do not understand is why you mention synthesizing a large number of go.mod files. Was it specifically limited to synthesizing older go.mod files that are not going to be selected as the final versions? If so, are you thinking they would be empty aside from the module foo line?

Expanding on an earlier point slightly, rather than synthesizing, you might be able use a filesystem-based module cache containing go.mod files. I suspect this is well known to you at this point, but:

• The module download cache location is controlled by GOPATH. In particular, go mod download, go build, etc. populate the module cache in GOPATH/pkg/mod.
• In addition, when you want to use a particular module cache, you can tell the go command to use a local module cache by setting GOPROXY=file:///file/path.
• You can put those two things together:

   # Populate a module download cache in /tmp/gopath-for-cache
   $ GOPATH=/tmp/gopath-for-cache  go mod download

   # Build using the contents of the module download cache in /tmp/gopath-for-cache
   $ GOPROXY=file:///tmp/gopath-for-cache/pkg/mod/cache/download  go build

In other words, that might be an approach to get the actual go.mod files of interest, rather than synthesizing a large number of older go.mod files.

Note that even though you are setting the GOPROXY environment variable in the steps above, there is no actual proxy process involved, and everything is just being read directly from the local filesystem. An even more detailed example is in the link I had mentioned above.

All that said, I don't know if that set of capabilities really helps you or not, and perhaps synthesizing is easier based on your use case.

[stapelberg]

Thanks for your reply.

Also, one specific area I do not understand is why you mention synthesizing a large number of go.mod files. Was it specifically limited to synthesizing older go.mod files that are not going to be selected as the final versions? If so, are you thinking they would be empty aside from the module foo line?

Yes to both questions.

In other words, that might be an approach to get the actual go.mod files of interest

I don’t follow. Where would they come from? Note that Linux distributions generally prohibit internet access during builds, so all files we want to use need to be shipped in other packages (build dependencies) or created by the build infrastructure.

[thepudds]

@bcmills @jayconrod Based on the conversation in #30831 (comment) and #30831 (comment) and related comments, is there some chance that a missing older go.mod file (e.g., some of the issues described in this issue here) could also be viewed as a non-fatal error if that older version does not end up in the final build list?

Any brief thoughts on maybe re-opening this, even if it might not be a near-term change here, and even if it might later end up getting closed again?

[jayconrod]

@thepudds I'm not sure if I followed completely, but a missing go.mod file should not be a fatal error. If there is no go.mod file in a dependency, the Go command will synthesize a trivial .mod file with just the module path and no requirements. That will end up in the cache and will be served by the proxy.

If a go.mod file is missing in an older version but is present with the wrong module path in a newer version (#30831), that's probably a good sign that we were importing packages with the wrong path. We might not need to know that to suppress the error though.

[stapelberg]

[…] but a missing go.mod file should not be a fatal error. If there is no go.mod file in a dependency, the Go command will synthesize a trivial .mod file with just the module path and no requirements.

The reason why I opened this issue is that I’m seeing the opposite behavior (or I’m misunderstanding you). Can you try to reproduce the steps provided in the initial message please?

[thepudds]

I think there might be two valid but different uses of the word "missing" here.

At least as I understood it:

• I think @jayconrod is referring to the case where a repository foo has not yet opted in to modules, and then when a module-aware consumer imports foo the go command will automatically synthesize the "missing" go.mod file that is not present in repository foo (as described in cmd/go: building requires otherwise-unused .mod files to be present  #29410 (comment)).

• I think @stapelberg is describing a case that arose where he was experimenting on approaches for packaging Go modules in Linux distributions (Debian in particular) where older go.mod files were eliminated from the on-disk module cache to simplify the packaging problem (e.g., rm /tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.*), but then the go command would complain about those old "missing" go.mod files (as seen in cmd/go: building requires otherwise-unused .mod files to be present  #29410 (comment) and the next few comments).

Separately, I had bounced this issue to link it to #30831 (comment). In @stapelberg's case, I suspect the older "missing" go.mod files might not be strictly required, perhaps similar to how in #30831 the old problematic grpc version is not strictly required in the final build list and hence was wondering if a similar solution "treat as a non-fatal error" approach might be possible here.

[stapelberg]

I talked to @bcmills in person at GopherCon just a few minutes ago, and here are the cliff notes of where we ended up with regards to packaging Go modules in a Linux distribution context (should also apply to NetBSD’s pkgsrc etc.):

1. Individual packages can just contain fragments of the module cache. E.g., to package honnef.co/go/tools, you can just persist pkg/mod/honnef.co/go/[email protected] and pkg/mod/cache/download/honnef.co/go/tools/@v/*.{info,mod}. You can obtain these paths programmatically by running go mod download -json.

2. When building, construct a module cache from these fragments and point the Go tool to it by setting e.g. GOPROXY=file:///tmp/gopath/pkg/mod/cache/download

3. To make the Go tool use the versions of packages which are available in the distribution build environment (and not look for any others), you’ll need to add replace directives in the module you’re building (e.g. github.com/junegunn/fzf).

   1. It is better to use the replace directive’s path=path@version form, not the path=/file/path form, because the former transparently works for Go code which does not yet use modules, whereas you would need to synthesize a go.mod file for the latter.

   2. You need to add replace directives for all modules involved in the build. Note that go.mod in the top-level (e.g. fzf) does not necessarily list all modules! go list -m all will, though. Typically you have the transitive closure of dependencies readily available at the distribution package manager level (e.g. Build-Dep statements in Debian).

   3. If you can, specify a version, not just latest. Using latest will work, but requires list files in your module cache so that the go tool can resolve latest to a version. Depending on how you construct your module cache, these files may need to be synthesized: the go mod download -json output does not include them, for example.

Notably, when following step ③, the go tool no longer tries to access the go.mod files, solving this issue.

[thepudds]

@qbit I would be curious if the solution outlined in the above comment (#29410 (comment)) would help in the case of OpenBSD as well.

[qbit]

@thepudds I have prototyped out something using mod download -json, I think it will work for our needs! (going to extend portgen) Thanks for pinging me on this!