cmd/go: go.mod inconsistency between "go mod tidy" and "go build"

[ceejatec]

What version of Go are you using (go version)?

$ go version
go version go1.14.4 linux/amd64

Does this issue reproduce with the latest release?

Yes (confirmed with gotip).

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ceej/.cache/go-build"
GOENV="/home/ceej/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ceej/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ceej/.cbdepscache/exploded/x86_64/go-1.14.4/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ceej/.cbdepscache/exploded/x86_64/go-1.14.4/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/foo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build286401846=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I have a project with two inter-dependent modules. Our overall build checks out both modules into sibling directories, and we use "replace" directives so that the code can find each other. This is necessary because both projects are actively developed and we don't want to have to release/tag each of them every time there's a change to get into a build.

One of those projects also depends on a separate package "go-jsonpointer" which is not yet converted to Go modules, and that package in turn depends on another package "gojson" which is also not yet converted to Go modules.

What did you expect to see?

We expect that "go mod tidy" immediately followed by "go build -mod=read" should succeed. This was confirmed by @jayconrod to be expected behaviour on #31248 .

What did you see instead?

What we see is that "go build" updates go.mod to add "gojson" as an //indirect dependency - this is correct. However, "go mod tidy" removes that entry from go.mod. What this means is that if we run "go mod tidy" (which is normal good practice), then "go build -mod=readonly" will fail with

/home/ceej/go/pkg/mod/github.com/dustin/[email protected]/bytes.go:9:2: cannot find module providing package github.com/dustin/gojson: import lookup disabled by -mod=readonly

Minimal reproducer

I've created a minimal test case which demonstrates this issue, including a script that shows the error when using "go build -mod=readonly": https://github.com/ceejatec/go-mod-bug

You can see the bug by just:

git clone git://github.com/ceejatec/go-mod-bug
cd go-mod-bug
./repro.sh

[ceejatec]

On #29773, @bcmills said that "go mod tidy" will include

• all of the direct dependencies, plus
• the subset of indirect dependencies that have been upgraded past their otherwise-implied versions.

I suspect that this logic is getting thrown off in this case by the fact that "foo" refers to itself using a "replace" directive. However, this replace is necessary given the way our modules are developed. foo depends on bar, while bar depends on foo. bar mentions foo with a replace directive in its own go.mod, but when building foo, only foo's go.mod is referenced. Therefore, for bar's reference to foo to resolve, foo/go.mod has to include a replace directive for itself.

[bcmills]

Duplicate of #34417

[bcmills]

(See specifically #34417 (comment).)

[ceejatec]

Request this ticket be re-opened. It does not describe the same issue as #34417. It is true that if #34417 is acted upon as written, that this ticket will be moot. However I do not think #34417 should be implemented as written, which would make this ticket relevant.

[jayconrod]

@ceejatec I don't understand how this issue is different from #34417. Could you elaborate? In both cases, there are two modules that require each other and are developed together. In both cases, the main module replaces itself, which is not handled correctly by go mod tidy or other commands.

I understand the solutions suggested in #34417 aren't necessarily satisfying, but unless this is a fundamentally different issue, it would be better to focus discussion there rather than splitting the conversation.

[ceejatec]

The reported symptoms are similar although not the same. However the subject line for #34417 says that the goal is to remove the ability to point a "replace" directive at the module directory itself, which is quite at odds with the goal of this ticket.

[jayconrod]

The reported symptoms are similar although not the same.

How are they different?

The original title of that issue was "go.mod changes on each build, -mod=readonly can never work", which sounds like the same symptoms. The repros people have posted seem structurally similar.

[ceejatec]

They are more similar than I thought, true - the modules which are getting removed and re-added in my reproducer are separate third-party ones, but it seems that's also the case on #34417 which wasn't clear from the description.

Again, though, the big difference IMHO is the goal. That ticket is discussing the correctness of removing the feature entirely. Do you believe it is implied that if the feature is NOT removed, then the underlying bugs with "go mod tidy" will be fixed?

[bcmills]

@ceejatec, replacing the selected version of the main module itself is not a “feature” — it is a bug.

You can replace specific older versions of the main module regardless of #34417.

[ceejatec]

It may be a bug, but as these tickets show, it is also a workaround to other bugs, because the go compiler does not honor "replace" directives other than at the top level.

[jayconrod]

Ignoring replace directives outside the main module is not a bug: the ecosystem couldn't scale if there were lots of modules with contradictory replace directives.

In any case, if you disagree with the approach in #34417, please explain further in #34417. I'd rather not split the discussion further.