cmd/go: tries to download v2.2.2 (no suffix) when checking a replaced v2.2.2+incompatible for a missing import

[leighmcculloch]

What version of Go are you using (go version)?

$ go version
go version go1.13rc1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/home/leighmcculloch/local/bin"
GOCACHE="/home/leighmcculloch/.cache/go-build"
GOENV="/home/leighmcculloch/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/leighmcculloch/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/leighmcculloch/local/bin/go/1.13rc1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/leighmcculloch/local/bin/go/1.13rc1/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/leighmcculloch/devel/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build816830480=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I run go mod tidy on the following code.

go.mod:

module test

go 1.13

require (
        github.com/throttled/throttled v2.2.2+incompatible
)

replace github.com/throttled/throttled v2.2.2+incompatible => github.com/bartekn/throttled v2.2.3-0.20181102161246-c99eef3ad70a+incompatible

main.go:

package main

import (
        "fmt"

        "github.com/throttled/throttled"
)

func main() {
        fmt.Println(throttled.HTTPRateLimiter{})
}

What did you expect to see?

I expected go mod tidy to exit status code of success (0).

$ go mod tidy
$ echo $?
0

What did you see instead?

The go mod tidy command had a failure exit status code (1) and a 410 Gone error message.

$ go mod tidy
go: downloading github.com/throttled/throttled v2.2.2
verifying github.com/throttled/[email protected]: github.com/throttled/[email protected]: reading https://sum.golang.org/lookup/github.com/throttled/[email protected]: 410 Gone
$ echo $?
1

If I visit the URL https://sum.golang.org/lookup/github.com/throttled/[email protected] it returns the text:

not found: invalid version

If I remove the replace directive, go mod tidy returns success and there is no verifying error message. The output in the section above where the exit status is 0 is what I see when the replace statement is removed. The problem I'm experiencing is isolated to when the replace directive is in use.

The version of the throttled package is v2.2.2+incompatible and so if the tool wants to query the sumdb about it, it should be attempting to query the URL https://sum.golang.org/lookup/github.com/throttled/[email protected]+incompatible, but for some reason it is attempting to go to the URL without the +incompatible suffix.

It also seems strange that it is attempting to download the source package when I have a replace directive present pointing to the fork. Possibly it shouldn't be doing that either?

Context

The above example is a reduced example of an error I'm seeing while I'm trying to migrate a larger Go repo from dep to Modules (stellar/go#1634).

[FiloSottile]

/cc @bcmills

[leighmcculloch]

I tried this on Go 1.12.9, and the go mod tidy command works as expected there, so this only seems to be broken on Go 1.13rc1. I assume that's because in 1.12 there is no call out to the sumdb.

[bcmills]

When you say that go mod tidy “works as expected” in Go 1.12.9, did it actually modify the go.mod file, or did it fail silently (#27063)?

[bcmills]

For reference, here is what @heschik and I observe with GOSUMDB=off:

example.com$ GOSUMDB=off go mod tidy
go: downloading github.com/throttled/throttled v2.2.2
go: extracting github.com/throttled/throttled v2.2.2
go: test imports
        github.com/throttled/throttled tested by
        github.com/throttled/throttled.test imports
        github.com/throttled/throttled/store: looping trying to add package

It's trying to resolve github.com/throttled/throttled v2.2.2 because it sees the v2.2.2 tag upstream and for some reason doesn't realize that it should be +incompatible.

I see the same problem (albeit with a worse diagnostic) using go1.12.9:

example.com$ go1.12.9 mod tidy
go: import "test" ->
        import "github.com/throttled/throttled" ->
        test ->
        import "github.com/throttled/throttled/store": looping trying to add package

[leighmcculloch]

Hmm, trying with a clean modcache in a new container I'm now seeing the exact same error as you when running with go1.12.9. Possibly something in my environment state was causing this to not to happen before? I no longer have the container I was running those tests in unfortunately to inspect it.

It does seem like somewhere the +incompatible is being lost though, even in earlier versions of Go.

[heschi]

From what I can see, if you start with the go.mod as specified in the original bug, you get one successful tidy, and then after that the error.

FTR, the proximate cause of the problem seems to be that tidy is looking at upstream throttled, which has tests that reference packages (store and children) that are deleted in the replace.

[leighmcculloch]

The full output of what I see when I run go1.12.9 mod tidy in a new container with the files in the pr description is:

$ go1.12.9 mod tidy                                                                                                                                                                                                                                                                       
go: finding github.com/bartekn/throttled v2.2.3-0.20181102161246-c99eef3ad70a+incompatible
go: downloading github.com/bartekn/throttled v2.2.3-0.20181102161246-c99eef3ad70a+incompatible
go: extracting github.com/bartekn/throttled v2.2.3-0.20181102161246-c99eef3ad70a+incompatible
go: downloading github.com/throttled/throttled v2.2.2
go: finding github.com/hashicorp/golang-lru v0.5.3
go: downloading github.com/hashicorp/golang-lru v0.5.3
go: extracting github.com/throttled/throttled v2.2.2
go: extracting github.com/hashicorp/golang-lru v0.5.3
go: finding github.com/throttled/throttled v2.2.2
go: import "test" ->
        import "github.com/throttled/throttled" ->
        test ->
        import "github.com/throttled/throttled/store": looping trying to add package

Doing same on go1.13rc1, first time I see:

$ go1.13rc1 mod tidy 
go: downloading github.com/bartekn/throttled v2.2.3-0.20181102161246-c99eef3ad70a+incompatible
go: extracting github.com/bartekn/throttled v2.2.3-0.20181102161246-c99eef3ad70a+incompatible
go: downloading github.com/throttled/throttled v2.2.2
go: finding github.com/hashicorp/golang-lru v0.5.3
go: downloading github.com/hashicorp/golang-lru v0.5.3
go: extracting github.com/hashicorp/golang-lru v0.5.3
verifying github.com/throttled/[email protected]: github.com/throttled/[email protected]: reading https://sum.golang.org/lookup/github.com/throttled/[email protected]: 410 Gone

Noteworthy is the line in both outputs that references github.com/throttled/throttled with version v2.2.2 that doesn't have +incompatible.

@heschik I don't see success on the first attempt like you, with either go1.12.9 or go1.13rc1.

[heschi]

Sorry -- with GOSUMDB=off the first succeeds, then I get the loopy error on the second.

[bcmills]

Found the bug. It's another variation on #32700.

[gopherbot]

Change https://golang.org/cl/212200 mentions this issue: cmd/go: avoid erroneous canonicalization when trying to resolve imports using replacements

[bcmills]

With the fix, I get a successful go mod tidy with upgraded versions:

test$ go mod tidy
go: downloading github.com/throttled/throttled v1.0.0
go: downloading github.com/throttled/throttled v2.2.4+incompatible
go: finding versions for github.com/throttled
go: finding versions for github.com/throttled/throttled/store
go: finding versions for github.com/throttled/throttled/store/memstore
go: found github.com/throttled/throttled/store in github.com/throttled/throttled v2.2.4+incompatible
go: downloading github.com/gomodule/redigo v1.7.0
go: downloading github.com/gomodule/redigo v2.0.0+incompatible
go: found github.com/gomodule/redigo/redis in github.com/gomodule/redigo v2.0.0+incompatible

test$

[bcmills]

This seems like a small and obvious enough fix to make Go 1.14. The bug has been present since CL 152739, but was masked by the lack of error reporting in go mod tidy in 1.12.

[gopherbot]

Change https://golang.org/cl/212201 mentions this issue: cmd/go: diagnose missing replacement directories