proxy.golang.org: accidentally publishing a major version interplays poorly with Go modules

[marten-seemann]

What version of Go are you using (go version)?

$ go version
go version go1.13 darwin/amd64

Does this issue reproduce with the latest release?

Yes. It only occurs with Go 1.13.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GONOPROXY=""
GONOSUMDB=""
GOSUMDB="sum.golang.org"

What did you do?

When publishing a new tag on Github, the proxy caches all available module versions and their respective checksums, as soon as it receives a request for the module. When go getting the module, go get will download the latest version available (which is determined according to semantic versioning).

If you accidentally publish a patch tag or a minor tag, this is easy to fix. Just publishing another patch release ensures that all downstream users will receive a working version of the module.

If you accidentally publish a major tag though, this is a practically unrecoverable situation: Since the introduction of Go modules, major versions include the version number in the import path. Now every single downstream user has to adjust the import path to receive a working version of the module.

This is what happened to us with go-libp2p. Before the introduction of Go modules, we used a hash-based package manager called gx for the libp2p project. Releases were identified by their hash value, but we nevertheless assigned version numbers, in order to increase human readability. When we switched to Go modules, we deleted all old tags, and started anew from v0.0.1. A few weeks ago, someone apparently accidentally pushed the old gx-tags back to our Github repo. We detected this within a few hours and removed the tags, but it was already too late: the Go checksum database already archived them for all eternity.
Running go get github.com/libp2p/go-libp2p now retrieves v6.0.23+incompatible instead of the actual current version v0.3.1.

What did you expect to see?

It seems like we're encountering an unfortunate interplay of the way Go handles major version updates (by including them in the import path) with the now irreversible one-step process (pushing a tag) of creating a new release.

Replacing the code associated with a given version is exactly what the Go checksum database is supposed to prevent. However, programmers are humans, and humans make mistakes, sometimes stupid ones like pushing an erroneous major version tag. Relying on a one-step process to create new releases and push them to a fleet of proxies, with no way of managing or reverting them seems like a brittle process.
In practice, many developers use GitHub. Unfortunately, GitHub doesn't support rejecting tag pushes, so it's not even possible to implement an automated sanity check for tag pushes.

I suggest that there needs to be some way to recover from such a mistake without forcing all downstream users to adjust their import paths. Maybe it would be possible to mark a certain version as known broken. The entry would still remain in the checksum database, but go get would ignore this version when determining which version is latest.

[ALTree]

Thanks for reporting this. It appears to me that the only sensible solution to this issue is indeed the one you propose:

there needs to be some way to recover from such a mistake without forcing all downstream users to adjust their import paths. Maybe it would be possible to mark a certain version as known broken.

But as it stands, you're basically asking for #24031 (cmd/go: allow package authors to mark older package versions as insecure, incompatible, or broken), so this issue is essentially a dup of that one.

[ALTree]

(you may want to add this use case to the discussion there).

[raulk]

@ALTree I think this issue is trying to get to the heart of a larger problem.

• Imagine you intended to push a v0.2.0 tag, but instead accidentally pushed v2.0.0.
• The bogus v2.0.0 gets cached by https://proxy.golang.org, and there's no way to revert it.
• Due the design decision to annex > v1 versions in import paths, and the implicit coupling between import paths and version management, you've now forced all downstream users to modify the import paths in all their sources.

We are all humans, and mistakes like the above happen. It's bound to happen in the GitHub Octoverse, due to the lack of tag protection.

The proxy you operate is like an unforgiving watchdog. There should be some kind of workflow to approve/retract caching of releases in the proxy. There's a reason why all package managers out there have a management user interface or explicit steps for release publishing (npm, bintray, maven, cargo).

I know the go team likes to be minimalistic, but disregarding the human factor is a footgun.

Something as simple as adding an "authors" directive in the go.mod file with a list of email addresses, and having the proxy send a confirmation email to approve new releases for proxy caching, would go a long way.

But ideally, you'd offer explicit management over releases, remove the proxy default in go1.13.1 (this was a questionable idea to begin with), and/or make proxy.golang.org caching opt-in.

[aofei]

• Imagine you intended to push a v0.2.0 tag, but instead accidentally pushed v2.0.0.
• The bogus v2.0.0 gets cached by https://proxy.golang.org, and there's no way to revert it.
• Due the design decision to annex > v1 versions in import paths, and the implicit coupling between import paths and version management, you've now forced all downstream users to modify the import paths in all their sources.

I have also imagined this scenario. I'm thinking, can we provide a "withdrawal" option? As long as someone can prove that he/she is the owner of a module version, then he/she can withdraw it by sending a request email to the [email protected]. Or, submit an issue with a prefix like "proxy.golang.org: withdrawal: ".

/cc @katiehockman here.

[hyangah]

@jayconrod @bcmills

I hoped the go command wouldn't pick the incompatible versions when there are matching, valid pseudo versions and the module major version in go.mod does not match. But that's not the case and indeed the go picks up the incompatible version.

go list --versions -json -m github.com/libp2p/go-libp2p go: finding github.com/libp2p/go-libp2p v6.0.23+incompatible { "Path": "github.com/libp2p/go-libp2p", "Version": "v6.0.23+incompatible", "Versions": [ "v0.0.1", "v0.0.2", "v0.0.3", "v0.0.4", "v0.0.5", "v0.0.6", "v0.0.7", "v0.0.8", "v0.0.9", "v0.0.10", "v0.0.11", "v0.0.12", "v0.0.13", "v0.0.14", "v0.0.15", "v0.0.16", "v0.0.17", "v0.0.18", "v0.0.19", "v0.0.20", "v0.0.21", "v0.0.22", "v0.0.23", "v0.0.24", "v0.0.25", "v0.0.26", "v0.0.27", "v0.0.28", "v0.0.29", "v0.0.30", "v0.1.0", "v0.1.1", "v0.1.2", "v0.2.0", "v0.2.1", "v0.3.0", "v0.3.1", "v1.0.0", "v2.0.1+incompatible", "v2.0.2+incompatible", "v2.0.3+incompatible", "v3.0.0+incompatible", "v3.1.0+incompatible", "v3.2.0+incompatible", "v3.2.1+incompatible", "v3.2.2+incompatible", "v3.2.3+incompatible", "v3.3.0+incompatible", "v3.3.1+incompatible", "v3.3.2+incompatible", "v3.3.3+incompatible", "v3.3.4+incompatible", "v3.3.6+incompatible", "v3.3.7+incompatible", "v3.4.0+incompatible", "v3.4.1+incompatible", "v3.4.2+incompatible", "v3.4.3+incompatible", "v3.5.0+incompatible", "v3.5.1+incompatible", "v3.5.2+incompatible", "v3.5.3+incompatible", "v3.5.4+incompatible", "v3.6.0+incompatible", "v4.0.0+incompatible", "v4.0.1+incompatible", "v4.0.2+incompatible", "v4.0.3+incompatible", "v4.0.4+incompatible", "v4.1.0+incompatible", "v4.2.0+incompatible", "v4.3.0+incompatible", "v4.3.1+incompatible", "v4.3.2+incompatible", "v4.3.3+incompatible", "v4.3.4+incompatible", "v4.3.5+incompatible", "v4.3.6+incompatible", "v4.3.7+incompatible", "v4.3.8+incompatible", "v4.3.9+incompatible", "v4.3.10+incompatible", "v4.3.11+incompatible", "v4.3.12+incompatible", "v4.4.0+incompatible", "v4.4.1+incompatible", "v4.4.2+incompatible", "v4.4.3+incompatible", "v4.4.4+incompatible", "v4.4.5+incompatible", "v4.5.0+incompatible", "v4.5.1+incompatible", "v4.5.2+incompatible", "v4.5.3+incompatible", "v4.5.4+incompatible", "v4.5.5+incompatible", "v5.0.0+incompatible", "v5.0.1+incompatible", "v5.0.2+incompatible", "v5.0.3+incompatible", "v5.0.4+incompatible", "v5.0.5+incompatible", "v5.0.6+incompatible", "v5.0.7+incompatible", "v5.0.8+incompatible", "v5.0.9+incompatible", "v5.0.10+incompatible", "v5.0.11+incompatible", "v5.0.12+incompatible", "v5.0.13+incompatible", "v5.0.14+incompatible", "v5.0.15+incompatible", "v5.0.16+incompatible", "v5.0.17+incompatible", "v5.0.18+incompatible", "v5.0.19+incompatible", "v5.0.20+incompatible", "v5.0.21+incompatible", "v6.0.0+incompatible", "v6.0.1+incompatible", "v6.0.2+incompatible", "v6.0.3+incompatible", "v6.0.4+incompatible", "v6.0.5+incompatible", "v6.0.6+incompatible", "v6.0.7+incompatible", "v6.0.8+incompatible", "v6.0.9+incompatible", "v6.0.10+incompatible", "v6.0.11+incompatible", "v6.0.12+incompatible", "v6.0.13+incompatible", "v6.0.14+incompatible", "v6.0.15+incompatible", "v6.0.16+incompatible", "v6.0.17+incompatible", "v6.0.18+incompatible", "v6.0.19+incompatible", "v6.0.20+incompatible", "v6.0.21+incompatible", "v6.0.22+incompatible", "v6.0.23+incompatible" ], "Time": "2018-10-24T21:56:21Z" }

The module mirror and the go command may offer withdrawal or blacklisting options in the context of #24031 as @ALTree said, but removing and reclaiming the published version tags from the global check sum database will not be possible or desirable by design.

[hyangah]

@heschik noticed https://proxy.golang.org/github.com/libp2p/go-libp2p/@v/v6.0.23+incompatible.zip doesn't include go.mod file, which explains why the go command accepted the incompatible version. Since the tag was deleted from the origin, I was confused.

[heschi]

To spell out a factual point:

Due the design decision to annex > v1 versions in import paths, and the implicit coupling between import paths and version management, you've now forced all downstream users to modify the import paths in all their sources.

This is incorrect: v6.0.23 doesn't have a go.mod (as above) and so it's +incompatible, which means that it has the same import path as v0/v1, i.e. github.com/libp2p/go-libp2p. If it did have a go.mod, it would either be invalid by declaring the wrong module path, and therefore not appear in the proxy, or it would be a different module, github.com/libp2p/go-libp2p/v6, which would not be proposed as a candidate to satisfy the import github.com/libp2p/go-libp2p. See https://github.com/golang/go/wiki/Modules#can-a-module-consume-a-v2-package-that-has-not-opted-into-modules-what-does-incompatible-mean for more information.

So, if you want, you can fix/work around this right now (regardless of any decision about removing content from proxy.golang.org) by publishing a later tag, v6.0.24 or v7.0.0 or such, that points to your current code. Maybe that sends the wrong message to your users, although deleting all tags from a repository also seems kind of confusing.

Note that there are other proxies in the world. It appears that gocenter.io has already picked up many of these versions, for example.

[thepudds]

@raulk

Due the design decision to annex > v1 versions in import paths, and the implicit coupling between import paths and version management, you've now forced all downstream users to modify the import paths in all their sources

I think I follow the general concern expressed in this issue, but just on that one point I quoted here -- to my knowledge, accidentally publishing a version where the sole mistake is incorrectly applying VCS tags with a higher-than-intended major version would not require downstream consumers to modify import paths, at least not to my knowledge.

Using the example from this issue, go get github.com/libp2p/go-libp2p retrieving v6.0.23+incompatible means consumers would be using the github.com/libp2p/go-libp2p import path (note no trailing /v6 in the import path there).

In general, a module can import a v2+ package that has not opted into modules, and if the imported v2+ package has a valid semver tag, it will be recorded with an +incompatible suffix, and the consuming module does not use /vN in import paths.

[raulk]

@heschik

So, if you want, you can fix/work around this right now (regardless of any decision about removing content from proxy.golang.org) by publishing a later tag, v6.0.24 or v7.0.0 or such, that points to your current code.

All downstream users would be forced to modify their import paths by suffixing v6 or v7 -- all due to a human error and the inexistence of checks or confirmation before perpetuating something in history. The DX here is terrible.

Maybe that sends the wrong message to your users, although deleting all tags from a repository also seems kind of confusing.

The nuance here is that these tags were created before Go modules existed, and before those major versions had any implications on import paths. So when we introduced go.mod, we archived those "ignorant" tags under the gx/ namespace, and started afresh from v0.

[raulk]

@thepudds

I think I follow the general concern expressed in this issue [...]

👍

a module can import a v2+ package that has not opted into modules, and if the imported v2+ package has a valid semver tag, it will be recorded with an +incompatible suffix, and the consuming module does not use /vN in import paths.

The scenario I'm illustrating is one where the project does use go.mod and a developer accidentally pushes a tag that sends the major version above v1. It's a pretty low bar to apocalipsis.

[thepudds]

The scenario I'm illustrating is one where the project does use go.mod and a developer accidentally pushes a tag that sends the major version above v1. It's a pretty low bar to apocalipsis.

To my knowledge, I think cmd/go will not allow you to get a version that is mismatched in the way that you described (at least as of Go 1.13) with a go.mod that reads module example.com/foo (without a trailing /vN or /v2 for example) but the VCS tags is v2.0.0 (or some other vN.x.y) with the end result being a user is forced to use a /vN or /v2 in the import path.

[hyangah]

@heschik

So, if you want, you can fix/work around this right now (regardless of any decision about removing content from proxy.golang.org) by publishing a later tag, v6.0.24 or v7.0.0 or such, that points to your current code.

All downstream users would be forced to modify their import paths by suffixing v6 or v7 -- all due to a human error and the inexistence of checks or confirmation before perpetuating something in history. The DX here is terrible.

I don't think that is true. If you publish with a later tag after deleting go.mod from your repo, those versions will be recognized as '+incompatible' versions, not "v6.0.24" or "v7.0.0". Users can access that withthem without v6 or v7 suffixed import paths.

If you decide to keep go.mod in the repo, any further incompatible version tags (major version above v1) will be rejected by go commands. I am not sure how go get and go list cope with those incompatible, but vN (N>1) suffixed versions included in the go proxies' resutls or client-side module cache already.

[thepudds]

To my knowledge, I think cmd/go will not allow you to get a version that is mismatched in the way that you described (at least as of Go 1.13) with a go.mod that reads module example.com/foo (without a trailing /vN or /v2 for example) but the VCS tags is v2.0.0 (or some other vN.x.y) with the end result being a user is forced to use a /vN or /v2 in the import path.

Here is a concrete example of the Go 1.12 version of cmd/go rejecting a mismatch in that scenario, where the lz4 go.mod does not have a trailing /v2, the lz4 vcs tag is v2.x.y, and the consumer's import path (or require) tries to use /v2:

# error if asked for v2.1.1 using /v2
require github.com/pierrec/lz4/v2 v2.1.1
go: github.com/pierrec/lz4/[email protected]: 
 go.mod has non-.../v2 module path "github.com/pierrec/lz4" (and .../v2/go.mod does not exist) at revision v2.1.1
go: error loading module requirements

# error if asked for latest using /v2
require github.com/pierrec/lz4/v2 latest
go: github.com/pierrec/lz4/[email protected]: 
 go.mod has non-.../v2 module path "github.com/pierrec/lz4" (and .../v2/go.mod does not exist) at revision v2.1.2
go: error loading module requirements

[raulk]

@hyangah

I don't think that is true. If you publish with a later tag after deleting go.mod from your repo, those versions will be recognized as '+incompatible' versions, not "v6.0.24" or "v7.0.0". Users can access that with v6 or v7 suffixed import paths.

If the last statement were true, we wouldn't have much of an issue here. Unfortunately, that's not the case. Those versions are accessible without version suffixing. A user importing go-libp2p (without version suffixes) in a fresh project will get v6.0.23+incompatible if they're using Google's proxy (which is enabled by default), because that's the "latest" version there.

[hyangah]

@raulk sorry that's my typo. I meant "Users can access that without v6 or v7 suffixed import paths". Actually, users shouldn't use those suffixes to pick up the intended version.

[jayconrod]

As I understand, the issue is that v6.0.23 (and other accidentally published versions) are considered part of the v0/v1 module since no go.mod file was present in those commits. Users running go get -u will upgrade to these automatically, and you can't publish v6.0.24 because you have a go.mod file now, and v6.0.24 isn't allowed without adding a /v6 suffix to the module path (and requiring users to do the same with their imports).

I think #24031 is probably the long-term solution for this. If that were implemented, you'd be able to tell proxies to stop advertising "bad" versions via the <modpath>/@v/list and <modpath>/@latest endpoints. The go command would also not consider these versions when choosing a new version with GOPROXY=direct. Users would not automatically upgrade to bad versions, e.g., with go get -u. However, modules that specifically reference "bad" versions would still build, since the bad versions would still be available, just not advertised. "Bad" version tags could not be recycled.

In this situation, you could mark all of the accidentally published versions as "bad", then you could continue publishing v1.x.y versions, and users would upgrade to those automatically instead of v6.0.23+incompatible.

Would this solve the problem for you?

[bcmills]

you could mark all of the accidentally published versions as "bad", then you could continue publishing v1.x.y versions, and users would upgrade to those automatically instead of v6.0.23+incompatible.

If there are a large number of intervening commits, you would also need to invalidate all pseudo-versions generated for those commits using the erroneous tag as the pseudo-version base. (But presumably you could identify at least the majority of those using index.golang.org.)

[bcmills]

Note that (at least with Go 1.13) this problem can only occur for versions of the module that lack an explicit go.mod file. If a go.mod file had been present, the go command would have rejected the v6.0.23 tag outright (rather than rewriting it to v6.0.23+incompatible).

In retrospect, I think we should not have allowed +incompatible versions in the first place — we should have instead rewritten them to path-appropriate pseudo-versions. Unfortunately, we can't feasibly change that now.

However, since we know that major-version bumps represent breaking changes, perhaps we can change the behavior of go get so that it does not upgrade from v0.*.* or v1.*.* to vN.*.*+incompatible for N > 1.

That wouldn't help much if users already explicitly required vN.*.* versions, but it would at least allow module owners to resume the v0 or v1 line as a module after having previously published incompatible versions (intentionally or otherwise) at the same path.

[bcmills]

Another option might be to declare +incompatible versions a mistake, and do our best to try to purge them from the module graph.

For example: we know that a +incompatible version cannot have a go.mod file, and therefore cannot impose any transitive requirements on other modules. So if the main module declares go 1.14 or higher, we could ignore +incompatible versions entirely during loading and version selection, and rewrite incompatible versions passed to go get to pseudo-versions (with compatible major versions) instead of retaining them with a +incompatible suffix.

...come to think of it, I like that idea a lot. I think I'll file a proposal!

[thepudds]

However, since we know that major-version bumps represent breaking changes, perhaps we can change the behavior of go get so that it does not upgrade from v0.*.* or v1.*.* to vN.*.*+incompatible for N > 1.

If that is done, it would probably also make sense to not upgrade from vN.*.*.+incompatible to vN+1.*.*.+incompatible for N > 1. (Maybe this was implied by what you wrote).

[bcmills]

Agreed — that was my intent.

[bcmills]

Filed my proposal as #34217. I believe it would address this issue as well, by allowing users to ignore the accidental +incompatible versions and proceed on the v0 or v1 line.

(An accidental v1.0.0 release would still be somewhat problematic, but it would at least mitigate the v6 problem.)

[gopherbot]

Change https://golang.org/cl/204440 mentions this issue: cmd/go: avoid upgrading to +incompatible versions if the latest compatible one has a go.mod file

[gopherbot]

Change https://golang.org/cl/204439 mentions this issue: cmd/go/internal/modfetch: prune +incompatible versions more aggressively

[bcmills]

#33558 (comment) is an interesting example here. There was a branch named v2.0.0, but no such tag, but the go command nonetheless picked up a v2.0.0+incompatible version and the proxy cached it.

Probably we should not treat branches as semantic versions. One may (very reasonably) expect to add further commits on the branch, and those commits would make the meaning of v2.0.0 ambiguous.

[bcmills]

Filed #35671 for treating the branch as a release in the first place.

[ferhatelmas]

(An accidental v1.0.0 release would still be somewhat problematic, but it would at least mitigate the v6 problem.)

@bcmills here is an example for v1.0.0 https://pkg.go.dev/mod/go.opentelemetry.io/otel/exporter/trace/[email protected]?tab=versions

As a summary, no workaround other than bumping to v1.0.0+, right?

[lizthegrey]

I'm wondering whether it might be a good idea, instead of allowing versions to be mutated, to instead have package owners be able to publish a badlist of release tags that should not be used for purposes of go get -u's latest computation. This avoids the issue of rewriting history, while still allowing package owners to mitigate creating bad user experiences.

... oh that's #24031, carry on ;)

[ianlancetaylor]

Is there something to do here for 1.14?

[bcmills]

The parts that were feasible for 1.14 are done.

[marten-seemann]

While the fixes for 1.14 solve the immediate problem for libp2p that I described in #34189 (comment), the problem that an accidental tagging of a major version cannot be undone (as it happened to OpenTelemetry) is still unresolved.

[bcmills]

@ferhatelmas, @marten-seemann: that is correct. The OpenTelemetry problem has no workaround in Go 1.14 or earlier. #24031 may provide a solution for similar cases in the longer term.

[bcmills]

The go get behavior is basically fixed by the retract directive added in Go 1.16 (#24031), and the combination of the exclude directive fixes in Go 1.16 (see https://go.dev/doc/go1.16#go-command) and module graph pruning (https://go.dev/ref/mod#graph-pruning) should allow users to gradually prune out any spurious dependencies on erroneously-tagged versions.