cmd/go: more graceful upgrades when a module changes its name #30831
Comments
|
My theory (sorry, it's late, so not sure if it's obviously right or wrong... also, this bug report is poorly written but I'm pretty frustrated and confused by all of this) Because we are allowed circular dependencies on modules, we have two libraries: [email protected] requires github.com/golang/lint [email protected] requires [email protected] A fixes its import to golang.org/x/lint, tags 0.2 [email protected] updates its A dependency, and now requires [email protected]. But [email protected] still requires [email protected] which requires [email protected] which requires [email protected]... which requires github.com/golang/lint. Of course, due to MVS, we only choose one version of the dependency at build time. However, So the only to break this cycle is to simultaneously tag new versions of A and B, and have them require each other. (incompatible with CI) In reality, we have more than just two things depending on each other, there's far more. golang.org/x/oauth2, google.golang.org/api/grpc, cloud.google.com/go, google.golang.org/api, go.opencensus.io basically all have dependencies on each other. |
|
/cc @bcmills We've discussed this general issue today (or yesterday) and Bryan said he might be aware of a fix that can be applied to |
|
Something along the lines of this, maybe? (see patch below) The idea is to not try to upgrade something when we already use a newer version. Theory: upgrading from 0.2 to 0.3 is less work than 0.1 to 0.3, and there's no need to check the same package if you already checked a newer version. edit: ah, yes, theory and tests fail because vN+1 may have downgraded one of its deps. @@ -85,6 +89,17 @@ func buildList(target module.Version, reqs Reqs, upgrade func(module.Version) mo
)
work.Do(10, func(item interface{}) {
m := item.(module.Version)
+
+ var skip bool
+ mu.Lock()
+ if v, ok := min[m.Path]; ok && v != "" && reqs.Max(v, m.Version) == v {
+ skip = true
+ }
+ mu.Unlock()
+ if skip {
+ return
+ }
+
required, err := reqs.Required(m)
mu.Lock()Simplified problem... M has a requirement on [email protected] and [email protected] "go get -u" still tries to upgrade C, because of M's dependency on [email protected], even though M won't use [email protected] or [email protected] in the build at all, only [email protected] and [email protected]. Another ecosystem problem is "indirect" entries in go.mod. As evidenced by zipkin including an indirect entry in github.com/golang/lint: [email protected] requires [email protected] M requires [email protected] and [email protected], indirectly depending on [email protected] even though it isn't used to build, because MVS upgrades B's [email protected] requirement to [email protected]. In this case, A = grpc or genproto, B = zipkin, C = github.com/golang/lint |
broady
changed the title
bcmills
added
the
NeedsFix
|
FWIW, I also suspect (a) something is wrong here, or (b) that the behavior here can be improved. Potentially related is #30455 (comment), which asks if
Also potentially related:
|
rsc
changed the title
|
Leaving the "too much upgrading" problem for #26902. It seems like we need a clear signal from the upgraded module that "I used to be this old name". And then if you are resolving the old name and land in the new module claiming "I used to be this old name", the build would automatically interpret the old name as the new name. Maybe that's a per-module signal; maybe it's a per-package signal. It needs to be an explicit signal, though. It's not enough to just "find code at X that seems to be module Y so pretend X means Y" because that breaks people who github fork X into their own account as Y. In that case we don't want go get of the fork to silently redirect back to the original. |
This requires the upgraded module to know its old name. In this specific case, it might be possible, because the canonical import path of lint used to be However, it didn't start enforcing import paths until much later (via import comment for GOPATH mode, and via go.mod file for module mode), so some projects may have started importing it via any other alias, not just Should it be responsibility of an upgraded module to know what were all the non-canonical import paths that some projects in the wild may have imported it as? |
In general that set is well-defined: we must have arrived at the repository somehow (via redirects), so the only possible names for it are things that redirect to it. I would guess that it's pretty rare for folks to set up redirectors for arbitrary repositories just to use them via different import paths. |
|
One short description of the build list (from the cmd/go documentation) is:
It is unlikely that I will describe this exactly correctly, but from the outside it seems that if the inconsistency between Or maybe that is not the right way to think about it -- maybe instead the "bad" consumer version gets added to the build list but is marked that it is bad, but that is not a fatal problem for the overall build as long as the bad version is not the version that gets selected in the end. It seems some type of approach along those line would have solved the original Edit: part of the problem with the current behavior is the degree to which old problems that were fixed a while ago are still a "hangover" for right now. E.g., |
I'm not sure this solves the problem. Suppose there's some requirement on some version of a module that points to the bad pseudoversion of lint that doesn't include the old name. cmd/go still tries to validate that bad version and fails. I think it needs to be treated as a non-fatal error (non-fatal because it doesn't end up in the final build graph).
|
The problem is “does include the new name”, not “doesn't include the old name”. The problem today is that folks depend on revisions of If the |
Understood. I was illustrating a situation* where proposed fix of "module can specify acceptable old names and/or aliases" might not work. I suppose that's an edge case, since it would mean an invalid go.mod file in the first place (e.g., if someone had manually changed golang.org/x/lint to github.com/golang/lint). '* the case where something depends on a version of github.com/golang/lint that doesn't specify its aliases. |
I don't know if
Perhaps a completely different approach would be -- rather than saying the old spelling of lint cannot be upgraded, could the old spelling of lint be allowed to upgrade to the version of lint just before the |
Absent explicit release tags, we would have to scan every commit to the repository to find such a version. |
|
What is the solution? |
As far as I am following, I think this issue here is more about a medium or long term fix by changing the go tool, but that is not happening immediately. However, there is a healthy chance there is a more immediate solution to whatever you are seeing. It probably would make sense for you to review the symptoms at googleapis/google-cloud-go#1359 and golang/lint#436 and see if your symptoms match one of those reports, and either add comments there or perhaps open a new issue. |
Now that tools' dependencies have been cleaned up in golang.org/cl/160837, lint doesn't have any transitive dependencies on modules that import it using the wrong path. It is now safe to reintroduce a go.mod file to this repo. I've checked using dmitshur's instructions in #436 (comment) that lint only appears with its canonical module path in the build list. Updates #436 Change-Id: I6343aa103408b20562e17ea019602b159b899fc6 Reviewed-on: https://go-review.googlesource.com/c/lint/+/166278 Run-TryBot: Michael Matloob <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>
|
This has surfaced again in googleapis/google-api-go-client#347 (git.apache.org/thrift.git became github.com/apache/thrift). AFAICT, without the Go tool changing in some way, library owners have to resort to alien things like making libraries rely on non-existent versions as kindly spelled out by @dmitshur here or temporarily omitting dependencies. Both of these solutions require a lot of coordination and releasing multiple tags across multiple. This is pretty painful. :| Are there any other medium-term solutions anyone else knows of? |
bcmills
changed the title
One of the Ankh dependencies is importing `github.com/golang/lint` which now specifies it must be imported as `golang.org/x/lint`. ```sh go mod edit -replace github.com/golang/lint@latest=golang.org/x/lint@latest ``` golang/go#30831 golang/lint#436
|
Change https://golang.org/cl/220080 mentions this issue: |
Updates golang/go#36460 Updates golang/go#27900 Updates golang/go#26955 Updates golang/go#30831 Updates golang/go#32058 Updates golang/go#32380 Updates golang/go#32419 Updates golang/go#33370 Updates golang/go#33669 Updates golang/go#36369 Change-Id: I1d4644e3e8b4e688c2fc5a569312495e5072b7d7 Reviewed-on: https://go-review.googlesource.com/c/proposal/+/220080 Reviewed-by: Russ Cox <[email protected]>
ghost
mentioned this issue
|
We've given this issue a lot of thought over the last several years. The current best-practice solution for changing a module's import path is to choose at least one of:
Optionally, also leave behind packages in a repo at the old path containing declarations that merely alias (or forward function calls) to the packages at the new path. |
seankhliao
changed the title
|
related #70443 |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
While investigating googleapis/google-cloud-go#1359, golang/lint#436, and opening census-instrumentation/opencensus-go#1064, I've found it impossible to figure out how to get almost any of these packages to build and successfully run
go get -u.It seems to be a never-ending loop of things depending on slightly older things, which ultimately end up depending on the invalid github.com/golang/lint.
For a lot, but not all (see below) of these packages, I was able to track the chain from github.com/golang/lint up to its depender using
go mod graph.Of course, because of MVS, the old version of grpc (and therefore github.com/golang/lint) is never used in the actual build, but for some reason still blocks a successful run of
go get -u.What did you expect to see?
go get -usucceeds, no mention of github.com/golang/lint because it shouldn't be included due to MVS rules.What did you see instead?
The text was updated successfully, but these errors were encountered: