From 85effef2002473b4bb7d08f4adc3dd5b7449a82d Mon Sep 17 00:00:00 2001 From: "Bryan C. Mills" Date: Mon, 10 Feb 2020 16:19:15 -0500 Subject: [PATCH] design/36460: add design for lazy module loading Updates golang/go#36460 Updates golang/go#27900 Updates golang/go#26955 Updates golang/go#30831 Updates golang/go#32058 Updates golang/go#32380 Updates golang/go#32419 Updates golang/go#33370 Updates golang/go#33669 Updates golang/go#36369 Change-Id: I1d4644e3e8b4e688c2fc5a569312495e5072b7d7 Reviewed-on: https://go-review.googlesource.com/c/proposal/+/220080 Reviewed-by: Russ Cox --- design/36460-lazy-module-loading.md | 696 ++++++++++++++++++++++ design/36460/all_pattern.txt | 146 +++++ design/36460/registry_incompatibility.txt | 128 ++++ 3 files changed, 970 insertions(+) create mode 100644 design/36460-lazy-module-loading.md create mode 100644 design/36460/all_pattern.txt create mode 100644 design/36460/registry_incompatibility.txt diff --git a/design/36460-lazy-module-loading.md b/design/36460-lazy-module-loading.md new file mode 100644 index 00000000..311e63ed --- /dev/null +++ b/design/36460-lazy-module-loading.md @@ -0,0 +1,696 @@ +# Proposal: Lazy Module Loading + +Author: Bryan C. Mills (with substantial input from Russ Cox, Jay Conrod, and +Michael Matloob) + +Last updated: 2020-02-20 + +Discussion at https://golang.org/issue/36460. + +## Abstract + +We propose to change `cmd/go` to avoid loading transitive module dependencies +that have no observable effect on the packages to be built. + +The key insights that lead to this approach are: + +1. If _no_ package in a given dependency module is ever (even transitively) + imported by any package loaded by an invocation of the `go` command, then an + incompatibility between any package in that dependency and any other package + has no observable effect in the resulting program(s). Therefore, we can + safely ignore the (transitive) requirements of any module that does not + contribute any package to the build. + +2. We can use the explicit requirements of the main module as a coarse filter + on the set of modules relevant to the main module and to previous + invocations of the `go` command. + +Based on those insights, we propose to change the `go` command to retain more +transitive dependencies in `go.mod` files and to avoid loading `go.mod` files +for “irrelevant” modules, while still maintaining high reproducibility for build +and test operations. + +## Background + +In the initial implementation of modules, we attempted to make `go mod tidy` +prune out of the `go.mod` file any module that did not provide a transitive +import of the main module. However, that did not always preserve the remaining +build list: a module that provided no packages might still raise the minimum +requirement on some _other_ module that _did_ provide a package. + +We addressed that problem in [CL 121304] by explicitly retaining requirements on +all modules that provide _directly-imported_ packages, _as well as_ a minimal +set of module requirement roots needed to retain the selected versions of +transitively-imported packages. + +In [#29773] and [#31248], we realized that, due to the fact that the `go.mod` +file is pruned to remove indirect dependencies already implied by other +requirements, we must load the `go.mod` file for all versions of dependencies, +even if we know that they will not be selected — even including the main module +itself! + +In [#30831] and [#34016], we learned that following deep history makes +problematic dependencies very difficult to completely eliminate. If the +repository containing a module is no longer available and the module is not +cached in a module mirror, then we will encounter an error when loading any +module — even a very old, irrelevant one! — that required it. + +In [#26904], [#32058], [#33370], and [#34417], we found that the need to +consider every version of a module separately, rather than only the selected +version, makes the `replace` directive difficult to understand, difficult to use +correctly, and generally more complex than we would like it to be. + +In addition, users have repeatedly expressed the desire to avoid the cognitive +overhead of seeing “irrelevant” transitive dependencies ([#26955], [#27900], +[#32380]), reasoning about older-than-selected transitive dependencies +([#36369]), and fetching large numbers of `go.mod` files ([#33669], [#29935]). + +### Properties + +In this proposal, we aim to achieve a property that we call lazy +loading: + +* In the steady state, an invocation of the `go` command should not load any + `go.mod` file or source code for a module (other than the main module) that + provides no _packages_ loaded by that invocation. + + * In particular, if the selected version of a module is not changed by a + `go` command, the `go` command should not load a `go.mod` file or source + code for any _other_ version of that module. + +We also want to preserve reproducibility of `go` command invocations: + +* An invocation of the `go` command should either load the same version of + each package as every other invocation since the last edit to the `go.mod` + file, or should edit the `go.mod` file in a way that causes the next + invocation on any subset of the same packages to use the same versions. + +## Proposal + +### Invariants + +We propose that, when the main module's `go.mod` file specifies `go 1.15` or +higher, every invocation of the `go` command should update the `go.mod` file to +maintain three invariants. + +1. (The import invariant.) The main module's `go.mod` file + explicitly requires the selected version of every module that contains one + or more packages that were transitively imported by any package in the main + module. + +2. (The argument invariant.) The main module's `go.mod` file + explicitly requires the selected version of every module that contains one + or more packages that matched an explicit [package pattern] argument. + +3. (The completeness invariant.) The version of every module that + contributed any package to the build is recorded in the `go.mod` file of + either the main module itself or one of modules it requires explicitly. + +The _completeness invariant_ alone is sufficient to ensure _reproducibility_ and +_lazy loading_. However, it is under-constrained: there are potentially many +_minimal_ sets of requirements that satisfy the completeness invariant, and even +more _valid_ solutions. The _import_ and _argument_ invariants guide us toward a +_specific_ solution that is simple and intuitive to explain in terms of the `go` +commands invoked by the user. + +If the main module satisfies the _import_ and _argument_ invariants, and all +explicit module dependencies also satisfy the import invariant, then the +_completeness_ invariant is also trivially satisfied. Given those, the +completeness invariant exists only in order to tolerate _incomplete_ +dependencies. + +If the import invariant or argument invariant holds at the start of a `go` +invocation, we can trivially preserve that invariant (without loading any +additional packages or modules) at the end of the invocation by updating the +`go.mod` file with explicit versions for all module paths that were already +present, in addition to any new main-module imports or package arguments found +during the invocation. + +### Module loading procedure + +At the start of each operation, we load all of the explicit requirements from +the main module's `go.mod` file. + +If we encounter an import from any module that is not already _explicitly_ +required by the main module, we perform a deepening scan. To perform +a deepening scan, we read the `go.mod` file for each module explicitly required +by the main module, and add its requirements to the build list. (The deepening +scan allows us to detect changes to the import graph without loading the whole +graph explicitly: if we encounter a new import from within a +previously-irrelevant package, the deepening scan will re-read the requirements +of the module containing that package, and will ensure that the selected version +of that import is compatible with all other relevant packages.) + +As we load each imported package, we also read the `go.mod` file for the module +containing that package and add its requirements to the build list — even if +that version of the module was already explicitly required by the main module. + +(If all dependencies satisfy the _import invariant_, then reading the `go.mod` +file for existing dependencies is theoretically redundant: the requirements of +the main module will already reflect any relevant dependencies, and the +_deepening scan_ will catch any previously-irrelevant dependencies that +subsequently _become_ relevant. However, reading the `go.mod` file for each +imported package makes the `go` command much more robust to inconsistencies in +the `go.mod` file — including manual edits, erroneous version-control merge +resolutions, incomplete dependencies, and changes in `replace` directives and +replacement directory contents.) + +In order to ensure that modules written against `go 1.14` or earlier retain +their current dependencies, if at any point we read any `go.mod` file that +specifies `go 1.14` or lower, we eagerly load all of the transitive requirements +of that module and treat those transitive requirements as explicit. + +If, after the _deepening scan,_ the package to be imported is still not found in +any module in the build list, we resolve the `latest` version of a module +containing that package and add it to the build list (following the same search +procedure as in Go 1.14). + +### The `all` pattern and `mod` subcommands + +#### In Go 1.11–1.14 + +In module mode in Go 1.11–1.14, the `all` package pattern matches each package +reachable by following imports _and tests of imported packages_ recursively, +starting from the packages in the main module. (It is equivalent to the set of +packages obtained by iterating `go list -deps -test ./...` over its own output +until it reaches a fixed point.) + +`go mod tidy` adjusts the `go.mod` and `go.sum` files so that the main module +transitively requires a set of modules that provide every package matching the +`all` package pattern, independent of build tags. After `go mod tidy`, every +package matching the `all` _package_ pattern is provided by some module matching +the `all` _module_ pattern. + +`go mod tidy` also updates a set of `// indirect` comments indicating versions +added or upgraded beyond what is implied by transitive dependencies. + +`go mod download` downloads all modules matching the `all` _module_ pattern, +which normally includes a module providing every package in the `all` _package_ +pattern. + +In contrast, `go mod vendor` copies in only the subset of packages transitively +_imported by_ the packages and tests _in the main module_: it does not scan the +imports of tests outside of the main module, even if those tests are for +imported packages. (That is: `go mod vendor` only covers the packages directly +reported by `go list -deps -test ./...`.) + +As a result, when using `-mod=vendor` the `all` and `...` patterns may match +substantially fewer packages than when using `-mod=mod` (the default) or +`-mod=readonly`. + + + +#### The `all` package pattern and `go mod tidy` + +We would like to preserve the property that, after `go mod tidy`, invocations of +the `go` command — including `go test` — are _reproducible_ (without changing +the `go.mod` file) for every package matching the `all` package pattern. The +_completeness invariant_ is what ensures reproducibility, so `go mod tidy` must +ensure that it holds. + +Unfortunately, even if the _import invariant_ holds for all of the dependencies +of the main module, the current definition of the `all` pattern includes +_dependencies of tests of dependencies_, recursively. In order to establish the +_completeness invariant_ for distant test-of-test dependencies, `go mod tidy` +would sometimes need to record a substantial number of dependencies of tests +found outside of the main module in the main module's `go.mod` file. + +Fortunately, we can omit those distant dependencies a different way: by changing +the definition of the `all` pattern itself, so that test-of-test dependencies +are no longer included. Feedback from users (in [#29935], [#26955], [#32380], +[#32419], [#33669], and perhaps others) has consistently favored omitting those +dependencies, and narrowing the `all` pattern would also establish a nice _new_ +property: after running `go mod vendor`, the `all` package pattern with +`-mod=vendor` would now match the `all` pattern with `-mod=mod`. + +Taking those considerations into account, we propose that the `all` package +pattern in module mode should match only the packages transitively _imported by_ +packages and tests in the main module: that is, exactly the set of packages +preserved by `go mod vendor`. Since the `all` pattern is based on package +imports (more-or-less independent of module dependencies), this change should be +independent of the `go` version specified in the `go.mod` file. + +The behavior of `go mod tidy` should change depending on the `go` version. In a +module that specifies `go 1.15` or later, `go mod tidy` should scan the packages +matching the new definition of `all`, ignoring build tags. In a module that +specifies `go 1.14` or earlier, it should continue to scan the packages matching +the _old_ definition (still ignoring build tags). (Note that both of those sets +are supersets of the new `all` pattern.) + +#### The `all` and `...` module patterns and `go mod download` + +In Go 1.11–1.14, the `all` module pattern matches each module reachable by +following module requirements recursively, starting with the main module and +visiting every version of every module encountered. The module pattern `...` has +the same behavior. + +The `all` module pattern is important primarily because it is the default set of +modules downloaded by the `go mod download` subcommand, which sets up the local +cache for offline use. However, it (along with `...`) is also currently used by +a few other tools (such as `go doc`) to locate “modules of interest” for other +purposes. + +Unfortunately, these patterns as defined in Go 1.11–1.14 are _not compatible +with lazy loading:_ they examine transitive `go.mod` files without loading any +packages. Therefore, in order to achieve lazy loading we must change their +behavior. + +Since we want to compute the list of modules without loading any packages or +irrelevant `go.mod` files, we propose that when the main module's `go.mod` file +specifies `go 1.15` or higher, the `all` and wildcard module patterns should +match only those modules found in a _deepening scan_ of the main module's +dependencies. That definition includes every module whose version is +reproducible due to the _completeness invariant,_ including modules needed by +tests of transitive imports. + +With this redefinition of the `all` module pattern, and the above redefinition +of the `all` package pattern, we again have the property that, after `go mod +tidy && go mod download all`, invoking `go test` on any package within `all` +does not need to download any new dependencies. + +Since the `all` pattern includes every module encountered in the deepening scan, +rather than only those that provide imported packages, `go mod download` may +continue to download more source code than is strictly necessary to build the +packages in `all`. However, as is the case today, users may download only that +narrower set as a side effect of invoking `go list all`. + +### Effect on `go.mod` size + +Under this approach, the set of modules recorded in the `go.mod` file would in +most cases increase beyond the set recorded in Go 1.14. However, the set of +modules recorded in the `go.sum` file would decrease: irrelevant modules would +no longer be included. + +- The modules recorded in `go.mod` under this proposal would be a strict + subset of the set of modules recorded in `go.sum` in Go 1.14. + + - The set of recorded modules would more closely resemble a “lock” file as + used in other dependency-management systems. (However, the `go` command + would still not require a separate “manifest” file, and unlike a lock + file, the `go.mod` file would still be updated automatically to reflect + new requirements discovered during package loading.) + +- For modules with _few_ test-of-test dependencies, the `go.mod` file after + running `go mod tidy` will typically be larger than in Go 1.14. For modules + with _many_ test-of-test dependencies, it may be substantially smaller. + +- For modules that are _tidy:_ + + - The module versions recorded in the `go.mod` file would be exactly those + listed in `vendor/modules.txt`, if present. + + - The module versions recorded in `vendor/modules.txt` would be the same + as under Go 1.14, although the `## explicit` annotations could perhaps + be removed (because _all_ relevant dependencies would be recorded + explicitly). + + - The module versions recorded in the `go.sum` file would be exactly those + listed in the `go.mod` file. + +## Compatibility + +The `go.mod` file syntax and semantics proposed here are backward compatible +with previous Go releases: all `go.mod` files for existing `go` versions would +retain their current meaning. + +Under this proposal, a `go.mod` file that specifies `go 1.15` or higher will +cause the `go` command to lazily load the `go.mod` files for its requirements. +When reading a `go 1.15` file, previous versions of the `go` command (which do +not prune irrelevant dependencies) may select _higher_ versions than those +selected under this proposal, by following otherwise-irrelevant dependency +edges. However, because the `require` directive continues to specify a minimum +version for the required dependency, a previous version of the `go` command will +never select a _lower_ version of any dependency. + +Moreover, any strategy that prunes out a dependency as interpreted by a previous +`go` version will continue to prune out that dependency as interpreted under +this proposal: module maintainers will not be forced to break users on new `go` +versions in order to support users on older versions (or vice-versa). + +Versions of the `go` command before 1.14 do not preserve the proposed invariants +for the main module: if `go` command from before 1.14 is run in a `go 1.15` +module, it may automatically remove requirements that are now needed. However, +as a result of [CL 204878], `go` version 1.14 does preserve those invariants in +all subcommands except for `go mod tidy`: Go 1.14 users will be able to work (in +a limited fashion) within a Go 1.15 main module without disrupting its +invariants. + +## Implementation + +`bcmills` is working on a prototype of this design for `cmd/go` in Go 1.15. + +At this time, we do not believe that any other tooling changes will be needed. + +## Open issues + +Because `go mod tidy` will now preserve seemingly-redundant requirements, we may +find that we want to expand or update the `// indirect` comments that it +currently manages. For example, we may want to indicate “indirect dependencies +at implied versions” separately from “upgraded or potentially-unused indirect +dependencies”, and we may want to indicate “direct or indirect dependencies of +tests” separately from “direct or indirect dependencies of non-tests”. + +Since these comments do not have a semantic effect, we can fine-tune them after +implementation (based on user feedback) without breaking existing modules. + +## Examples + +The following examples illustrate the proposed behavior using the `cmd/go` +[script test] format. For local testing and exploration, the test files can be +extracted using the [`txtar`] tool. + +### Importing a new package from an existing module dependency + +```txtar +cp go.mod go.mod.old +go mod tidy +cmp go.mod go.mod.old + +# Before adding a new import, the go.mod file should +# enumerate modules for all packages already imported. + +go list all +cmp go.mod go.mod.old + +# When a new import is found, we should perform a deepening scan of the existing +# dependencies and add a requirement on the version required by those +# dependencies — not re-resolve 'latest'. + +cp lazy.go.new lazy.go +go list all +cmp go.mod go.mod.new + +-- go.mod -- +module example.com/lazy + +go 1.15 + +require ( + example.com/a v0.1.0 + example.com/b v0.1.0 // indirect +) + +replace ( + example.com/a v0.1.0 => ./a + example.com/b v0.1.0 => ./b + example.com/c v0.1.0 => ./c1 + example.com/c v0.2.0 => ./c2 +) +-- lazy.go -- +package lazy + +import ( + _ "example.com/a/x" +) +-- lazy.go.new -- +package lazy + +import ( + _ "example.com/a/x" + _ "example.com/a/y" +) +-- go.mod.new -- +module example.com/lazy + +go 1.15 + +require ( + example.com/a v0.1.0 + example.com/b v0.1.0 // indirect + example.com/c v0.1.0 // indirect +) + +replace ( + example.com/a v0.1.0 => ./a + example.com/b v0.1.0 => ./b + example.com/c v0.1.0 => ./c1 + example.com/c v0.2.0 => ./c2 +) +-- a/go.mod -- +module example.com/a + +go 1.15 + +require ( + example.com/b v0.1.0 + example.com/c v0.1.0 +) +-- a/x/x.go -- +package x +import _ "example.com/b" +-- a/y/y.go -- +package y +import _ "example.com/c" +-- b/go.mod -- +module example.com/b + +go 1.15 +-- b/b.go -- +package b +-- c1/go.mod -- +module example.com/c + +go 1.15 +-- c1/c.go -- +package c +-- c2/go.mod -- +module example.com/c + +go 1.15 +-- c2/c.go -- +package c +``` + +### Testing an imported package found in another module + +```txtar +cp go.mod go.mod.old +go mod tidy +cmp go.mod go.mod.old + +# 'go list -m all' should include modules that cover the test dependencies of +# the packages imported by the main module, found via a deepening scan. + +go list -m all +stdout 'example.com/b v0.1.0' +! stdout example.com/c +cmp go.mod go.mod.old + +# 'go test' of any package in 'all' should use its existing dependencies without +# updating the go.mod file. + +go list all +stdout example.com/a/x + +go test example.com/a/x +cmp go.mod go.mod.old + +-- go.mod -- +module example.com/lazy + +go 1.15 + +require example.com/a v0.1.0 + +replace ( + example.com/a v0.1.0 => ./a + example.com/b v0.1.0 => ./b1 + example.com/b v0.2.0 => ./b2 + example.com/c v0.1.0 => ./c +) +-- lazy.go -- +package lazy + +import ( + _ "example.com/a/x" +) +-- a/go.mod -- +module example.com/a + +go 1.15 + +require example.com/b v0.1.0 +-- a/x/x.go -- +package x +-- a/x/x_test.go -- +package x + +import ( + "testing" + + _ "example.com/b" +) + +func TestUsingB(t *testing.T) { + // … +} +-- b1/go.mod -- +module example.com/b + +go 1.15 + +require example.com/c v0.1.0 +-- b1/b.go -- +package b +-- b1/b_test.go -- +package b + +import _ "example.com/c" +-- b2/go.mod -- +module example.com/b + +go 1.15 + +require example.com/c v0.1.0 +-- b2/b.go -- +package b +-- b2/b_test.go -- +package b + +import _ "example.com/c" +-- c/go.mod -- +module example.com/c + +go 1.15 +-- c/c.go -- +package c +``` + +### Testing an unimported package found in an existing module dependency + +```txtar +cp go.mod go.mod.old +go mod tidy +cmp go.mod go.mod.old + +# 'go list -m all' should include modules that cover the test dependencies of +# the packages imported by the main module, found via a deepening scan. + +go list -m all +stdout 'example.com/b v0.1.0' +cmp go.mod go.mod.old + +# 'go test all' should use those existing dependencies without updating the +# go.mod file. + +go test all +cmp go.mod go.mod.old + +-- go.mod -- +module example.com/lazy + +go 1.15 + +require ( + example.com/a v0.1.0 +) + +replace ( + example.com/a v0.1.0 => ./a + example.com/b v0.1.0 => ./b1 + example.com/b v0.2.0 => ./b2 + example.com/c v0.1.0 => ./c +) +-- lazy.go -- +package lazy + +import ( + _ "example.com/a/x" +) +-- a/go.mod -- +module example.com/a + +go 1.15 + +require ( + example.com/b v0.1.0 +) +-- a/x/x.go -- +package x +-- a/x/x_test.go -- +package x + +import _ "example.com/b" + +func TestUsingB(t *testing.T) { + // … +} +-- b1/go.mod -- +module example.com/b + +go 1.15 +-- b1/b.go -- +package b +-- b1/b_test.go -- +package b + +import _ "example.com/c" +-- b2/go.mod -- +module example.com/b + +go 1.15 + +require ( + example.com/c v0.1.0 +) +-- b2/b.go -- +package b +-- b2/b_test.go -- +package b + +import _ "example.com/c" +-- c/go.mod -- +module example.com/c + +go 1.15 +-- c/c.go -- +package c +``` + + + +[package pattern]: https://tip.golang.org/cmd/go/#hdr-Package_lists_and_patterns + "go — Package lists and patterns" +[script test]: https://go.googlesource.com/go/+/refs/heads/master/src/cmd/go/testdata/script/README + "src/cmd/go/testdata/script/README" +[`txtar`]: https://pkg.go.dev/golang.org/x/exp/cmd/txtar + "golang.org/x/exp/cmd/txtar" +[CL 121304]: https://golang.org/cl/121304 + "cmd/go/internal/vgo: track directly-used vs indirectly-used modules" +[CL 122256]: https://golang.org/cl/122256 + "cmd/go/internal/modcmd: drop test sources and data from mod -vendor" +[CL 204878]: https://golang.org/cl/204878 + "cmd/go: make commands other than 'tidy' prune go.mod less aggressively" +[#26904]: https://golang.org/issue/26904 + "cmd/go: allow replacement modules to alias other active modules" +[#27900]: https://golang.org/issue/27900 + "cmd/go: 'go mod why' should have an answer for every module in 'go list -m all'" +[#29773]: https://golang.org/issue/29773 + "cmd/go: 'go list -m' fails to follow dependencies through older versions of the main module" +[#29935]: https://golang.org/issue/29935 + "x/build: reconsider the large number of third-party dependencies" +[#26955]: https://golang.org/issue/26955 + "cmd/go: provide straightforward way to see non-test dependencies" +[#30831]: https://golang.org/issue/30831 + "cmd/go: 'get -u' stumbles over repos imported via non-canonical paths" +[#31248]: https://golang.org/issue/31248 + "cmd/go: mod tidy removes lines that build seems to need" +[#32058]: https://golang.org/issue/32058 + "cmd/go: replace directives are not thoroughly documented" +[#32380]: https://golang.org/issue/32380 + "cmd/go: don't add dependencies of external tests" +[#32419]: https://golang.org/issue/32419 + "proposal: cmd/go: conditional/optional dependency for go mod" +[#33370]: https://golang.org/issue/33370 + "cmd/go: treat pseudo-version 'vX.0.0-00010101000000-000000000000' as equivalent to an empty commit" +[#33669]: https://golang.org/issue/33669 + "cmd/go: fetching dependencies can be very aggressive when going via an HTTP proxy" +[#34016]: https://golang.org/issue/34016 + "cmd/go: 'go list -m all' hangs for git.apache.org" +[#34417]: https://golang.org/issue/34417 + "cmd/go: do not allow the main module to replace (to or from) itself" +[#34822]: https://golang.org/issue/34822 + "cmd/go: do not update 'go.mod' automatically if the changes are only cosmetic" +[#36369]: https://golang.org/issue/36369 + "cmd/go: dependencies in go.mod of older versions of modules in require cycles affect the current version's build" diff --git a/design/36460/all_pattern.txt b/design/36460/all_pattern.txt new file mode 100644 index 00000000..5a6f5db7 --- /dev/null +++ b/design/36460/all_pattern.txt @@ -0,0 +1,146 @@ +# This example illustrates the relationship between the 'all' pattern and +# the dependencies of the main module. + + +# 'go list -deps' lists transitive imports of non-tests in the main module. + +$ go1.14rc1 list -f '{{with .Module}}{{.Path}}{{end}}' -deps ./... | sort -u +example.com/a +example.com/b +example.com/main + + +# 'go list -deps -test' lists transitive imports of tests and non-tests in the +# main module. + +$ go1.14rc1 list -f '{{with .Module}}{{.Path}}{{end}}' -deps -test ./... | sort -u +example.com/a +example.com/b +example.com/main +example.com/t +example.com/u + + +# 'go list all' lists the fixpoint of iterating 'go list -deps -test' starting +# with the packages in the main module. + +$ go1.14rc1 list -f '{{with .Module}}{{.Path}}{{end}}' all | sort -u +example.com/a +example.com/b +example.com/c +example.com/main +example.com/t +example.com/u +example.com/w + +$ go1.14rc1 list -f '{{with .Module}}{{.Path}}{{end}}' -deps -test $(go1.14rc1 list -deps -test ./... | grep -v '[ .]') | sort -u +example.com/a +example.com/b +example.com/c +example.com/main +example.com/t +example.com/u +example.com/w + + +# 'go mod vendor' copies in only the packages transitively imported by the main +# module. As a result, the 'all' and '...' patterns report fewer packages when +# using '-mod=vendor'. + +$ go1.14rc1 mod vendor + +$ go1.14rc1 list -mod=vendor all +example.com/a +example.com/b +example.com/main +example.com/t +example.com/u + + +-- go.mod -- +module example.com/main + +go 1.14 + +require ( + example.com/a v0.1.0 + example.com/b v0.1.0 + example.com/t v0.1.0 +) + +replace ( + example.com/a v0.1.0 => ./a + example.com/b v0.1.0 => ./b + example.com/c v0.1.0 => ./c + example.com/t v0.1.0 => ./t + example.com/u v0.1.0 => ./u + example.com/w v0.1.0 => ./w +) +-- main.go -- +package main + +import _ "example.com/a" + +func main() {} +-- main_test.go -- +package main_test + +import _ "example.com/t" +-- a/go.mod -- +module example.com/a + +go 1.14 + +require ( + example.com/b v0.1.0 + example.com/c v0.1.0 +) +-- a/a.go -- +package x + +import _ "example.com/b" +-- a/a_test.go -- +package x_test + +import _ "example.com/c" +-- b/go.mod -- +module example.com/b + +go 1.14 +-- b/b.go -- +package b +-- c/go.mod -- +module example.com/c + +go 1.14 +-- c/c.go -- +package c +-- t/go.mod -- +module example.com/t + +go 1.14 + +require ( + example.com/u v0.1.0 + example.com/w v0.1.0 +) +-- t/t.go -- +package t + +import _ "example.com/u" +-- t/t_test.go -- +package t_test + +import _ "example.com/w" +-- u/go.mod -- +module example.com/u + +go 1.14 +-- u/u.go -- +package u +-- w/go.mod -- +module example.com/w + +go 1.14 +-- w/w.go -- +package w diff --git a/design/36460/registry_incompatibility.txt b/design/36460/registry_incompatibility.txt new file mode 100644 index 00000000..43f81a6b --- /dev/null +++ b/design/36460/registry_incompatibility.txt @@ -0,0 +1,128 @@ +# This example illustrates a case in which a seemingly-irrelevant requirements +# in one module fixes an incompatibility with a package in another. +# +# Two packages that do not import each other may interact via a third (mutual) +# dependency, so their module requirements must be respected even if those +# requirements do not directly correspond to package imports. + + +# Before discovering the import of package c, we depend on b v1.0.0. + +$ go1.14rc1 list -m b +b v1.0.0 => ./b0 + + +# When we import package c, we must update the selected version of b according +# to c's requirements, _even though_ neither package imports the other. +# Otherwise, the program will panic at run time. + +$ go1.14rc1 build c +go: found c in c v1.0.0 + +$ go1.14rc1 list -m b +b v1.0.1 => ./b1 + + +-- go.mod -- +module main + +go 1.15 + +require ( + a v1.0.0 + b v1.0.0 + d v1.0.0 +) + +replace ( + a v1.0.0 => ./a + b v1.0.0 => ./b0 + b v1.0.1 => ./b1 + c v1.0.0 => ./c + d v1.0.0 => ./d +) +-- main.go -- +package main + +import ( + _ "a" + + // Adding an import of c should increase the selected version of b to 1.0.1 + // in order to avoid a registration collision caused by a bug in b 1.0.0. + // Module b is not otherwise relevant to package c or its containing module. + _ "c" +) + +func main() {} +-- a/go.mod -- +module a + +go 1.15 + +require b v1.0.0 +-- a/a.go -- +package a + +import _ "b" +-- b0/go.mod -- +module b + +go 1.15 + +require d v1.0.0 +-- b0/b.go -- +package b + +import "d" + +func init() { d.Register("c") } // This is a bug that breaks package c. +-- b1/go.mod -- +module b + +go 1.15 + +require d v1.0.0 +-- b1/b.go -- +package b + +import "d" + +func init() { d.Register("b") } // The bug has been fixed. +-- c/go.mod -- +module c + +require ( + b v1.0.1 + d v1.0.0 +) +-- c/c.go -- +package c + +import "d" + +func init() { d.Register("c") } + +-- c/tidy/tidy.go -- +// +build tidy-only + +package tidy + +// Maintain a false dependency on b v1.0.1, to prevent b v1.0.0 +// from erroneously registering itself with our path if anything +// that imports us also imports package b. +import _ "b" +-- d/go.mod -- +module d + +go 1.15 +-- d/d.go -- +package d + +var registered = map[string]bool{} + +func Register(key string) { + if registered[key] { + panic("duplicate registation for " + key) + } + registered[key] = true +}