cmd/compile, cmd/link: support arbitrary import file map configurations

[rsc]

Because of multiple versions of packages being used for different builds, proper support for package management in the go command is very likely going to need better support for caching build artifacts (long-standing issue #4719). I have a prototype of this caching that fixes #4719 but also various more specific bugs, like #11193, #14271, #15752, #15799, #18981, #19340, #20137, #20512 and almost certainly others I've missed.

But it's a big change. I'd like to publish an experimental go command (named xgo) that people can download and try, but I want xgo to use an unmodified Go 1.9 release, so that people trying it don't have to opt in to the instability of Go tip toolchain.

The toolchain support required by the caching prototype is fairly straightforward. The compiler and linker both need to support loading a definition of how to resolve imports instead of using the default file-name-construction lookup. For vendoring we added half of what the compiler needs, namely -importmap, which maps from import path to "fully-qualified import path" (the path containing /vendor/). The other half is a map from fully-qualified import path to exact package file to open for that import. The compiler needs this for imports it will see, and the linker needs this for all imports in the program.

The -importmap flag works for a few imports but doesn't scale well, since it must be repeated for every import, and some systems place limits on command-line size. Instead, I propose to add an -importcfg flag to both the compiler and linker that takes a single file name argument. The named file contains a sequence of lines like

importmap path=fqpath
importmap path2=fqpath2
packagefile fqpath=/path/to/file.a

There is one more change required in the compiler: it assumes that it can use the file name extension to decide whether to expect a package archive or a Go object. A trivial change lets it look at the first line of the file to make that decision (it was already reading the first line anyway) instead of using the file name.

Because I have a working prototype of proper build caching using this functionality, I'm confident these two changes are sufficient to enable the "experiment using an extra binary and the standard Go 1.9 toolchain" approach.

The -importcfg change changes very little of the code and is implemented such that it clearly has no effect if the flag is not used. As such, it should be safe to add this in Go 1.9 even now just before the beta.

[gopherbot]

CL https://golang.org/cl/44850 mentions this issue.

[gopherbot]

CL https://golang.org/cl/44851 mentions this issue.

[gopherbot]

CL https://golang.org/cl/44853 mentions this issue.