x/tools/go/packages: support for loading files/syntax irrespective of build tags

[myitcv]

What version of Go are you using (go version)?

go version go1.11.1 linux/amd64
go/packages commit 29f11e2b93f4d66f7c335bd7b2892836d4944f5c

Does this issue reproduce with the latest release?

Yes, per above

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPROXY=""
GORACE=""
GOROOT="/home/myitcv/gos"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build137760326=/tmp/go-build -gno-record-gcc-switches"

Picking up on the discussion from the 2018-10-09 golang-tools catch up

In the pre-modules world, go/build and go/parser could be used in combination to get file and syntax information for packages and their dependencies irrespective of GOOS, GOARCH and build tags via the UseAllFiles go/build.Context field.

This use case is important for tools like govers and gogrep.

go/packages is driven by Config which includes GOOS, GOARCH and build tags and hence its response is specific to the provided config. go/packages cannot, therefore, be used in these use cases.

This issue is a placeholder to continue discussion about how best to support loading of files, imports and syntax in a GOOS, GOARCH and build tags-agnostic way.

cc @ianthehat @matloob @alandonovan @bcmills @rogpeppe @mvdan

[myitcv]

One option discussed so far has been to make multiple calls to go/packages.Load for various GOOS, GOARCH and build tag combinations. This could conceivably be done concurrently with the results being added to the same *token.FileSet.

As was noted on yesterday's call, returning type information in this situation doesn't make sense.

However the go/packages.LoadMode currently specifies LoadSyntax as a "higher numbered mode" than LoadTypes. So it seems unclear how to get syntax for packages without types.

@matloob - perhaps you could comment on this particular question?

[mvdan]

If I recall our meeting correctly, what @ianthehat was suggesting is a separate package that works on directories, not buildable packages. I presume that it would look more like a mix of the old go/build and go/parser.ParseDir.

That sounds fine to me, and I presume it would require no changes to go/packages.

[bcmills]

They're not strictly separate, unfortunately.

For example, in a tool like goforward, we ideally want the source files for all architectures in the move phase, but build-system-agnostic source files in the replace and forward phases. It would be pretty irritating to have to use separate packages (and separate types!) for those phases, since they're doing a single multi-step transformation on the package.

[matloob]

Responding to @myitcv's question, it's not just the type information that doesn't make sense: adding the same file twice to a fileset will create two entries for that file, with different 'pos' values. So I don't think changing or adding a new Mode will help too much with this.

I think the best we can do is to call Load in LoadFiles mode and call parsefile ourselves. Calling parsefile is one of the simpler things go/packages does.

We suggested that this is something that could be done with go/packages, but I think go/packages isn't a good fit for this problem. Like @mvdan mentioned, a separate packages that works on directories might be the better solution. And because it wouldn't have to support other buildsystems, it wouldn't be hamstrung by the limitations go/packages faces.

[myitcv]

Related to #27900 in as much as go mod why operates independent of build tags (or put another way, for all possible build combinations) and what's being discussed here is akin to go list -nobuild (or whatever name such a flag would have).

[bcmills]

This feature probably requires #42504.