cmd/go: avoid work to compute unused fields in 'go list'

[hyangah]

We want to get the list of available versions of a module
and the best way is to use go list -m --versions <module>.

Under the hood, the command retrieves the tag list (which we expected)
and also fetches some versions (which is surprising).

Our guess is that the go list does so in order to find information
like the latest version's commit time, etc. For our use case, we don't need
that information and the extra fetch slows down the go list especially
when the repository is large.

Can we have a flag to skip the fetch?

An alternative mentioned by @bcmills is to make go list smarter so it
performs steps necessary to fill in the requested information (e.g. fields
mentioned in -f, or with an additional option to specify fields to prune if
-json is requested)

$ GO111MODULE=on tipgo list -x -m -f '{{.Versions}}' --versions rsc.io/quote

mkdir -p $GOPATH/pkg/mod/cache/vcs # git2 https://github.com/rsc/quote
# lock $GOPATH/pkg/mod/cache/vcs/$HASH.lockmkdir -p $GOPATH/pkg/mod/cache/vcs/$HASH # git2 https://github.com/rsc/quote
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git init --bare
0.006s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git init --bare
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git remote add origin https://github.com/rsc/quote 
0.004s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git remote add origin https://github.com/rsc/quote
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git ls-remote -q origin
0.298s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git ls-remote -q origin
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git cat-file --batch
0.004s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git cat-file --batch

cd $GOPATH/pkg/mod/cache/vcs/$HASH; git fetch -f origin refs/tags/v2.0.0:refs/tags/v2.0.0 refs/tags/v2.0.1:refs/tags/v2.0.1 refs/tags/v3.0.0:refs/tags/v3.0.0
0.365s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git fetch -f origin refs/tags/v2.0.0:refs/tags/v2.0.0 refs/tags/v2.0.1:refs/tags/v2.0.1 refs/tags/v3.0.0:refs/tags/v3.0.0

cd $GOPATH/pkg/mod/cache/vcs/$HASH; git cat-file --batch
0.004s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git cat-file --batch
go: finding rsc.io/quote v1.5.2
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git tag -l
0.003s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git tag -l
cd $GOPATH/pkg/mod/cache/vcs/$HASH; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' refs/tags/v1.5.2
0.004s # cd $GOPATH/pkg/mod/cache/vcs/$HASH; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' refs/tags/v1.5.2

[v1.0.0 v1.1.0 v1.2.0 v1.2.1 v1.3.0 v1.4.0 v1.5.0 v1.5.1 v1.5.2 v1.5.3-pre1]

@bcmills @heschik @katiehockman @jayconrod

[heschi]

From what I can piece together, Hana's example above is particularly poor, but probably unfixable. It's looking for a v1 module, but it needs to download every non-v1 version too to make sure that they all have a go.mod. (If one didn't, then it'd be +incompatible and a valid answer for the query.)

Previously we had been looking at the way it behaves when fetching rsc.io/quote/v2, or a v1 module from a repo that doesn't have v2 yet. That can probably be improved.

[bcmills]

I would rather do the field-based pruning than add a separate flag. Field-based pruning gives us the opportunity to eliminate a lot of different kinds of redundant work, whereas a “skip timestamp fetching” flag would have a very narrow benefit.

[bcmills]

One nice option for the JSON field-pruning might be to allow the -json flag to accept a comma-separated list of fields with an explicit = connector:

go list -m -json=Path,Versions,Replace.Path --versions rsc.io/quote

[bcmills]

It's looking for a v1 module, but it needs to download every non-v1 version too to make sure that they all have a go.mod.

It occurs to me that this could interact with caching in an interesting way: if you're only looking for new versions and you know when you last checked the repo, then it's probably safe to assume that any tag that was +incompatible still is, and any tag that was a module still is. So perhaps some sort of flag to restrict the result to “tags added since timestamp T” would avoid the need for that fetch too.

[jayconrod]

I really like the idea of -json accepting a list of fields that are actually needed. I think we should do that on the regular go list (not -m) side as well.

[bcmills]

Retitled to address the general problem: we won't be adding a specific flag to avoid VCS fetches, but we certainly can optimize the work that we do based on the requested output.

[jayconrod]

In CL 270858, we noticed that go list -m all loads .info files for each module in the build list in order to find the version's timestamp. That's printed with -json and maybe -f. If neither of those flags are specified, then .info files aren't used: go build and go get don't need them. We should skip loading those files if we don't need their content.

[gopherbot]

Change https://golang.org/cl/309190 mentions this issue: cmd/go/internal/modload: avoid loading the module graph to list only the name of the main module

[gopherbot]

Change https://golang.org/cl/309191 mentions this issue: cmd/go/internal/modload: avoid loading the full module graph when listing specific modules

[gopherbot]

Change https://go.dev/cl/402736 mentions this issue: cmd/go: skip computing BuildInfo in go list unless it's needed

[gopherbot]

Change https://go.dev/cl/417595 mentions this issue: doc/go1.19: add a release note for 'go list -json=SomeField'

[matloob]

Closing this issue as the -json=SomeField feature has been released in Go 1.19

[gopherbot]

Change https://go.dev/cl/468075 mentions this issue: cmd/go: don't compute Embed fields if they're not needed