cmd/go: provide straightforward way to see non-test dependencies

[rogpeppe]

It's important to be able to see the full set of dependencies used by production code exclusive of tests, particularly with different major versions as distinct modules, as inadvertently using an import with a different major version can introduce bugs (also, some external dependencies can be toxic to have in production code).

The go mod why command provides a -vendor flag to exclude tests; perhaps go list could provide a similar flag to exclude testing dependencies.

In addition to the above, it is also useful to be able to be able to see modules used by local production code and tests, but not transitively by tests in external dependencies, as external dependencies are harder to change, and inadvertently forked modules are still an issue for tests too. I'm not sure of a good way to specify that, though.

[thepudds]

An example where a benchmark-only dependency appearing in go.mod caused consternation:
https://github.com/scylladb/go-set/pull/3/files#r208130318

In that example, one of the main points of this new project is to resuscitate a dead project (in a cleaner form), and there was surprise at seeing the dead project listed as a dependency of the new project. (There was surprise and consternation on the part of the authors, but one could also imagine that being a deterrent to a user evaluating whether or not to download and/or use this new project if it appears the new project relies on a dead project).

[flibustenet]

We should see if it's better to have this information directly in the go.mod file (like // test or separate sections of require). Or if it's better to use a cmd or a web site like godoc.org ?

[myitcv]

@flibustenet - just in case you haven't seen #26913 (comment) yet.

I think it's worth focussing on what people are trying to achieve here... which I agree has some overlap with their workflow and hence the tools they are using, but in the case of using godoc.org, I believe the underlying question that someone is trying to answer is "what dependencies does module M have?" and making some judgement on whether or not to use M using that information. Getting more detail on that will, I think, help to steer us towards the building blocks we then need in place.

As I noted on Slack:

Because all of these sorts of queries are possible using go list.... but the very fact we're having this discussion means that that solution is probably too esoteric. And if the queries are common enough it probably makes sense to provide something more accessible for day-to-day use

That said, we don't need to load everything into the go tool; if it provides the building blocks for some other solution that answers these same questions but via a different UI/UX then that's just as good.

[rogpeppe]

As an example of the issue here, I just noticed a dependency on github.com/jtolds/gls which I wasn't expecting. The go mod why output looked like this:

# github.com/jtolds/gls
github.com/juju/charmstore-client/cmd/charm
github.com/juju/juju/juju/osenv
github.com/juju/juju/juju/osenv.test
github.com/juju/juju/testing
github.com/juju/juju/api/block
github.com/juju/juju/api/block.test
github.com/juju/juju/state
github.com/juju/juju/state.test
github.com/juju/juju/component/all
github.com/juju/juju/cmd/juju/commands
github.com/juju/juju/provider/all
github.com/juju/juju/provider/ec2
gopkg.in/ini.v1
gopkg.in/ini.v1.test
github.com/smartystreets/goconvey/convey
github.com/jtolds/gls

So the shortest path to this dependency takes us through four levels of external test dependencies. In practice I almost never care about test dependencies of external dependencies.

[wsc1]

Hi, I'm redirecting the comment here

Russ's comment about scopes is compelling to have unified go.mod, but not compelling to hide scopes (build tags x test).

I think having go mod why not refer to scopes can be confusing or worse: go mod why is arguably not providing the functionality advertised (i.e. explaining why) if it doesn't also give scope information, at least for the main module.

[dmitris]

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module. In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

One idea is for go to add comments like github.com/gopherjs/gopherjs v0.0.0-20181004151105-1babbf986f6f // indirect-test for indirect dependencies that are dependencies from tests in the dependencies' modules. Then one could look at go.mod and see that packages/modules marked as // indirect-test are needed only if you want to run go test all or go test ... but not for the "regular" workflow of go install -v .... && go test -v ./... (to build and test your own code).

[flibustenet]

Tests dependencies should be visible, even for direct test dependencies. It's a valuable information. I'd like to choose a lib with as few dependency as possible to run but the most tests (and test dependencies if needed) as possible...

[bcmills]

In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

That's probably a task better suited to GOPROXY than to go.mod. (In particular, I would expect the company to provide a GOPROXY server containing all of the approved/vetted versions of external modules.)

[wsc1]

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module.

I think the go tool would be better. But, in theory the problem of deciding whether or not a file has satisfiable build tags is NP-complete :)

[bcmills]

perhaps go list could provide a similar flag to exclude testing dependencies.

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

[myitcv]

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

Per my answer in #27900 (comment), I think it depends if you expect/want that answer to be irrespective of build constraints.

[rogpeppe]

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

Yes, this is pretty much exactly what I was after (I wasn't aware of the -deps flag). For the request in my final paragraph, go list -test -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}'| sort -u does the job. Not that either of these are particularly easy to type, but I can write a script. It would be nice if go mod why could work with this call graph instead of the regular one though.

PS I'd be happier if the go command never looked at external test dependencies (they can be enormous and it's often impossible or uneconomic to run all tests in all third party modules), but that's another story.

[advdv]

Story
I want to share my frustrating journey to this problem. I recently started a new project and from the get go I wanted dependency management to be done purposefully. Meaning that i want to be careful and conscious about how codes get into my binary.

I'm not building any thing exotic, a basic web app with a database. This is my go.mod file:

module github.com/my/project

go 1.12

require (
	github.com/advanderveer/go-test v1.0.1
	github.com/golang-migrate/migrate/v4 v4.2.4
	github.com/google/wire v0.2.1
	github.com/hashicorp/go-uuid v1.0.1
	github.com/lib/pq v1.0.0
)

As a general policy I wanted all my dependencies to be at version v1 or higher. So unfortunately wire stands out here. I though that wire was only for code generation purposes so I was confused as to why it was here. Ergo, i run go why github.com/google/wire and it gives me no new information as to "why" it is there even though i guarded it from building //+build wireinject flag. Googling around a bit I start to accept that the whole build flag thing is a bit confusing for determining what modules are important here. So i cut my losses and simply add a comment to the mod file for the future, I'm unsure if this comment stays intact why I'm adding more dependencies but I have bigger fish to fry:

Because during my research above I went on to to open my go.sum file. Even though I managed my dependencies carefully I somehow ended up with 200 lines of packages I don't directly seem to use. Some of them quite big:

...
cloud.google.com/go 
github.com/aws/aws-sdk-go
github.com/cockroachdb/cockroach-go
github.com/gogo/protobuf
github.com/gopherjs/gopherjs
google.golang.org/grpc
...

So why are they there? go mod why github.com/gogo/protobuf says: # github.com/gogo/protobuf (main module does not need package github.com/gogo/protobuf). Eehm Ok. So why are they listed there? After some googling I run into this reddit thread which refers me to this issues where it is said: "If you have dependencies only needed for tests, then they will show up in your go.mod, and go get will download their go.mods, but it will not download their code" Even though they are not in my go.mod I think believe this changed 1.12(?) as I used to have his behaviour. At this point I'm just thinking that these dependencies are listed in my go.sum for testing purposes. It is cool that I can test my code and all dependencies but it is impractical for me because when I do this go test all it fails to compile on some deep package github.com/jackc/pgx and starts testing cockroachdb itself (which is huge).

At this point I start googling and find all kinds of commands in issues such as go mod why all go mod why -vendor, go list or event go list -test -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}'| sort -u to figure how how these dependencies end up in my project. None of them seem to give a conclusive answer so I take a step back:

Want I want to know is which code ends up in the binary I ship. So I can also just look at the build and see which packages are included for that, so I run: go build -o /tmp/my-project -v. I was pretty sure that the -v command would show which packages are being build (help says: "print the names of packages as they are compiled." ) but instead I got empty output... Hmm maybe its using some cache? Sure enough, when I add the -a flag I finally get mostly the information i'm looking for.

Solution
If you're looking for the list of packages that:

• Are not (indirect) test dependencies
• Take into account build flags

Simply build your binary using go build -v -a and see which packages are listed there. It also lists standard library packages so it still convoluted with information that you might not be interested in but at least it gives answer to the question.

Ideas
It is not that i'm new to programming with Go, and have spent considerate amount teaching the language to (new) engineer. I'm a big fan of the design rational (having read all the blog posts, they are awesome!) but experiences like the one above leave a lot to be desired. I have the following ideas that could solve the (problematic) user story above.

• have a comment in the go.sum file that indicate whether a dependency is indirect and/or just for testing. A I understand is the .sum file acts as the ultimate list of all external dependencies that are needed in one-way or the other and new comers always like to look a files first. Having the information there makes it more accessible
• the go why should always have an answer for each module in .sum file. And specifically without any extra flags. A user user goes like this: "Hey there is a dependency here I didn't expect, why?" and expects to get some sort of answer with "go why "
• Have a command that shows which external packages will be used for a build without the binary actually being build while taking into account build flags. I my case I just wanted to know what external code was shipping with my application.
• Maybe update the FAQ in the wiki with some of the information above?
• Maybe have the -v flag of the build command show when packages are not being build because of the cache?

Hopefully this information is usefully and I know that there are probably complex underlying issues that I don't understand as i'm not very knowledgable about how it all works. But go modules is also a part of the tooling new comers will interact with from the get go (phun intentded), so it makes sense to put some effort in the usability aspects.

Cheers,

Ad

[bcmills]

Want I want to know is which code ends up in the binary I ship.

That's what the -deps flag to go list is for:

go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./my-project | sort -u

• the go why should always have an answer for each module in .sum file

go.sum is not intended for human consumption. However, go mod graph should include the information you're looking for. (See also #27900.)

[advdv]

Thank you for your response. I think in general my feedback here is that, besides making sure that dependencies are present, the module system is also a means for engineers to understand a codebase. Especially in large projects, developers need to understand what parts are included when they open and start working on it. The module system acts as a BoM (bill of materials) in that sense, it acts as the blueprint of a piece of software. In my experience (having tough this practice) the current tooling is ill suited for this activity. I understand that somewhere between:

• go list [....]
• go mod why [...]
• go mod graph [...]
• the go.mod file
• the go.sum file

there is a way of answering these questions but as it is now this information and spread out and confusing.

[flibustenet]

Is there something that prevent to put something like // indirect-test in go.mod ?
It's not clear if it's not because we don't want this design or because of a technical difficulty ?

[bcmills]

@flibustenet, not all dependencies of your module are listed in your module's go.mod file. It only includes requirements that satisfy direct imports from your module, or that are needed by a transitive dependency but not required in any dependency's go.mod file.

So a comment in the go.mod file would be actively misleading: it would convey the impression that all indirect dependencies are listed there, and in the long term the vast majority probably won't be.

[akyoto]

Sometimes users of a lib will check go.mod files to see how heavy a given dependency for their project is. If the repository lists 5 binary dependencies and 95 test dependencies it is definitely less scary to use than a package that lists 100 dependencies (numbers exaggerated, but you get the point).

Separating the binary deps and test deps into 2 separate categories (build and test) inside the go.mod file would avoid this confusion.

require.build (
	dep/1
	dep/2
)

require.test (
	dep/3
	dep/4
)

[eclipseo]

I would like to add that in downstream Linux packaging, we would like to have an easy way to split tests dependencies from non-tests dependencies. It is important for us to be able to disable tests and tests deps to solve cyclic dependencies (since we build iteratively, not everything together, it happens often).
(speaking as a Go-SIG member for Fedora).

[thepudds]

Hello @nim-nim and @eclipseo, one quick comment is that if you haven't already read this, it is probably worth reading #29410 (comment) and some of the related discussion in #29410 around Go modules packaging approaches for Debian. Debian is distinct from Fedora, so that might not be useful, but maybe there is some overlap at least in terms of some of the challenges. There is also a short comment there from @qbit (who I think works on packaging for OpenBSD) at #29410 (comment). There is also an older discussion on OpenBSD packaging challenges with modules in #26852. Again, perhaps not useful, but wanted to at least mention it.

I have followed some of the different issues you have submitted, but not all of them, so sorry if this is redundant or otherwise not helpful.

[davidkhala]

I am laymen.
I simply look for when ater I run go mod tidy, those package apparently used only within *_test.go file will not be external

[komuw]

hubris got the better of me and I created: https://github.com/komuw/ote which adds a // test comment next to test dependencies

[tim-gp]

It would be useful if there was a way to get an equivalent list with go list as you get from go version -m binary. If you are planning to distribute the binary, it's valuable to know this list quickly perhaps without having to run a build.