go mod graph list dependencies that aren't expected and contradicts go mod why

[didrocks]

What version of Go are you using (go version)?

go version go1.11.1 linux/amd64

Does this issue reproduce with the latest release?

Yes (1.11.1)

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/didrocks/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/didrocks/.go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build108021283=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Clone https://github.com/ubuntu/ubuntu-report (outside of GOPATH)

# outside of GOPATH
$ git clone https://github.com/ubuntu/ubuntu-report 
$ cd ubuntu-report
$ $ go mod graph
github.com/ubuntu/ubuntu-report github.com/cpuguy83/[email protected]
github.com/ubuntu/ubuntu-report github.com/davecgh/[email protected]
github.com/ubuntu/ubuntu-report github.com/inconshreveable/[email protected]
github.com/ubuntu/ubuntu-report github.com/onsi/[email protected]
github.com/ubuntu/ubuntu-report github.com/pkg/[email protected]
github.com/ubuntu/ubuntu-report github.com/pmezard/[email protected]
github.com/ubuntu/ubuntu-report github.com/russross/[email protected]
github.com/ubuntu/ubuntu-report github.com/sirupsen/[email protected]
github.com/ubuntu/ubuntu-report github.com/spf13/[email protected]
github.com/ubuntu/ubuntu-report github.com/spf13/[email protected]
github.com/ubuntu/ubuntu-report github.com/stretchr/[email protected]
github.com/ubuntu/ubuntu-report github.com/stretchr/[email protected]
github.com/ubuntu/ubuntu-report golang.org/x/[email protected]
github.com/ubuntu/ubuntu-report gopkg.in/airbrake/[email protected]
github.com/ubuntu/ubuntu-report gopkg.in/gemnasium/[email protected]
github.com/onsi/[email protected] github.com/fsnotify/[email protected]
github.com/onsi/[email protected] github.com/golang/[email protected]
github.com/onsi/[email protected] github.com/hpcloud/[email protected]
github.com/onsi/[email protected] github.com/onsi/[email protected]
github.com/onsi/[email protected] golang.org/x/[email protected]
github.com/onsi/[email protected] golang.org/x/[email protected]
github.com/onsi/[email protected] golang.org/x/[email protected]
github.com/onsi/[email protected] golang.org/x/[email protected]
github.com/onsi/[email protected] gopkg.in/[email protected]
github.com/onsi/[email protected] gopkg.in/[email protected]
github.com/onsi/[email protected] gopkg.in/[email protected]
gopkg.in/[email protected] gopkg.in/[email protected]

# I wasn't expecting gomega to show up (or having all indirect dependencies to show up)
$ grep -r gomega *
go.mod:	github.com/onsi/gomega v1.4.2 // indirect
go.sum:github.com/onsi/gomega v1.4.2 h1:3mYCb7aPxS/RU7TI1y4rkEn1oKmPRjNJLNEXgw7MH2I=
go.sum:github.com/onsi/gomega v1.4.2/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
$ go mod why github.com/onsi/gomega
# github.com/onsi/gomega
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
github.com/sirupsen/logrus
github.com/sirupsen/logrus.test
gopkg.in/gemnasium/logrus-airbrake-hook.v2
gopkg.in/airbrake/gobrake.v2
gopkg.in/airbrake/gobrake.v2.test
github.com/onsi/gomega

# coming thus from the .test package from logrus (indirect dep, as a stated on go.mod)

# This is inconsistent with testify, which doesn't show up in go mod graph, and requires -m in go mod why:
$ grep -r testify *
go.mod:	github.com/stretchr/testify v1.2.2 // indirect
go.sum:github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
go.sum:github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
$ go mod why github.com/stretchr/testify
# github.com/stretchr/testify
(main module does not need package github.com/stretchr/testify)
$ go mod why -m github.com/stretchr/testify
# github.com/stretchr/testify
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
github.com/sirupsen/logrus
github.com/sirupsen/logrus.test
github.com/stretchr/testify/assert

What did you expect to see?

I was expecting to see similar behavior for testify and gomega:

• Both shown in go mod graph output or none (I was expecting "none" as those are indirect dependencies)
• go mod why to require -m for both, as both are needed directly or indirectly by logrus.test.

What did you see instead?

I can't spot why there is a difference in those outputs for similar kind of indirect imports to me.

[myitcv]

go mod graph simply shows dependencies between all modules, or put in terms of the documentation:

Graph prints the module requirement graph (with replacements applied)
in text form.

Hence go mod graph with a main module of github.com/ubuntu/ubuntu-report/cmd/ubuntu-report shows all of the requirements for modules (transitively) required by github.com/ubuntu/ubuntu-report/cmd/ubuntu-report. Some of the "module" dependencies aren't yet modules, hence some dependencies are visible as // indirect as you note.

Refreshing on the documentation for go mod why:

Why shows a shortest path in the import graph from the main module to
each of the listed packages. If the -m flag is given, why treats the
arguments as a list of modules and finds a path to any package in each
of the modules.

So in summary:

• go mod graph simply reflects the module requirement graph in text form, nothing else
• go mod why -m github.com/stretchr/testify tries to find a package path from the main module to the github.com/stretchr/testify module:

$ go mod why -m github.com/stretchr/testify
# github.com/stretchr/testify
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
github.com/sirupsen/logrus
github.com/sirupsen/logrus.test
github.com/stretchr/testify/assert

• github.com/stretchr/testify/assert is one such package in the github.com/stretchr/testify module.. Hence go mod why github.com/stretchr/testify will show us the same path:

$ go mod why github.com/stretchr/testify/assert
# github.com/stretchr/testify/assert
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
github.com/sirupsen/logrus
github.com/sirupsen/logrus.test
github.com/stretchr/testify/assert

• Same story for github.com/onsi/gomega:

$ go mod why -m github.com/onsi/gomega
# github.com/onsi/gomega
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
github.com/sirupsen/logrus
github.com/sirupsen/logrus.test
gopkg.in/gemnasium/logrus-airbrake-hook.v2
gopkg.in/airbrake/gobrake.v2
gopkg.in/airbrake/gobrake.v2.test
github.com/onsi/gomega
$ go mod why github.com/onsi/gomega
# github.com/onsi/gomega
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
github.com/sirupsen/logrus
github.com/sirupsen/logrus.test
gopkg.in/gemnasium/logrus-airbrake-hook.v2
gopkg.in/airbrake/gobrake.v2
gopkg.in/airbrake/gobrake.v2.test
github.com/onsi/gomega

Everything appears to be working as expected here so I'm going to speculatively close this issue, but please shout if I've missed anything.

[didrocks]

Thanks for the answer @myitcv. That makes a lot more sense.

So, go mod why can't be used to assert that a module is included in a package binary from what I get.
If I take back your example of testify:

$ go mod why -m github.com/stretchr/testify/assert
# github.com/stretchr/testify/assert
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
github.com/sirupsen/logrus
github.com/sirupsen/logrus.test
github.com/stretchr/testify/assert

As part of the path, there is logrus.test and I can assume logrus doesn't import logrus.test, so I can only infer that testify/assert isn't imported.

Same for gopkg.in/check.v1:

go mod why -m gopkg.in/check.v1
# gopkg.in/check.v1
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report.test
github.com/spf13/cobra/doc
gopkg.in/yaml.v2
gopkg.in/yaml.v2.test
gopkg.in/check.v1

So, I guess there is no flag to get the list of imported packages, according to build flags (no test, excuded files) apart from building the binary itself and opening the package manifest inside the binary introduced by Go 1.11?

[myitcv]

So, I guess there is no flag to get the list of imported packages, according to build flags (no test, excuded files) apart from building the binary itself and opening the package manifest inside the binary introduced by Go 1.11?

I think you're after go list -json, specifically the .Deps field:

$ go list -json github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
{
        "Dir": "/tmp/ubuntu-report/cmd/ubuntu-report",
        "ImportPath": "github.com/ubuntu/ubuntu-report/cmd/ubuntu-report",
        "Name": "main",
...

To get a list of non-standard library dependencies you need to get a bit more creative:

$ go list -deps -f "{{if not .Standard}}{{.ImportPath}}{{end}}" github.com/ubuntu/ubuntu-report/cmd/ubuntu-report
golang.org/x/sys/unix
golang.org/x/crypto/ssh/terminal
github.com/sirupsen/logrus
github.com/spf13/pflag
github.com/spf13/cobra
github.com/pkg/errors
github.com/ubuntu/ubuntu-report/internal/sender
github.com/ubuntu/ubuntu-report/internal/utils
github.com/ubuntu/ubuntu-report/internal/metrics
github.com/ubuntu/ubuntu-report/pkg/sysmetrics
github.com/ubuntu/ubuntu-report/cmd/ubuntu-report

I've been planning to do a "Go Modules By Example" on the use of go mod why, go mod why -m, go mod graph and go list for a while... Will try and do that today.

[didrocks]

Makes a lot of sense now! Yeah, a Go Modules By Example would be awesome! 👍 Thanks again for your answers.

[bcmills]

See also #27900.