cmd/go: '"bzr": executable file not found in $PATH' when fetching github.com/influxdata/platform

[stevenh]

What version of Go are you using (go version)?

go version go1.11.4 freebsd/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

GOARCH="amd64"
GOBIN="/data/go/bin"
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOOS="freebsd"
GOPATH="/data/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/data/go/src/github.com/multiplay/go/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build442059268=/tmp/go-build -gno-record-gcc-switches"

What did you do?

GO111MODULE=on go get -u=patch

or

GO111MODULE=on go mod vendor

What did you expect to see?

I expected the commands to succeed.

What did you see instead?

go: launchpad.net/[email protected]: bzr branch --use-existing-dir https://launchpad.net/~niemeyer/gocheck/trunk . in /data/go/pkg/mod/cache/vcs/f46ce2ae80d31f9b0a29099baa203e3b6d269dace4e5357a2cf74bd109e13339: exec: "bzr": executable file not found in $PATH
go: labix.org/v2/[email protected]: bzr branch --use-existing-dir https://launchpad.net/mgo/v2 . in /data/go/pkg/mod/cache/vcs/ca61c737a32b1e09a0919e15375f9c2b6aa09860cc097f1333b3c3d29e040ea8: exec: "bzr": executable file not found in $PATH
go: error loading module requirements

The cause of this is we have a dependency on github.com/influxdata/platform which has a dependency in its go.mod file of launchpad.net/gocheck and labix.org/v2/mgo however neither packages are reference by github.com/influxdata/platform

This means that both go get and go mod vendor when run with module support incorrectly assume that intermediate go.mod files are valid.

Even worse is a matching replace in the local go.mod doesn't apply and hence can not be used to override the 3rd party dependency.

This means that the failure of a 3rd party to update their go.mod can prevents any dependent projects from downloading / updating.

[stevenh]

Ok its even worse than I first through, having downloaded github.com/influxdata/platform when I run any module related commands it fails due to the inability to get the dependencies even when I'm trying to get it to update the dependencies.

go-dev:platform> go mod vendor
go: launchpad.net/[email protected]: bzr branch --use-existing-dir https://launchpad.net/~niemeyer/gocheck/trunk . in /data/go/pkg/mod/cache/vcs/f46ce2ae80d31f9b0a29099baa203e3b6d269dace4e5357a2cf74bd109e13339: exec: "bzr": executable file not found in $PATH
go: labix.org/v2/[email protected]: bzr branch --use-existing-dir https://launchpad.net/mgo/v2 . in /data/go/pkg/mod/cache/vcs/ca61c737a32b1e09a0919e15375f9c2b6aa09860cc097f1333b3c3d29e040ea8: exec: "bzr": executable file not found in $PATH
go: error loading module requirements

go-dev:platform> go mod tidy
go: launchpad.net/[email protected]: bzr branch --use-existing-dir https://launchpad.net/~niemeyer/gocheck/trunk . in /data/go/pkg/mod/cache/vcs/f46ce2ae80d31f9b0a29099baa203e3b6d269dace4e5357a2cf74bd109e13339: exec: "bzr": executable file not found in $PATH
go: labix.org/v2/[email protected]: bzr branch --use-existing-dir https://launchpad.net/mgo/v2 . in /data/go/pkg/mod/cache/vcs/ca61c737a32b1e09a0919e15375f9c2b6aa09860cc097f1333b3c3d29e040ea8: exec: "bzr": executable file not found in $PATH
go: error loading module requirements

If I edit the go.mod manually to remove the unneeded dependencies go mod tidy and go mod vendor then work, so it seems to indicate that go mod is incapable of fixing broken dependencies automatically.

[mark-rushakoff]

We're discussing separately in influxdata/influxdb#11222, but the dependencies are valid albeit very indirect:

go mod why -m launchpad.net/gocheck labix.org/v2/mgo
go: finding github.com/SermoDigital/jose/jwt latest
go: finding github.com/SermoDigital/jose/jws latest
# launchpad.net/gocheck
github.com/influxdata/platform/nats
github.com/nats-io/nats-streaming-server/server
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/go-msgpack/codec.test
labix.org/v2/mgo/bson
labix.org/v2/mgo/bson.test
launchpad.net/gocheck

# labix.org/v2/mgo
github.com/influxdata/platform/nats
github.com/nats-io/nats-streaming-server/server
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/go-msgpack/codec.test
labix.org/v2/mgo/bson

[dpinela]

@stevenh Since exec: "bzr": executable file not found in $PATH appears in your error messages, it seems that the actual problem is that the dependencies are hosted on Bazaar repos, but you don't have Bazaar installed.

The issue with the go tool trying to fetch replaced dependencies is #27859, which is already fixed at tip and also in the 1.12 beta.

[bcmills]

When filing an issue, please title the issue according to the actual symptom you observed, not a hypothesis as to its cause.

The go tool does assume that all of the go.mod files in the module graph are well-formed, but that seems irrelevant here: the problem as reported in the error message is that your PATH does not include a bzr executable.

From influxdata/influxdb#11222 (comment), it appears that your issue has been resolved. Is there anything remaining to do here?

[stevenh]

Sorry about the title was trying to be helpful and didn’t think downloads fail when brz is missing correctly conveyed this actual problem, as that's an environment setup issue not a tooling issues which was what this was raised to convey.

You are right in this case brz isn’t in the path, we knew that, but it could have been for any number of reasons e.g repo down, moved etc however that doesn’t change the fact its trying to download things it doesn’t need at it assumes all dependent repos have correct go.mod files.

Even when the influx devs kindly fixed the issue, I would expect the standard flow i.e. to just re-run the go get to work, but unfortunately it still failed. I ended up have to manually update our go.mod file as the tools seem to insist on downloading the current listed versions before it processes any updates.

According to feedback from the influx devs the dependency was for tests only which raises another potential issue that there doesn’t seem to be any way in go.mod to flag a dependency as a test only dependency?

Back to the main issue of dependency go.mod files we all know that the world isn’t perfect so issues with dependencies will happen so there needs to be a way for top level projects to recover from this; which sounds like has been by fix for #27859, we'll need to test.

In this case we were lucky, influx was a direct dependency it could have been many levels deep so we could have been stuck waiting for many repos update before we do anything, which seems to be a serious problem with go.mod

The hope this makes sense?

[bcmills]

Even when the influx devs kindly fixed the issue, I would expect the standard flow i.e. to just re-run the go get to work, but unfortunately it still failed.

In general go get makes a minimal change to the module graph, but if the module graph can't be loaded then it can't determine what the minimal change is. (Fixes for #26454 and #26474 might make it more tolerant to errors of this sort, particularly if the get targets are strict upgrades.)

[stevenh]

Thanks @bcmills will re-test this flow when 1.12 is released, which seems like it fixing a lot of issues in this area, and will update the thread based on that.

[bcmills]

According to feedback from the influx devs the dependency was for tests only which raises another potential issue that there doesn’t seem to be any way in go.mod to flag a dependency as a test only dependency?

A test dependency is a dependency. We intend for go mod tidy to leave the go.mod file in a state such that go test all remains reproducible (see #27920 (comment)).

That said, your dependency on launchpad.net/gocheck may be a bug: it seems to be for a test of a package that is not itself in the transitive import graph, which (as far as I am aware) isn't required for go test all.

[bcmills]

a package that is not itself in the transitive import graph, which (as far as I am aware) isn't required for go test all.

I was mistaken: the all target currently includes transitive dependencies-of-tests-of-dependencies, so at least they're consistent.

[stevenh]

So just to confirm my understanding; are you saying that that building any project which is go.mod enabled will have to download the dependencies for all tests of dependencies?

e.g.
mypackage -> dependency1 -> depenency2 (dependency of only tests for dependency1)

If so this sounds like projects are going to suffer from dependency bloat quickly and could even break if said dependencies aren't publicly available.

[bcmills]

Depends on what you mean by “download the dependencies”.

You will need to download the go.mod files for all of the modules in the module graph, which includes modules that provide dependencies for tests of dependencies. However, you will not need to download the source code for those modules unless you actually execute a build from those sources (for example, using go test all).

[stevenh]

Thanks for clarifying, so it sounds like size bloat wont be a direct issue then, but this could still easily lead to issues when test only dependencies aren't available publicly or there's some other issue downloading go.mod files for those dependencies.

Do you think it would make sense to make said failures a warning and not fatal error, which they appear to be now?

[bcmills]

Do you think it would make sense to make said failures a warning and not fatal error, which they appear to be now?

No, they cannot be warnings. In general we do not know how the missing modules might impact the selected versions of other modules in the build graph, so if we just drop those modules we might (more-or-less silently) end up with a different configuration from what we expected for the given version.

Reproducibility is a critical feature of modules, so we shouldn't degrade to a non-reproducible configuration without explicit user intervention.

[stevenh]

Do you see any solution to the specific problem of test only dependency causing failures such as this?

Pretty much every other package manager has the concept of separate runtime and build / test dependencies. Which when translated to golang, as a development language, would mean separate build and test dependencies; without which there is a real risk of 3rd party dependencies resulting in failure as demonstrated by what we saw here.

Other areas of the golang tool chain clearly know the difference between normal code and test code so it’s seems reasonable to continue this.

With this is mind I believe we need a clear method of mitigating such an issue if the tool is to be successful and not cause frustration that would prevent adoption.

[bcmills]

The solution currently on the roadmap is to provide a module proxy that can cache and serve go.mod files, so most users will be able to fetch those directly from the proxy (via HTTPS) rather than installing more esoteric tools (like bzr) or relying on origin servers that may or may not be reliable.

We're certainly keeping an eye on issues surrounding test dependencies, but at the moment we do not plan to separate test (or tool) dependencies from other general module dependencies.

[bcmills]

As far as I can tell there are no remaining direct actions to take for this issue.

We can revisit the story around transitive dependencies once we see how the use of module proxies shakes out.