cmd/go: allow "go get" when outside a module in module mode

[smyrman]

UPDATE: This is a feature request to be able to install a program/main package to a user's home directory (basically $GOBIN) outside of a module context.

Background

While dep, glide and other tools provide a way of fetching project dependencies, part of what they are missing which I think belongs in any official go get with versioning, is a way to install programs (tools, CLI, or a binary compiled from any main package) into a user's PATH (e.g. GOBIN, which may be relevant, even if GOPATH get's deprecated).

---

Please answer these questions before submitting your issue. Thanks!

What did you do?

$ vgo get github.com/golang/dep/cmd/[email protected]

Example command only; but this would be the closest to how things currently work with go get.

What did you expect to see?

dep binary (in this case) installed to either $GOBIN, $GOPATH[0]/bin or $HOME/go/bin, in that order.

What did you see instead?

$ vgo get github.com/golang/dep/cmd/[email protected]
cannot determine module root; please create a go.mod file there

System details

go version go1.10 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/smyrman/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/smyrman/.local"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/sj/d5g1dfkd3g75ccz83k505v3r0000gn/T/go-build848907380=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""
GOROOT/bin/go version: go version go1.10 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.10
uname -v: Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.13.3
BuildVersion:	17D102
lldb --version: lldb-900.0.64
  Swift-4.0

[kardianos]

@smyrman I think this is a genuine concern, but it isn't just about pulling down a specific version of a program; it is using vgo outside the context of a module. You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.

Maybe:
x/vgo: no longer supports installing an executable from outside of a module
?

[smyrman]

You'll get the same error if you omit the versuib @v0.0.0 specifier as well. vgo thinks you are trying to pull in a dependency; you are trying to install a program into GOBIN.

@kardianos, thanks for the response. I am actually not concerned about the error message being unclear. This is meant as a feature request to (somehow) still be able to install binaries to a user's PATH, no matter what your current working directory is, similar to what we can with go get today. Even better, the possibility to install a program at as specific version. I believe a language package manger should not be just about dependencies.

I am not to particular about the syntax, but maybe there could be a flag like -home, -user or even -gopath, that allowed installing stuff to either $GOBIN or $GOPATH/bin. This would be similar to how e.g. pip works, for Python, to mention one, where if you use e.g. venev or virtualenv for a per-project experience, you can still add an option to pip to install tools to a user's home:

• https://pip.pypa.io/en/stable/user_guide/#user-installs
• https://docs.python.org/3/install/index.html#alternate-installation-the-user-scheme

While OS package managers are good at doing more or less the same thing (not to a user's home of course, but system wide), there is often a delay for programs to be packaged, and language power users may prefer to use a language package manger for installing his/hers software, while also being able to pull down unreleased software, as we do today with go get. Users operating on a shared system may have a similar need to install things in their home directories.

Further, even though vgo deprecates GOPATH/GOBIN for project development, it's not necessarily right to deprecate GOPATH for other potentially valid use-cases, but that's probably another discussion, where the only thing I am interested in through this issue, is the GOBIN part.

[smyrman]

PS! I am also not concerned about this issue being solved right away, but it would be really cool to solve it before vgo goes main-line, if that's what going to happen.

[smyrman]

Updated the description.

[rsc]

$ vgo get github.com/golang/dep/cmd/[email protected]
cannot determine module root; please create a go.mod file there

This clearly must work eventually. The thing I'm not sure about is exactly what this does as far as the version is concerned: does it create a temporary module root and go.mod, do the install, and then throw it away? Probably. But I'm not completely sure, and for now I didn't want to confuse people by making vgo do things outside go.mod trees. Certainly the eventual go command integration has to support this.

Note that installs already happen to $GOBIN or else $GOPATH/bin (so default $HOME/go/bin), not the modue tree.

Will mark NeedsDecision because we still need to work out how to do this, but yes absolutely the command must work.

[smyrman]

does it create a temporary module root and go.mod, do the install, and then throw it away? Probably.

This is a reasonable approach, and have some obvious benefits, such as the result of go get github.com/golang/dep/cmd/[email protected] not being affected by other tools installed on the system. To make a decision easier, I will supply an alternative approach and highlight some problems with it:

I suppose an alternative would be to have a "default" module that is used when a flag is provided to the go get command.

E.g. one could picture a default module in $GOPATH[0]/src/u (first entry of GOPATH) that is automatically created and used when the -user flag is given to the go get command line. This would let go get -user github.com/golang/dep/cmd/[email protected] be semantically equivalent to:

#!/bin/sh
set -e
_module_path=$(echo $GOPATH | awk -F':' '{ print $1}')/src/u
mkdir -p ${_module_path}
cd  ${_module_path}
test -f go.mod || echo 'module "u"' > go.mod
go get  github.com/golang/dep/cmd/[email protected]

Some benefits

• One easy flag to access a user-specific module for user-installed cross-project binaries.
• The module path is preserved on disk, so power users could sync the folder via tools like Git or Dropbox to keep their tools with them at specific versions across machines.

Major drawback

• Tool installation (e.g. installation of dep at a specific version as used in the example above), may be affected by (minimal) version selection of overlapping dependencies with other tools, leading to potentially weird bugs.

Conclusion

For me, I belive the drawback outweighs any benefit, and a temporary module root / mod.go root is the semantically correct solution; perhaps implementation wise the temporary module root and go.mod file only exists in memory.

[rsc]

This isn't blocking Go 1.11. The main reason being that if GO111MODULE=auto (the default) then we can't reasonably do anything with modules when outside a go.mod tree. That's the opt-in. And inside a go.mod tree this already works.

[earthboundkid]

I agree that this doesn't block Go 1.11, but I do think it should be a blocker for 1.12. There are a lot of repos that have READMEs saying "just run go get github.com/me/myrepo to install". That should continue to work once GO111MODULE changes to "on".

[bcmills]

Agreed. I've marked it as a release-blocker for 1.12.

[nhooyr]

@bcmills

Regarding @cespare's comment

I'm not sure I totally follow the entire lengthy discussion here, but it sounds like even with this change we would still need to do cd /somewhere/outside/of/a/module && go get blah.

This seems to be a major missing feature. Its nice things work concurrently now but sometimes you're in your project directory and want to download a tool. What should I do in this case? Just cd to a tmp dir?

[nhooyr]

Nvm.

To anyone wondering about my question in the future: https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md