Skip to content
This repository has been archived by the owner on Jul 15, 2023. It is now read-only.

GOPROXY environment not being passed #2026

Closed
lrstanley opened this issue Oct 18, 2018 · 25 comments
Closed

GOPROXY environment not being passed #2026

lrstanley opened this issue Oct 18, 2018 · 25 comments

Comments

@lrstanley
Copy link

I'm not use to running on Windows, so excuse me if it's something specific I have to do on Windows.

Using the latest version of vscode-go, 0.6.92, I am seeing that the builtin functionality to install the analysis tools, the GOPROXY environment variable which is globally defined on windows, (and passed to vscode), isn't being passed when this library executes go.exe. When I start a terminal inside of vscode, it gets picked up as expected..

PS C:\Users\liam\code\my-package> go env
[...]
set GOPROXY=http://TRUNCATED.internal.com:3000
[...]

Trying to download modules through the Athens proxy, it is still throwing certificate errors when hitting Github (corp firewall restrictions). When doing it through the terminal, it downloads just fine.

Is there a way to get the go plugin to run 'go env' to see what environment variables it's getting passed?

Would also be worth while to add an additional setting for this as well (to override GOPROXY, much like GOROOT and GOPATH) -- I expect with GOPROXY making it much easier for corporations with restricted firewalls to use Go, this will be a frequently requested feature.

@ramya-rao-a
Copy link
Contributor

Is there a way to get the go plugin to run 'go env' to see what environment variables it's getting passed?

Running go env in the terminal should give the same result as when it is run by the plugin.
To see the environment variables the plugin gets, run Help -> Toggle Developer Tools -> Console -> process.env

Speaking of proxies, can you see if anything in https://code.visualstudio.com/docs/setup/network helps?

@lrstanley
Copy link
Author

What I see in process.env:
image

So, the plugin is getting the environment variable, but it's not passed through to the child process, go.exe.. as running the same command returns the expected output when GOPROXY is missing.

(this fails with ssl errors due to corp firewall):

Installing 1 tool at C:/Users/liam/go\bin
  goreturns

Installing github.com/sqs/goreturns FAILED

1 tools failed to install.

goreturns:
Error: Command failed: C:\Go\bin\go.exe get -u -v github.com/sqs/goreturns
github.com/sqs/goreturns (download)
# cd .; git clone https://github.com/sqs/goreturns C:\Users\liam\go\src\github.com\sqs\goreturns
Cloning into 'C:\Users\liam\go\src\github.com\sqs\goreturns'...
fatal: unable to access 'https://github.com/sqs/goreturns/': SSL certificate problem: unable to get local issuer certificate
package github.com/sqs/goreturns: exit status 128
github.com/sqs/goreturns (download)
# cd .; git clone https://github.com/sqs/goreturns C:\Users\liam\go\src\github.com\sqs\goreturns
Cloning into 'C:\Users\liam\go\src\github.com\sqs\goreturns'...
fatal: unable to access 'https://github.com/sqs/goreturns/': SSL certificate problem: unable to get local issuer certificate
package github.com/sqs/goreturns: exit status 128

When I go to the terminal in VSCode and run the exact same command, it properly sees GOPROXY, and uses it for the go modules:

PS C:\Users\liam\code\pipeline-gen> C:\Go\bin\go.exe get -u -v github.com/sqs/goreturns
go: finding github.com/sqs/goreturns latest
PS C:\Users\liam\code\pipeline-gen> goreturns
usage: goreturns [flags] [path ...]
  -b    remove bare returns
[...more output...]

In regards to VSCodes networking related proxy settings, this is unrelated. GOPROXY is something new with the new Go modules support -- it allows you to pull dependencies from a central location. Trying to use the go proxy as a normal http proxy will completely fail.

Here is the documentation for GOPROXY, pulled from one of my linux instances:

$ go help goproxy
The go command by default downloads modules from version control systems
directly, just as 'go get' always has. The GOPROXY environment variable allows
further control over the download source. If GOPROXY is unset, is the empty string,
or is the string "direct", downloads use the default direct connection to version
control systems. Setting GOPROXY to "off" disallows downloading modules from
any source. Otherwise, GOPROXY is expected to be the URL of a module proxy,
in which case the go command will fetch all modules from that proxy.
No matter the source of the modules, downloaded modules must match existing
entries in go.sum (see 'go help modules' for discussion of verification).

A Go module proxy is any web server that can respond to GET requests for
URLs of a specified form. The requests have no query parameters, so even
a site serving from a fixed file system (including a file:/// URL)
can be a module proxy.

The GET requests sent to a Go module proxy are:

GET $GOPROXY/<module>/@v/list returns a list of all known versions of the
given module, one per line.

GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata
about that version of the given module.

GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file
for that version of the given module.

GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive
for that version of the given module.

To avoid problems when serving from case-sensitive file systems,
the <module> and <version> elements are case-encoded, replacing every
uppercase letter with an exclamation mark followed by the corresponding
lower-case letter: github.com/Azure encodes as github.com/!azure.

The JSON-formatted metadata about a given module corresponds to
this Go data structure, which may be expanded in the future:

    type Info struct {
        Version string    // version string
        Time    time.Time // commit time
    }

The zip archive for a specific version of a given module is a
standard zip file that contains the file tree corresponding
to the module's source code and related files. The archive uses
slash-separated paths, and every file path in the archive must
begin with <module>@<version>/, where the module and version are
substituted directly, not case-encoded. The root of the module
file tree corresponds to the <module>@<version>/ prefix in the
archive.

Even when downloading directly from version control systems,
the go command synthesizes explicit info, mod, and zip files
and stores them in its local cache, $GOPATH/pkg/mod/cache/download,
the same as if it had downloaded them directly from a proxy.
The cache layout is the same as the proxy URL space, so
serving $GOPATH/pkg/mod/cache/download at (or copying it to)
https://example.com/proxy would let other users access those
cached module versions with GOPROXY=https://example.com/proxy.

@ramya-rao-a
Copy link
Contributor

So, the plugin is getting the environment variable, but it's not passed through to the child process, go.exe..

The extension host process gets the same environment variables as what you saw in the console. The same are passed to the child process that runs go.exe.

Can you run the extension from source to see the exact environment variables being passed? Its is pretty straighforward.

@lrstanley
Copy link
Author

I'm sorry for the delay, I was in SF for hashiconf. I didn't run from source, as I don't have nodejs, and corp laptop makes it hard. I just modified the transpiled source to include the following two lines inside roughly the same place you mentioned (Promise.resolve([])).then):

        goStatus_1.outputChannel.appendLine('process.env: ', process.env['GOPROXY']);
        goStatus_1.outputChannel.appendLine('envForTools: ', envForTools['GOPROXY']);

and the output is:

Installing 7 tools at C:\Users\liam\go\bin
  gopkgs
  go-outline
  go-symbols
  guru
  gorename
  dlv
  godoc

Installing github.com/uudashr/gopkgs/cmd/gopkgs FAILED
Installing github.com/ramya-rao-a/go-outline FAILED
Installing github.com/acroca/go-symbols FAILED
Installing golang.org/x/tools/cmd/guru FAILED
Installing golang.org/x/tools/cmd/gorename FAILED
Installing github.com/derekparker/delve/cmd/dlv FAILED

process.env: 
envForTools: 
7 tools failed to install.

gopkgs:
Error: Command failed: C:\Go\bin\go.exe get -u -v github.com/uudashr/gopkgs/cmd/gopkgs
github.com/uudashr/gopkgs (download)
# cd .; git clone https://github.com/uudashr/gopkgs C:\Users\liam\go\src\github.com\uudashr\gopkgs
Cloning into 'C:\Users\liam\go\src\github.com\uudashr\gopkgs'...
fatal: unable to access 'https://github.com/uudashr/gopkgs/': SSL certificate problem: unable to get local issuer certificate
package github.com/uudashr/gopkgs/cmd/gopkgs: exit status 128
github.com/uudashr/gopkgs (download)
# cd .; git clone https://github.com/uudashr/gopkgs C:\Users\liam\go\src\github.com\uudashr\gopkgs
Cloning into 'C:\Users\liam\go\src\github.com\uudashr\gopkgs'...
fatal: unable to access 'https://github.com/uudashr/gopkgs/': SSL certificate problem: unable to get local issuer certificate
package github.com/uudashr/gopkgs/cmd/gopkgs: exit status 128
[...lots more failures...]

As you can see, it's printing nothing after the prefix..:

process.env: 
envForTools: 

And yet, when I open debug tools in vscode, and print the environment variable, it's there:

image

What I'm wondering is if it's being stripped out before it's being passed to the extension. Maybe vscode strips anything that has proxy in the name, as it assumes it's related to the network proxy settings? In any case, it doesn't look like it's getting passed to the go executable, even though it's being passed to vscode, and vscode itself sees it.

Can you replicate this? All it takes is to run the athens proxy docker container temporarily, and update your GOPROXY environment to point to it, so the athens proxy caches all of the downloaded go v1.11.x modules.

@lrstanley
Copy link
Author

Any update to this?

@ramya-rao-a
Copy link
Contributor

Hey @lrstanley,

Apologies, I didn't have time to try out athens myself. But by what you are sharing it looks like VS Code is getting the environment variable (because you see it in the integrated terminal), but it isn't passing it on to the extension host process that runs all the VS Code extensions (this is why you don't see it when you tried to print it out when you changed the source code).

I am including @arschles in this thread who uses athens as well as VS Code.

@arschles Have you seen anything like this before?

@lrstanley One last thing I would like you to try is to run VS Code directly from the command line. Just type code (or code-insiders if you are using the Insiders version of VS Code) from the command line. This ensures that VS Code inherits all the environment variables from the command line.

@lrstanley
Copy link
Author

Yes, I have also invoked it from the command line, from the same environment where go env shows the GOPROXY environment variables (and go get <etc> requests through Athens work).

For @arschles -- note that this is solely for the portion where the vscode-go plugin attempts to go get dependencies. This is not related to using the embedded terminal to go get dependencies.

@ramya-rao-a
Copy link
Contributor

Yes, I have also invoked it from the command line, from the same environment where go env shows the GOPROXY environment variables (and go get requests through Athens work).

And this is the same environment where echo %GOPROXY% is the same as go env GOPROXY?

@lrstanley
Copy link
Author

Using Git Bash is how I usually invoke it on windows:

liam@hostname MINGW64 ~
$ echo $GOPROXY
http://TRUNCATED.internal.com:3000

liam@hostname MINGW64 ~
$ go env GOPROXY
http://TRUNCATED.internal.com:3000

Using command prompt though:

C:\Users\liam>echo %GOPROXY%
http://TRUNCATED.internal.com:3000

C:\Users\liam>go env GOPROXY
http://TRUNCATED.internal.com:3000

@ramya-rao-a
Copy link
Contributor

Ok, so the "GOPROXY" environment variable is available in your Git Bash and command prompt, but when you run VS Code from either, it doesnt pass the same to the extension host.

I have logged an upstream bug microsoft/vscode#65113 in VS Code. Lets see if someone there can help

@lrstanley
Copy link
Author

As a followup to what I mentioned in microsoft/vscode#65113; I think there may still be some issues in vscode-go, not vscode itself.

@ramya-rao-a, you told me to use:

process.env['GOPROXY'] and envForTools['GOPROXY']

I made the modification in the generated plugin at $HOME/.vscode/extensions/ms-vscode.go-0.8.0/out/src/goInstallTools.js. This time around, I am doing the modifications to the same file.

I just tested again, trying to print all of process.env using goStatus_1.outputChannel, and it was empty as well. Testing console.log(process.env) prints everything to the console properly. seems passing non-strings to goStatus_1.outputChannel doesn't work.. I see GOPROXY defined in console.log(process.env).

Using goStatus_1.outputChannel.appendLine('process.env', process.env['GOPROXY']) still doesn't work, not sure why. Now that I think about it, it probably only takes one argument, and we're trying to pass it multiple.

In any case, I think I know what's going on. It looks like you're explicitly setting:

envForTools['GO111MODULE'] = 'off';

Which.. GOPROXY doesn't work when module support is disabled. This is more of an issue with GOPROXY than with this plugin, since enabling module support would mean the tools would be added to the go.mod file.. which I don't think is what we'd want. However, since I am in a corporate environment, and we can't easily go get things, I don't think it's the worst to have these things dumped into the dependency list of the given project (and... downloaded over and over again for multiple projects).

@lrstanley
Copy link
Author

I suspect people using GOPROXY are in corporate environments.. and as such, would be in the same situation. Maybe if GOPROXY is set (and not set to off), we don't set GO111MODULE=off (or maybe set to on?)

Or, maybe add an option to the extension to Download extension dependencies with go module support or similar?

@ramya-rao-a
Copy link
Contributor

Good to know that the env variables are getting passed around properly.

So to sum this up, since I disable module support when running go get for the Go tools, GOPROXY doesn't work which then lands you with proxy issues in corporate environment.

I am in a corporate environment, and we can't easily go get things,

There have been other users who had these proxy issues in corporate environment before. Using the http.proxy setting has helped out in those cases. Can you give that a try?

I don't think it's the worst to have these things dumped into the dependency list of the given project (and... downloaded over and over again for multiple projects).

We really don't want to get into a place where the tools need to be downloaded into more than 1 location

@lrstanley
Copy link
Author

There have been other users who had these proxy issues in corporate environment before. Using the http.proxy setting has helped out in those cases. Can you give that a try?

Unfortunately, we cannot setup a proxy in this fashion. The only way is via a controlled environment through something like Athens for our usecase.

We really don't want to get into a place where the tools need to be downloaded into more than 1 location

Which makes sense,, however, with the changes to go modules, GOPATH is being deprecated (the words "deprecated" show up in golang/go#24301, I'm not sure to what extent -- I expect support to still exist, at least for Go 1.x.x). A good amount of work has gone into things like cache being moved out of $GOPATH, as well as $GOPATH/pkg being moved into the new build cache locations (iirc, this is where they are now).

Moving forward, might be better to focus on the use of $GOBIN so things don't have to be downloaded in multiple locations.

I'm not saying what I recommended is the best option, but I do think it will become more of an issue, and warrants more discussion on alternatives.

@lrstanley
Copy link
Author

Filtering through the proposal a bit more, I do see:

The plan, subject to proposal approval, is to release module support in Go 1.11 as an optional feature that may still change. The Go 1.11 release will give users a chance to use modules “for real” and provide critical feedback. Even though the details may change, future releases will be able to consume Go 1.11-compatible source trees. For example, Go 1.12 will understand how to consume the Go 1.11 go.mod file syntax, even if by then the file syntax or even the file name has changed. In a later release (say, Go 1.12), we will declare the module support completed. In a later release (say, Go 1.13), we will end support for go get of non-modules. Support for working in GOPATH will continue indefinitely.

Primarily:

Support for working in GOPATH will continue indefinitely.

So that is good at least -- though, there is still a usecase for $GOBIN here.

@ramya-rao-a
Copy link
Contributor

ramya-rao-a commented Jan 1, 2019

Does running go get on all the dependent Go tools using Athens work even for tools that do not support modules yet?

The reason why I disabled GO111MODULES when running go get was #1919.

Wont keeping GO111MODULES on result in the tools that dont use modules to fail the go get process?

@lrstanley
Copy link
Author

I was able to install all of them through Athens, if I ran the go get's myself (so I could keep GO111MODULE enabled). They should without using Athens as well. I think what the user in #1919 was experiencing is a bug in Go, as to why the go get's were failing:

golang/go#27215 (primarily this one)
golang/go#29363
golang/go#29484

@ramya-rao-a
Copy link
Contributor

Why is that you dont see the same issue as #1919 when using Athens and GO111MODULE enabled?

@lrstanley
Copy link
Author

lrstanley commented Jan 2, 2019

My guess is that go get in module mode is expecting something from e.g. Github (in terms of branches or tags maybe), that Github/most repos don't have, but Athens returns those as some kind of default maybe? Could be as simple as interpreting /... wrong in some cases with the response (not the request).

When I see:

go get golang.org/x/tools/...: no matching versions for query "latest"

The first thing this makes me think there is a mismatch with tag/branch names, and what it finds for latest.. only a guess though. Modules should be backwards compatible with things that haven't been converted to a module yet, even for cases where the go get call includes /... based queries.

I've also used Artifactory's Golang support, and I believe it suffered from issues with ... as well.


Snippet from the docs: https://tip.golang.org/cmd/go/#hdr-Module_aware_go_get

Note that package patterns are allowed and are expanded after resolving the module versions. For example, 'go get golang.org/x/perf/cmd/...' adds the latest golang.org/x/perf and then installs the commands in that latest version.

And, according to golang/go#27215 (comment), what the docs say doesn't actually work!

Here is that example from the doc from scratch:

$[...]
$ go get golang.org/x/perf/cmd/...
go: finding golang.org/x/perf/cmd/... latest
go: finding golang.org/x/perf/cmd latest
go: finding golang.org/x/perf latest
go get golang.org/x/perf/cmd/...: no matching versions for query "latest"

@xgfone
Copy link

xgfone commented Sep 9, 2019

I have also met with this trouble.

The environment GOPROXY didn't work.

Installing 1 tool at C:\Users\xgf\go\bin
  gocode-gomod

Installing github.com/stamblerre/gocode FAILED

1 tools failed to install.

gocode-gomod:
Error: Command failed: C:\Go\bin\go.exe get -u -v -d github.com/stamblerre/gocode
github.com/stamblerre/gocode (download)
package golang.org/x/tools/go/packages: unrecognized import path "golang.org/x/tools/go/packages" (https fetch: Get https://golang.org/x/tools/go/packages?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
github.com/stamblerre/gocode (download)
package golang.org/x/tools/go/packages: unrecognized import path "golang.org/x/tools/go/packages" (https fetch: Get https://golang.org/x/tools/go/packages?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
OS: Windows 10 X86-64
VSCode: 1.38.0(commit: 3db7e09f3b61f915d03bbfa58e258d6eee843f35)
vscode-go: 0.11.4
Shell: git-bash-2.16.1.windows.1
$ go version
go version go1.13 windows/amd64

$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\xgf\AppData\Local\go-build
set GOENV=C:\Users\xgf\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\xgf\go
set GOPRIVATE=
set GOPROXY=https://gocenter.io
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=D:\gopath\src\gitlab.sndaurl.cn\BackendDev\cwalarm\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\xgf\AppData\Local\Temp\go-build045237998=/tmp/go-build -gno-record-gcc-switches

When I do it myself, however, it works.

$ go get -u -v -d github.com/stamblerre/gocode
go: finding github.com/stamblerre/gocode latest
go: downloading github.com/stamblerre/gocode v0.0.0-20181128172141-22843d89bc5a
go: extracting github.com/stamblerre/gocode v0.0.0-20181128172141-22843d89bc5a
go: finding golang.org/x/tools latest

@stamblerre
Copy link
Contributor

Tool installation should work as expected with the beta version of the extension, if anyone would like to try it out and confirm.

@felixhao28
Copy link

@stamblerre I just tested the beta.5 version in your link and it works wonders.

@ramya-rao-a
Copy link
Contributor

@lrstanley & @xgfone, Can you try the latest beta version of this extension and see if you still have issues?

@randalvance
Copy link

Hi @ramya-rao-a, I have tried version 0.11.8-beta.3 but it still won't recognize GOPROXY setting.

@ramya-rao-a
Copy link
Contributor

@randalvance Can you open a new issue, share your Go related settings and provide details on the issue you are seeing?

@vscodebot vscodebot bot locked and limited conversation to collaborators Dec 5, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants