cmd/go: define HTTP authentication extension mechanism

[draftcode]

NOTE: The accepted proposal is #26232 (comment).

---

Problem

The custom import path mechanism (?go-get=1 and meta tag) works for public URLs, but it doesn't work for auth required URLs. This is because when go-get fetches the URL with ?go-get=1, it uses net/http.DefaultClient, and it doesn't know the credentials it needs to access the URL. A user cannot run go get against private source code hosting service because of this.

Goal

Make go get git.mycompany.com/private-repo work, where https://git.mycompany.com/private-repo requires authentication.

Idea 1 (credential helper)

Introduce a credential helper mechanism like git-credential-helpers. A user specifies a path to a binary via GOGET_CREDENTIAL_HELPER and go get executes that with the import path as an argument. The credential helper binary returns HTTP headers (like "Authorization: Basic blah\n", and go get adds these headers when it tries to fetch go-get=1 URLs.

• PRO: Straightforward solution to the problem description.
• CON: Supporting multiple credential helpers becomes complicated. Git's credential helper mechanism supports multiple credential helper, and Git runs each in order. This sometimes makes an unexpected behavior that is hard to debug.

Idea 2 (go-get helper)

Introduce a custom source code fetching mechanism. When go get needs to fetch the source code for the import path git.mycompany.com/private-repo, it looks for the binary go-get-git.mycompany.com based on the host name of the import path. When such binary exists in $PATH, it executes that binary with the import path and a destination in $GOPATH (for example, go-get-git.mycompany.com git.mycompany.com/private-repo $GOPATH/src/git.mycompany.com/private-repo). The binary is responsible for fetching the source code to the specified $GOPATH location.

• PRO: As a side effect, this make go get work with VCSs other than git/hg/svn/bzr.
• CON: I'm not sure how this works with go get -f or go get -insecure.

[draftcode]

Related issues: #16315 #11032

[rsc]

The original vgo prototype just read $HOME/.netrc. I'd rather do something like that than shell out to a whole separate program.

[draftcode]

.netrc works for HTTP basic auth with unlimited lifetime credentials. But it doesn't work for other types, like:

• OAuth2 access tokens. OAuth2 access tokens usually have a limited lifetime, and it needs to be issued dynamically. Because of this, this cannot be in a static file.
• Cookie based auth. netrc as a format is not a format that represent an HTTP cookie. Alternatively, Netscape's cookie file format (the same format as .gitcookies) can be used. This cannot support dynamically generated credentials as well. We actually have Git repositories that are behind dynamically generated cookie-based auth at Google (BeyondCorp at Google: https://cloud.google.com/beyondcorp/).

[rsc]

Note that even "access denied" pages can have meta tags - go get does not require a 200 response, exactly for this kind of thing. So git.mycompany.com could serve the tags unauthenticated as one workaround.

Are there any more general "HTTP auth credential helpers" in the world besides git-credential-helper? How do other systems deal with this? If there's something standard to hook into (like .netrc) then that's better than designing our own.

[draftcode]

If the access denied page contains the meta tag, it can be used as a prober. Let's say git.mycompany.com has an not-yet announced new product, collaborating with another company other-tech-company.com, and they host a Git repository at git.mycompany.com/ng-product/collab-other-tech-company. An attacker can probe if a repository exists by looking at the go-get's meta tag, and they can learn mycompany.com will have a business relationship other-tech-company.com to create a new product (and maybe benefit from this insider info).

As far as I know, there's no generic mechanism that majority of people use for storing and obtaining credentials through API. The closest thing I can think of is ssh-agent. Or if it's OS X, OS X Keychain.

[rsc]

It's only a prober if you check that the thing exists before serving the meta-tag.

$ curl 'https://rsc.io/asdgihopajdfklgadfglhifsdfj?go-get=1'
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="rsc.io/asdgihopajdfklgadfglhifsdfj git https://github.com/rsc/asdgihopajdfklgadfglhifsdfj">
<meta http-equiv="refresh" content="0; url=https://godoc.org/rsc.io/asdgihopajdfklgadfglhifsdfj">
</head>
<body>
Redirecting to docs at <a href="https://godoc.org/rsc.io/asdgihopajdfklgadfglhifsdfj">godoc.org/rsc.io/asdgihopajdfklgadfglhifsdfj</a>...
</body>
</html>
r$ 

[rsc]

Clearly we could use a better idea here but I'd like to have just one thing that's standard instead of inventing our own (that is, I want something like .netrc).

[draftcode]

It's only a prober if you check that the thing exists before serving the meta-tag.

I cannot interpret this. The problem is that an attacker can use this to extract the repository existence by using this meta-tag. If the page checks if the repository exists and change the response, it can be used as a prober.

[rsc]

If the page checks if the repository exists and change the response, it can be used as a prober.

Then don't check, and don't change the response.

[bradfitz]

Are there any more general "HTTP auth credential helpers" in the world besides git-credential-helper?

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

https://github.com/GoogleCloudPlatform/docker-credential-gcr

[draftcode]

Then don't check, and don't change the response.

So the page should not do any auth, and should not contain meta tag? I thought you're suggesting to add a meta tag and in the next response you're saying not to add meta tag?

[draftcode]

An example leakage: https://play.golang.org/p/rCYzVWQSQni

We want to avoid this. By adding a meta-tag to the response as rsc suggests, the repository existence is leaked.

[bradfitz]

@draftcode, I assumed what was meant as you should advertise it for all URLs, including /android/device/g000gle/foo-bar-not-exist , regardless of whether it actually exists.

[draftcode]

I assumed what was meant as you should advertise it for all URLs, including /android/device/g000gle/foo-bar-not-exist , regardless of whether it actually exists.

Note that the third part in meta tag should be a git repository URL. This means, if a repository exists, the third part should be a valid Git repository path. In the example, I used example.com/android/device/google/marlin-kernel/foo/bar as an example package path. The valid meta tag for this is <meta name="go-import" content="example.com git http://example.com/android/device/google/marlin-kernel">, not <meta name="go-import" content="example.com git http://example.com/android/device/google/marlin-kernel/foo/bar">. Because of this, I can tell whether a repository exists by adding paths that wouldn't exist. When I make a request, if meta tag's returned path is modified, it means the repository exists.

[draftcode]

An example that for this: https://play.golang.org/p/OWb6zRUD02r

[rsc]

@draftcode I am saying that if you pick a trivial rule like "example.com/x/y/z/w redirects to the repo at git.example.com/x/y", then you can implement it with no Auth check and no repo validity check, so that it responds to any possible x/y/z/w with the same answer. Then there is no leakage. That was the point of my curl example above: there is no actual package at rsc.io/asdgihopajdfklgadfglhifsdfj but the curl still reads back meta tags for it.

If you need example.com/x/y/z/w to redirect to different git servers based on the exact values of x,y,z,w then that's more difficult of course and leakage is hard to avoid.

This is all just a suggested workaround. What I'd really like is to find out about some plausibly general (not git-specific, not docker-specific) semi-standard way to interact with credential helpers. Maybe none exists and we have to come up with something. But that would be my last choice.

[draftcode]

If you need example.com/x/y/z/w to redirect to different git servers based on the exact values of x,y,z,w then that's more difficult of course and leakage is hard to avoid.

OK. We've considered that option when we investigated how GitHub deals with this, and found out that GitHub won't allow slashes in a repository name, so they can do what you've said. We cannot do that, so filed this bug.

What I'd really like is to find out about some plausibly general (not git-specific, not docker-specific) semi-standard way to interact with credential helpers. Maybe none exists and we have to come up with something. But that would be my last choice.

So far, I've shown what Git does for a similar problem. @bradfitz showed what Docker and GCP does for a similar problem (ADC now works only for service accounts, so it's a bit different). If there's a some standard way to get a cred, considering the size of these tools' community, there should be some implementation of that, but it seems there's no such thing. In fact, Docker created a credential helper mechanism following what Git did. From these, such standard, if exists, is something that Git and Docker communities at least are not aware of and are not using. @rsc, @bradfitz, and I are also not aware of such generic way that would be called as "semi-standard", it seems.

[rsc]

Given that there doesn't seem to be an agreed-upon standard, I guess the next step is for someone to propose a design that Go should use. I looked at git credential helpers but stopped looking when I saw bash snippets.

[draftcode]

Questions on the requirements:

• My impression is that @rsc strongly prefers using a file instead of executing a command and using stdin/stdout. Is this a hard requirement?
  • If this is a requirement, this means that Go cannot support OS-standard credential management mechanisms.
  • Also this makes it hard to protect credentials. If this is done in stdin/stdout, the credential manager side can do a necessary re-auth.
• This is just "defining a protocol between Go's toolchain and a user-defined credential manager", right? We do not want to create a password manager like LastPass for this purpose.
• With a user-defined credential manager, users should be able to:
  • Add an HTTP header (including cookies)
  • Add an SSL client certificate
  • Anything else?

[josharian]

My impression is that @rsc strongly prefers using a file instead of executing a command and using stdin/stdout. Is this a hard requirement?

He objected to bash snippets. It is possible (I do not know) that executing a binary rather than a shell might sit better with him.

If it were indeed command execution (a la EDITOR), then e.g. those of us who use 1password could use their command line support. I think macOS keychain has similar support.

[rsc]

Command-line execution seems necessary, since you want to lock things down and give cmd/go access to just one password, not your whole password set. Josh, what did you have in mind? Want to propose a starting point design?

[dmitris]

It would be great if we could achieve the stated above goal ("Make go get git.mycompany.com/private-repo work, where https://git.mycompany.com/private-repo requires authentication") without forcing the users to perform custom modifications to their ~/.netrc or ~/.gitconfig files (or installing extra binaries) - at least in the (I believe) common cases of GitHub [Enterprise] with SSH authentication (from the ssh agent).

Assuming the server returns a meta tag such as <meta name="go-import" content="git.mycompany.com/org/repo git https://git.mycompany.com/org/repo.git">, could the go tool try the following:

• shell out git ls-remote -q https://git.mycompany.com/org/repo.git first (as now)
• if it gets a response with an auth prompt: Username for 'https://git.mycompany.com':, try next the command with the corresponding ssh URL: git ls-remote -q ssh://[email protected]/org/repo.git - and if it works, remember / cache that protocol for other git operations with the server (at least, within the same go invocation). (The git user must be standard for GitHub as it is indicated on https://help.github.com/articles/testing-your-ssh-connection/ and https://help.github.com/enterprise/2.14/user/articles/testing-your-ssh-connection/). This "retrial" step can be limited to the cases where server replies with Server: Github.com HTTP header ([1], [2]).

I believe this would make possible to use git with the SSH auth without having everyone to add the insteadOf stanzas in ~/.gitconfig on every server / VM or putting passwords in ~/.netrc etc. (To avoid this, we currently have to use a redirect server on an equivalent of go.mycompany.com that sends back <meta name="go-import" content="git.mycompany.com/org/repo git ssh://[email protected]/org/repo"> - but it would be so nice to be able to use the "natural" git.mycompany.com/org/repo as imports / package names instead of go.mycompany.com and forcing everyone to learn and use the git.mycompany.com => go.mycompany.com replacement!) Having all Go users in the company to install an additional binary to bridge go and auth secrets would IMO be even more cumbersome (and the security team will likely consider it as yet another thing to worry about...).

I think it is an incredible strength of the Go tool that you can install and use it "right out of the box" - would love to see that feature and stellar user experience preserved and extended! 😄

[1] https://go-review.googlesource.com/c/go/+/33171 (abandoned; issue https://golang.org/issue/17898)
[2] Russ's comment - #24076 (comment)

[draftcode]

Assuming the server returns a meta tag

@dmitris This proposal is for the servers that cannot return meta tags without an auth (see #26232 (comment)). It seems to me that your problem is not related to this.

[samthanawalla]

go.dev/cl/605275 is submitted.

This adds support for the built in git method as specified in #26232 (comment)

I'm hoping to finish the user provided authentication protocol before the Go 1.24 Release freeze on Nov 21st.

In the meantime, feel free to play around with this at tip and let me know if there are any issues. Cheers

[samthanawalla]

Update: we finished the user provided authentication protocol in time for the 1.24 freeze.

Running go help goauth will describe the protocol more in-depth:

$ go help goauth
GOAUTH is a semicolon-separated list of authentication commands for go-import and
HTTPS module mirror interactions. The default is netrc.

The supported authentication commands are:

off
        Disables authentication.
netrc
        Uses credentials from NETRC or the .netrc file in your home directory.
git dir
        Runs 'git credential fill' in dir and uses its credentials. The
        go command will run 'git credential approve/reject' to update
        the credential helper's cache.
command
        Executes the given command (a space-separated argument list) and attaches
        the provided headers to HTTPS requests.
        The command must produce output in the following format:
                Response      = { CredentialSet } .
                CredentialSet = URLLine { URLLine } BlankLine { HeaderLine } BlankLine .
                URLLine       = /* URL that starts with "https://" */ '\n' .
                HeaderLine    = /* HTTP Request header */ '\n' .
                BlankLine     = '\n' .

        Example:
                https://example.com/
                https://example.net/api/

                Authorization: Basic <token>

                https://another-example.org/

                Example: Data

        If the server responds with any 4xx code, the go command will write the
        following to the programs' stdin:
                Response      = StatusLine { HeaderLine } BlankLine .
                StatusLine    = Protocol Space Status '\n' .
                Protocol      = /* HTTP protocol */ .
                Space         = ' ' .
                Status        = /* HTTP status code */ .
                BlankLine     = '\n' .
                HeaderLine    = /* HTTP Response's header */ '\n' .

        Example:
                HTTP/1.1 401 Unauthorized
                Content-Length: 19
                Content-Type: text/plain; charset=utf-8
                Date: Thu, 07 Nov 2024 18:43:09 GMT

        Note: at least for HTTP 1.1, the contents written to stdin can be parsed
        as an HTTP response.

Before the first HTTPS fetch, the go command will invoke each GOAUTH
command in the list with no additional arguments and no input.
If the server responds with any 4xx code, the go command will invoke the
GOAUTH commands again with the URL as an additional command-line argument
and the HTTP Response to the program's stdin.
If the server responds with an error again, the fetch fails: a URL-specific
GOAUTH will only be attempted once per fetch.

Things left to do:

• add more tests
• update release notes
• a blog post with some examples (if there's demand)

[silverwind]

semicolon-separated

Shouldn't it be comma-separated, like other environment vars like GOPROXY already are? See https://go.dev/ref/mod#environment-variables.

[samthanawalla]

@silverwind
Shouldn't it be comma-separated

I agree that ideally it would have made sense to keep that format but an argument passed to the command could contain commas so we decided on semi-colons.

[skunkworker]

Will these changes that hopefully will be in 1.24 allow for private subgroups to be pulled properly? https://docs.gitlab.com/ee/user/project/use_project_as_go_package.html#authenticate-git-requests-to-private-subgroups

I would hope that gitlab.com/namespace/subgroup/go-module would work without an issue (and properly configured auth) where currently a workaround is to do gitlab.com/namespace/subgroup/go-module.git. Currently I am running into this issue with subgroups when a ssh certificate is being used for auth.

[firelizzard18]

@skunkworker the accepted proposal:

Before the first HTTPS fetch, the go command will invoke each GOAUTH command in the list with no additional arguments and no input. The GOAUTH command may reply with zero or more credentials to cache.

[...]

If the server responds with any 4xx code and the response body does not satisfy the request (for example, by including a valid tag), the go command will invoke the GOAUTH commands again, this time with the URL as an additional command-line argument.

I don't think there's any real changes for GitLab. You still have to set up credentials before you make a request, as GitLab's current implementation will never respond to a go-import request with 4xx. If one of the GOAUTH commands includes credentials for GitLab in the initial invocation then it should work but IIUC that already works if you add credentials to .netrc before you make a request.

[dmitshur]

Adding release-blocker to track updating Go 1.24 release notes.

[skunkworker]

@skunkworker the accepted proposal:

Before the first HTTPS fetch, the go command will invoke each GOAUTH command in the list with no additional arguments and no input. The GOAUTH command may reply with zero or more credentials to cache.
[...]
If the server responds with any 4xx code and the response body does not satisfy the request (for example, by including a valid tag), the go command will invoke the GOAUTH commands again, this time with the URL as an additional command-line argument.

I don't think there's any real changes for GitLab. You still have to set up credentials before you make a request, as GitLab's current implementation will never respond to a go-import request with 4xx. If one of the GOAUTH commands includes credentials for GitLab in the initial invocation then it should work but IIUC that already works if you add credentials to .netrc before you make a request.

@firelizzard18

Interesting, I will have to check and see if this works with certificate based authentication for gitlab, username&password based authentication would not be allowable due to policy.

[firelizzard18]

I will have to check and see if this works with certificate based authentication for gitlab

@skunkworker As I understand it, if you add mygitlabauth to GOAUTH, Go will execute it. Your command can do whatever it needs to do to authenticate, but in the end it needs to return a list of URLs and headers. I am not familiar with certificate based authentication, but if the authentication process results in some header such that curl -H "Your-Header: <value>" https://gitlab.com/api/v4/<private project> would succeed, then GOAUTH should work (and vice versa).

[seankhliao]

There's a separate issue #30119 for cmd/go mTLS, though it may be possible to extend the current auth mechanism for it (perhaps using : prefixed pseudo-headers like http2 does for reserved info).

[jameshartig]

I don't think certificates will work over HTTP. What would need to happen for SSH certs is Go would have to try cloning gitlab.com/namespace/subgroup/go-module.git then if that fails try gitlab.com/namespace/subgroup.git, etc but I'm not sure how Go would know to assume git for the import gitlab.com/namespace/subgroup/go-module. Maybe a separate configuration knob for mapping a URL (prefix) to protocol (git+ssh) (which would skip the ?go-get=1 request) but that seems outside the scope of GOAUTH.

[mknyszek]

The RC is planned for next week, and we need a full draft of the release notes before then. Please prioritize writing the release notes for this. Thanks!

[gopherbot]

Change https://go.dev/cl/634235 mentions this issue: _content/doc: add release notes for GOAUTH

[gopherbot]

Change https://go.dev/cl/635255 mentions this issue: cmd/go: add more tests for GOAUTH's user provided authenticator

[gopherbot]

Change https://go.dev/cl/635340 mentions this issue: _content/doc/go1.24: use backticks for GOAUTH

[samthanawalla]

The work related to the original issue is completed. I will close this issue.
See #26232 (comment)

Let's continue the discussion around mTLS authentication in #30119 and create new issues/proposals for any additional feature requests that would require approval :)

Additionally I have created #70872 to cleanup the old reference implementation.

Thanks