Support callback-based authentication for HTTP and HTTP proxies

[danchr]

I've managed to get HTTP authentication working in hg-git by intercepting dulwich.client.HTTPUnauthorized, but this has some issues. First of all, it doesn't distinguish between the HTTP call itself and the proxy, and it doesn't reveal the scheme — which is used by Mercurial's password manager.

[jelmer]

It's currently just raised for a 401, not for a 407. We could possibly raise a different exception for HTTPProxyUnauthorized.

I've updated the exception in trunk to include the URL.

[danchr]

Actually, I meant the realm not the scheme, but the URL is nice too, sorry! Rather than adding another specific HTTP exception class, perhaps it would be better to add a general subclass of GitProtocolError for HTTP exceptions?

Also, it seemed a bit odd to me that as a client of this library, I had to parse the WWW-Authenticate header; that feels like something Dulwich should be doing…

Anyway, the current support for authentication is mostly fine, and I managed to make it work in hg-git. Adding support for authenticated proxies seems harder, especially considering that the proxy I'm running into requires either NTLM or SPNego authentication 😕

[jelmer]

I've now added a HTTPProxyUnauthorized exception as well.

I'm not sure if Dulwich should get in the business of parsing WWW-Authenticate in this particular code path. One of the intentions here is to provide flexibility for consumers like hg-git and bzr-git, and the ability to plug in other HTTP libraries or authentication (e.g. on Windows you may want to support Negotiate and GSSAPI). (still trying to navigate this, open to suggestions on what you think works best for hg-git)

The dulwich porcelain should provide an out of the box authentication experience that e.g. uses the git credentials helpers.

[danchr]

Well, what'd be really nice for hg-git is using urllib2 like Mercurial rather than urllib3 — but that's probably too much to ask?

[jelmer]

For dulwich itself, yes - migrating to urllib3 was painful enough and I don't want to make library users go through that again. However, we could provide an abstract base class where you can just bring your own http_request(method, url, data) function?

[raphaeljoie]

Because Poetry relies on Dulwich, I kind of have the same issue.

I need to clone a git+https:// repo behind an HTTP proxy. Unfortunately, Poetry will rely on the default urllib3.PoolManager generated by default_urllib3_manager() (code). And that default generator misses a critical part: it won't populate the credentials in the proxy_header parameter.

This is a critical part because HTTPS over HTTP proxy requires tunneling (see urllib3 implementation connection_requires_http_tunnel() here)

... and when using tunneling, the proxy credentials should be in Proxy-Authorization: Basic XXXX header. Otherwise it is nowhere else.

Suggestion: when proxy_url contains a username+password => populate the proxy_header with Proxy-Authorization: Basic XXXX

[jelmer]

@raphaeljoie please file a separate issue, that's different from this one.

edit: Actually nevermind - I've just opened #1227, since github makes splitting quite easy now.