Add support for Mount API

[tomchristie]

Closes #977

Allows users to do this sort of thing:

# Redirect all HTTP requests to HTTPS.
mounts = {"http://": HTTPSRedirectTransport()}
client = httpx.Client(mounts=mounts)

Or...

# Mock out the example.org domain.
mounts = {"all://example.org": MockTransport()}
client = httpx.Client(mounts=mounts)

Or...

# Support URLs like "file:///Users/tomchristie/GitHub/encode/httpx/README.md"
mounts = {"file://": FileSystemTransport()}
client = httpx.Client(mounts=mounts)

Once we've got a nice httpx.HTTPTransport() it'll also be a good route for use-cases that require different configuration across different hosts.

Still to do:

• Documentation.
• Test case that actually makes a request to a mount point.
• Potentially add support for eg. {"https://www.example.com/prefix": transport} (?)
• Make sure we support httpx.Client(mounts={"custom://"}), since I've a suspicion there's possibly something in the cookie handling or elsewhere that'll currently fail on non-http/https schemes.

Also what do we want to do about the gnarly case where a user does the following?...

# Same instance mounted in two places.
mounts = {"http://": transport, "https://": transport}
client = httpx.Client(mounts=mounts)

Note that if the same transport is mounted twice then it'll end up have __enter__/__exit__/close called multiple times.

Perhaps that's not actually an issue? Possible that we could just defer thinking about that unless/until we've got an actual live case to consider where it's problematic? After-all, we do already have a similar issue if a user does the following...

transport = ...
client_1 = httpx.Client(transport=transport)
client_2 = httpx.Client(transport=transport)

[cdeler]

# Same instance mounted in two places.
mounts = {"http://": transport, "https://": transport}
client = httpx.Client(mounts=mounts)

It might be a silly idea, but can we make our transports reentrant? I dont know, may be to add some reference counters? From another side, it doesn't look pythonic.

[cdeler]

Do I understand correctly that the mounts overrides proxies? Like:

client = Client(..., proxies={"http://": SomeTransport()}, ..., mounts={"http://": SomeOtherTransport())
assert isinstance(client._mounts["http://"], SomeOtherTransport)

Do we need to outline it somewhere?

[cdeler]

So only the thing, why we introduce self._mounts is a lookup over the dict items.

Then why it's a dictionary? May we have a list with tuples there?

Update: the only benefit we get from self._mounts presented by dict is the keys uniques.

[tomchristie]

Okay, docs are up to scratch on this now, and I think it's good to go...

• Supporting path prefixes, for mounts={"https://www.example.com/prefix": transport} and proxies={"https://www.example.com/prefix": proxy} is something that we could potentially support, but there's no need for us to treat that as in-scope here.
• I'm okay with transports having to gracefully deal with 1 or more close operations being called at this point in time. It's not actually a problem anywhere, and again it's something that we could iterate on later if we chose to.

[florimondmanca]

Looking "dope"! Very exciting.

Seconding @cdeler's note about whether there's any confusion to be anticipated re: proxies. What do we expect this to do…?

Client(proxies={"http://": "..."}, mounts={"http://": "..."})

[lundberg]

Sweet @tomchristie!

Just one thing about mocking...it's quite common that you don't have access to the actual client in the test cases, i.e. testing some remote API behind your integration API. Meaning, it's not really possible to mount a mock transport during a test case.

One way I see that doable is by having specific client mounts picked up from somewhere else in your app/project, e.g. django settings or alike, so that they could differ between production and tests. Works but maybe not so straightforward IMO.

From a RESPX perspective, HTTPX is currently patched globally to mock requests/responses, using a decorator or context manager in your tests. It would be really nice to allow this without patching and to use the mount-feature instead.

Just an idea...what about adding support for global/default mounts? Maybe httpx.configure(mounts=...), picked up as default by the Client, much like the request defaults the client are merging when building a request.

[lundberg]

One more thought ;-)

If mounts were able to be configured globally, i.e. httpx config/settings, then the get_environment_proxies could be moved from the client to the config, used as default mounts by the client, but overridable by the client like now.

[tomchristie]

@lundberg That's a really interesting idea yup! We would need to be super careful with being able to override-at-a-distance, so there might be something to think about there.

[tomchristie]

@florimondmanca @cdeler I'd been a bit slow on responding to the "mounts that precedence over proxies" in the case of two identical keys being used, because I wasn't sure if we ought to document that as expected behaviour, or if it ought to be treated as ambiguous. (Ie. "hey user, probably don't do that, because it's non-obvious what you're expecting")

I'd suggest we get this in as is, but happy to consider any docs follow ups.