CONNECT support for HTTP/1.1 upstreams

[nhinds]

Description:

I attempted to configure the second scenario from #1451, where Envoy accepts a raw TCP connection and uses HTTP CONNECT to establish a tunnel through an upstream proxy. This currently works fine when the upstream proxy is also Envoy (e.g. connecting envoy --config-path configs/encapsulate_in_connect.v3.yaml --base-id 1 to envoy --config-path configs/terminate_connect.v3.yaml using the sample YAML files), as the Envoy proxies talk HTTP/2 to each other, but it does not work when the upstream proxy only supports HTTP/1.1 as mentioned on #1451:

Client --[plain HTTP]--> Envoy --[HTTP CONNECT]--> Upstream Proxy --[HTTPS]--> Server
We use some upstream non-envoy servers that only support HTTP/1 CONNECT, so HTTP/2 CONNECT wouldn't work for our use case.

It appears that removing http2_protocol_options from the cluster configuration is not sufficient to disable HTTP/2 when tunneling_config has been specified in the listener's TCP proxy filter. This matches the current .proto docs for TunnelingConfig:

Currently, only HTTP/2 is supported

Raising this as a separate feature request as it seems like the HTTP/2 support is working fine.

[mattklein123]

Yeah let's track this as an independent issue. I think either @alyssawilk or @lambdai are planning on implementing this but I'm not sure of the current thinking.

[alyssawilk]

@lambdai do folks on your end need this? AFIK @chradcliffe and co were interested too.

[froismo]

I'm interested in this feature.

[irozzo-1A]

I opened this issue #11569, which is related to this feature request. I would gladly propose my help to implement it.
Disclaimer, I have not lot of experience with C++ and Envoy code base so it may take some time, but I've plenty of motivation ;-). WDYT @alyssawilk @mattklein123 ?

[alyssawilk]

Feel free to give it a shot! It shouldn't be too bad with the HTTP/2 code and tests to model off of :-)

[irozzo-1A]

Ok thx I will give it a try. I will let you know if I encounter any issue.

[ggmm-0]

@irozzo-1A, is that in progress?

where Envoy accepts a raw TCP connection and uses HTTP CONNECT to establish a tunnel through an upstream proxy

I have a scenario where I'v got a plain HTTP request and I need to apply some path matching (HTTP Connection Manager), then route it to a cluster through upstream HTTPS (not Envoy) proxy.

It is basically a scenario similar to the one described by @chradcliffe:

Client --[plain HTTP]--> Envoy --[HTTP CONNECT]--> Upstream Proxy --[HTTPS]--> Server

It seems that tunneling_config is only for TCP proxy filter, not HTTP Connection Manager.

@alyssawilk
Am I right that the scenario above is currently not supported?

[irozzo-1A]

@irozzo-1A, is that in progress?

@grzegorz-mirek unfortunately I had to reshuffle my priorities and I did not manage to start working on this yet. I'm still interested on doing that but I cannot give an ETA right now. If you have urgency feel free to take it over.

[alyssawilk]

the config is connect_config on HCM, and tunneling config for TCP proxy.

but yeah, while you can "decap" HTTP/1 or HTTP/2 CONNECT requests into TCP, currently Envoy only "encapsulates" (instantiates CONNECT) for HTTP/2 so unless your upstream supports CONNECT-over-H2 (which Envoy does but many other servers do not) I don't think your use case will be supported until this issue is closed off as implemented.

[edits for correctness]

[chradcliffe]

Internally, we had an immediate requirement to inspect CONNECT-tunnelled traffic, but the solution we came up with isn't necessarily the best. We created an HTTP filter that pushes the CONNECT-tunnelled traffic into a Network::FilterManagerImpl that owns an "inner" HCM. We had to provide a Network::Connection implementation that forwards most of the calls to the real connection, which itself (sadly) requires a const_cast of the real connection.

I'm hoping that the work being done towards #11725 will be the better long-term solution -- I think it also means that we can also process TLS traffic carried by the CONNECT tunnel.

[alyssawilk]

I've been talking to @grzegorz-mirek about this and it turns out he needs to handle L7 HTTP, then encapsulate that in CONNECT, where previously we were talking about using the TCP proxy session to encapsulate CONNECT, which is a somewhat different thing.

I think the best way to do this long term would be to support TCP HTTP/1.1 CONNECT, then forward the normal upstream serialized HTTP traffic to a virtual TCP listener using the work @lambdai is doing. We'd have full CONNECT support for raw proxy more and HTTP/1 via L7, and it could do all the Envoy bells and whistles (say take the payload and wrap in TLS before encapsulating in the CONNECT)

Given that work isn't landed though, it may be worth an interim solution using a custom HTTP upstream (like http/http/upstream_request.cc) which serialized synthetic CONNECT headers before forwarding on [headers][body][data] so it would basically do [CONNECT][headers][body][data]. It would only work with raw HTTP/1.1 and we know we'd want to replace it. Just want to lay the options out here so we can weigh pros and cons.

[mattklein123]

I just synced up with @alyssawilk on this offline so I think I understand all the permutations now.

It's pretty ugly, but as a stopgap would looping back to a local TCP listener work for your use case? You would essentially use connect_config to loop back to local Envoy. On that listener, you can do whatever you want, and then initiate a CONNECT upstream. We would still need to implement HTTP/1.1 CONNECT for tunneling_config, but that seems like it wouldn't be that bad.

Later, the ugly loop back can be replaced with the internal listener work that @lambdai is doing as a perf optimization?

[lambdai]

Also see #13001 for a concrete example of looping back to local listener configs. Thanks @wlhee

[ggmm-0]

Thanks for all the replies. @mattklein123, do you mean something like that?

client --> HCM --> TcpProxy --CONNECT over HTTP/1.1--> upstream proxy

where HCM and TcpProxy are on the same Envoy instance.

@lambdai, that's a very good example. Thank you.

[alyssawilk]

Yes, that's exactly what we'd suggest for now. It gets you L7 routing, HTTP/1.1 connect with full Envoy filter chain (in case you want to say encrypt the inner payload or outer CONNECT) and hopefully soon you can bypass the loopback listener and save some CPU with @lambdai 's virtual listener route.

[irozzo-1A]

@alyssawilk Would using UDS instead of a loopback listener to chain the two stages make sense as a small optimisation? I tried a similar approach here

[alyssawilk]

Yeah, using a pipe is fine too. Either way it's basically using the kernel to do 2 passes through Envoy filter chains (one L4, one L7) and should work great. Avoiding the kernel step just helps on performance.

[irozzo-1A]

Probably I will be able to start working on this during this we.

Yesterday I tried to simply remove those lines:

envoy/source/common/tcp_proxy/tcp_proxy.cc

Lines 459 to 463 in c4a1d12

	if ((cluster->info()->features() & Upstream::ClusterInfo::Features::HTTP2) == 0) {
	ENVOY_LOG(error, "Attempted to tunnel over HTTP/1.1, this is not supported. Set "
	"http2_protocol_options on the cluster.");
	return false;
	}

and I was happily surprised to see that the HTTP/1.1 tunneling seemed to work out of the box. I did a manual test terminating the CONNECT with squid proxy and everything went fine.

@alyssawilk Please let me know if I'm missing something important here and if there are some cases where I should put particular care while writing the tests, but my current understanding (maybe I'm naive :-)) is that my PR should consist of:

• Removing the check preventing from tunneling over HTTP1.1 (see the aforementioned lines)
• Fixing/adding some tests here: https://github.com/envoyproxy/envoy/blob/267cb56/test/integration/tcp_tunneling_integration_test.cc
• Updating the documentation

[alyssawilk]

Sweet, I was hoping it would be pretty simple!

This is one of the cases where we just want to be really careful about the details.
Offhand I expected tunneling to mostly work but we'll definitely want to

1. change HttpUpstream::setRequestEncoder to not send host/scheme/path for HTTP/1.1, but send
   CONNECT [hostname]
   which I think can be done setting {Http::Headers::get().Path, hostname} if the configured codec is HTTP/1.1
2. change HttpUpstream::isValidBytestreamResponse to make sure we only accept well formed HTTP connect responses. I assume this means no content length and no chunked encoding (which might have to be validated in the codec?) but you'll want to check the RFCs and not rely on my memory ;-)
3. test the heck out of it, making sure we've got integration tests for T-E chunked, whacky content lengths, and tests that the raw HTTP/1.1 headers are what we expect.

And then yeah, update docs :-)

[ggmm-0]

@irozzo-1A, I also played with that change and manually tested some scenarios with different proxies that support CONNECT over HTTP/1.1 only. The only issue I found so far is in this scenario (though I'm not fully sure it's related to HTTP/1.1):

client -HTTP-> envoy -CONNECT-over-HTTP/1.1-then-HTTPS-> upstream proxy

Specifying :443 in tunneling_config doesn't help - the upstream proxy tunnels plain HTTP (http://hostname:443) instead of https://hostname. As a workaround, I used loop back to local Envoy suggested above:

client -HTTP-> envoy -HTTPS-> envoy -CONNECT-over-HTTP/1.1-then-HTTPS-> upstream proxy

[irozzo-1A]

@grzegorz-mirek Thx for the info!

I also played with that change and manually tested some scenarios with different proxies that support CONNECT over HTTP/1.1 only. The only issue I found so far is in this scenario (though I'm not fully sure it's related to HTTP/1.1):

client -HTTP-> envoy -CONNECT-over-HTTP/1.1-then-HTTPS-> upstream proxy

From my current understanding I think this cannot be achieved without the two stages, but my opinion is not authoritative ;-)

[alyssawilk]

It depends - if you want to do L7 routing and encapsulate in L4 connect you do the double pass through Envoy. If you just want to take incoming traffic (HTTP or otherwise) and wrap it in CONNECT you only need the one pass.

[irozzo-1A]

It depends - if you want to do L7 routing and encapsulate in L4 connect you do the double pass through Envoy. If you just want to take incoming traffic (HTTP or otherwise) and wrap it in CONNECT you only need the one pass.

Ok, my understanding was good I think.

@grzegorz-mirek, in the scenario you depicted you don't just want to tunnel your incoming plain HTTP traffic with HTTP/1.1 CONNECT, but also using TLS with the end destination (not the upstream proxy). Did I get it right?

[ggmm-0]

@irozzo-1A, yes - that's true. Even without L7 routing (for now).

[govale]

Are there any optimistic plans to support this in the near future? 🙂

[alyssawilk]

oops - leaving this open until the last TODOs are done and config is not hidden.

[irozzo-1A]

thx @alyssawilk, I will try to address the open points ASAP

[cortex93]

@irozzo-1A, I also played with that change and manually tested some scenarios with different proxies that support CONNECT over HTTP/1.1 only. The only issue I found so far is in this scenario (though I'm not fully sure it's related to HTTP/1.1):

client -HTTP-> envoy -CONNECT-over-HTTP/1.1-then-HTTPS-> upstream proxy

Specifying :443 in tunneling_config doesn't help - the upstream proxy tunnels plain HTTP (http://hostname:443) instead of https://hostname. As a workaround, I used loop back to local Envoy suggested above:

client -HTTP-> envoy -HTTPS-> envoy -CONNECT-over-HTTP/1.1-then-HTTPS-> upstream proxy

Would you mind sharing your configuration ?

[alyssawilk]

Can you share config for

"client -HTTP-> envoy -CONNECT-over-HTTP/1.1-then-HTTPS-> upstream proxy"

that wasn't doing TLS? You'd need to configure TLS on the uptream cluster*, I wouldn't expect 443 on tunneling config to be helpful.

*https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl

[ggmm-0]

@alyssawilk, sorry - I don't have this config anymore. I don't remember exactly, but perhaps the issue was that specifying TLS on the upstream cluster caused CONNECT to the proxy to be encrypted (which my proxy does not support). My goal was to have plaintext HTTP/1.1 CONNECT to the proxy and then TLS in the tunnel to the end server. I achieved that with the loop back to local Envoy as described above.

@cortex93, sorry for the delay, do you want the loop back config?

[alyssawilk]

Ahh, I think (offhand) Envoy today will forward encrypted tcp over a plaintext CONNECT, or encrypt both the payload and the CONNECT request, but won't take in plaintext tcp data, encrypt, send CONNECT in the clear and ciphertext payload in a single pass.

You could accomplish that with the loopback hack, doing tcp proxy adding TLS to a loopback address, taking that now encrypted payload and sending it out as CONNECT. If there's areas we could update the docs to make this more clear, let me know!

[cortex93]

@grzegorz-mirek

Yes, that would be kind

[ggmm-0]

@cortex93, this should work: https://gist.github.com/grzegorz-mirek/c028a24ec2b32dfa47b1fdb904ed3baa

[cortex93]

@cortex93, this should work: https://gist.github.com/grzegorz-mirek/c028a24ec2b32dfa47b1fdb904ed3baa

Made a fork with UDS and upstream proxy authentication support :
https://gist.github.com/cortex93/6d7cd9738b077d0447ab891cbacd0d5e

[askquestions]

I want to ask a related question on dynamic configuration of forwarding raw TCP traffic.

Does anyone have experience in using lds.yaml and cds.yaml to make raw TCP packet to be matched to the filter:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: tcp
cluster: tcp_proxy_cluster

Can anyone tell me any error in my lds.yaml file shown below? Why envoy proxy just won't identify raw TCP packets by using this lds.yaml in filesystem-based dynamic configuration?

resources:

• "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_0
  address:
  socket_address:
  address: 0.0.0.0
  port_value: 10000
  filter_chains:
  • filters:
    • name: envoy.http_connection_manager
      typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
      stat_prefix: ingress_http
      http_filters:
      • name: envoy.router
        route_config:
        name: local_route
        virtual_hosts:
        • name: local_service
          domains:
          • "*"
            routes:
          • match:
            prefix: "/"
            route:
            cluster: http_proxy_cluster
    • name: envoy.filters.network.tcp_proxy
      typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
      stat_prefix: tcp
      cluster: tcp_proxy_cluster