mTLS over TCP tunnel via HTTP/2 CONNECT

[wlhee]

Hi, thanks to the TCP-tunnel via HTTP/2 CONNECT in Envoy, we can tunnel TCP over the following set up:

client -> (TCP) -> client-side Envoy -> (HTTP/2 CONNECT) -> server-side Envoy -> (TCP) -> server

Our question is how do we configure the Envoys to enable mTLS? It seems like the mTLS is always between a cluster and its immediate upstream?

Please see the example configs below. What we want to achieve to have mTLS between cluster_0 of client-side Envoy and listener2 of server-side Envoy.

Thanks for any tips!

1. client-side Envoy:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 10000
    filter_chains:
    - filters:
      - name: tcp
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: tcp_stats
          cluster: "cluster_0"
          tunneling_config:
            hostname: host.com
  - name: listener_1
    address:
      socket_address:
        address: 127.0.0.1
        port_value: 10001
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
                - "*"
              routes:
                - match:
                    prefix: "/"
                  route:
                    cluster: cluster_1
          http_filters:
          - name: envoy.filters.http.router
          http2_protocol_options:
            allow_connect: true
          upgrade_configs:
            - upgrade_type: CONNECT
  clusters:
    - name: cluster_0
      connect_timeout: 5s
      http2_protocol_options:
        {}
      load_assignment:
        cluster_name: cluster_0
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 10001
    - name: cluster_1
      connect_timeout: 5s
      http2_protocol_options:
        allow_connect: true
      load_assignment:
        cluster_name: cluster_1
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 10002

2. server-side Envoy:

static_resources:
  listeners:
  - name: listener_2
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 10002
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
                - "*"
              routes:
                - match:
                    connect_matcher:
                      {}
                  route:
                    cluster: cluster_2
                    upgrade_configs:
                      - upgrade_type: CONNECT
                        connect_config:
                          {}
          http_filters:
          - name: envoy.filters.http.router
          http2_protocol_options:
            allow_connect: true
          upgrade_configs:
            - upgrade_type: CONNECT

  clusters:
  - name: cluster_2
    connect_timeout: 2s
    load_assignment:
      cluster_name: cluster_2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 7777

[wlhee]

@htuch @alyssawilk

[moderation]

I've stumbled on this as we were looking for the same functionality for the Kubernetes ingress controller Contour which leverages Envoy for it's data plane. Issue at projectcontour/contour#2885.

I think you'll need x-forwarded-client-cert configured in your HCM to pass the certificate material upstream. See https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#config-http-conn-man-headers-x-forwarded-client-cert

[wlhee]

Thanks @moderation for the information! I want to make sure my understanding is correct:

1. enable tls transport socket in the cluster_0
2. configure x-forwawrded-client-cert is configured in listener_1

[moderation]

I might have confused things here. Your examples are based on the links from the documentation at https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/upgrades#tunneling-tcp-over-http-2 but it looks like you have added a HCM on the client side

So thinking about this I'm not sure you can do a L4-7 mTLS chain with L3 TCP??? If what you are talking about is possible I would think you need the xfcc config set at cluster_0 so that the cert material is passed to listener_1 with the forward_client_cert_details, set_current_client_cert_details configurations

Definitely need to defer to @htuch and @alyssawilk on this one.

[wlhee]

Right, my example is based on the doc, but I extended in a way that basically the client-side Envoy converts the downstream TCP stream into a HTTP/2 stream (that's why it has two listeners); the server-side Envoy unwraps the TCP stream from the HTTP/2 stream then passes to the server upstream.

[wlhee]

yeah, I think I need to enable xfcc config in cluster_0 in addition to the tls config, do you know if there is a good example to enable xfcc ?

[htuch]

I think it's probably worth explaining whose identity you are trying to verify/preserve. If it's only between the two Envoys, then I think you just need to set TLS transport socket for cluster_1 and listener_2.

Why do you want to start the mTLS at cluster_0? I think that would only make sense if you want the TCP stream forwarded from cluster_2 to be TLS.

I think #11725 is also relevant, CC @lambdai

[htuch]

Synced with @wlhee offline. The idea is to preserve an inner identity, associated with the original TCP stream, while client/server Envoy might have an outer TLS connection. I think mTLS from cluster_0 to listener_1, with XFCC, would provide you a way to propagate the identity of cluster_0 over the connection. If that's all you need, I think that works. If you need full tunneling of mTLS, I think you need to potentially do another loopback where you layer TLS on the TCP stream before wrapping with CONNECT. I'll defer to @lambdai and @alyssawilk on this one.

[wlhee]

Thanks @htuch

I tried to use xfcc in cluster_0, but it doesn't seem like I am configuring things properly. It looks like cluster_0 can't talk to the loopback listener_1 due to cert enabled in cluster_0:

client-side envoy config:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 10000
    filter_chains:
    - filters:
      - name: tcp
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: tcp_stats
          cluster: "cluster_0"
          tunneling_config:
            hostname: host.com

  - name: listener_1
    address:
      socket_address:
        address: 127.0.0.1
        port_value: 10001
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
                - "*"
              routes:
                - match:
                    prefix: "/"
                  route:
                    cluster: cluster_1
          http_filters:
          - name: envoy.filters.http.router
          http2_protocol_options:
            allow_connect: true
          upgrade_configs:
            - upgrade_type: CONNECT
          forward_client_cert_details: ALWAYS_FORWARD_ONLY
          set_current_client_cert_details:
            cert: true


  clusters:
    - name: cluster_0
      connect_timeout: 5s
      http2_protocol_options:
        {}
      load_assignment:
        cluster_name: cluster_0
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 10001
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          common_tls_context:
            tls_certificates:
              certificate_chain: { "filename": "cert.pem" }
              private_key: { "filename": "key.pem" }

    - name: cluster_1
      connect_timeout: 5s
      http2_protocol_options:
        allow_connect: true
      load_assignment:
        cluster_name: cluster_1
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 10002

server-side envoy config:

static_resources:
  listeners:
  - name: listener_2
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 10002
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
                - "*"
              routes:
                - match:
                    connect_matcher:
                      {}
                  route:
                    cluster: cluster_2
                    upgrade_configs:
                      - upgrade_type: CONNECT
                        connect_config:
                          {}
          http_filters:
          - name: envoy.filters.http.router
          http2_protocol_options:
            allow_connect: true
          upgrade_configs:
            - upgrade_type: CONNECT
          forward_client_cert_details: ALWAYS_FORWARD_ONLY
          set_current_client_cert_details:
            cert: true
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
              certificate_chain: { "filename": "cert.pem" }
              private_key: { "filename": "key.pem" }
          require_client_certificate: true

  clusters:
  - name: cluster_2
    connect_timeout: 2s
    load_assignment:
      cluster_name: cluster_2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 7777

[htuch]

@wlhee I think for listener_1 to be able to process the HTTP/2 it would need to also be able terminate TLS and have transport socket options that correspond.

[wlhee]

Per offline discussion with @htuch to figure out.

client envoy:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 10000
    filter_chains:
    - filters:
      - name: tcp
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: tcp_stats
          cluster: "cluster_0"

  - name: listener_1
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 10001
    filter_chains:
    - filters:
      - name: tcp
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: tcp_stats
          cluster: "cluster_1"
          tunneling_config:
            hostname: host.com

  clusters:
    - name: cluster_0
      connect_timeout: 5s
      http2_protocol_options:
        {}
      load_assignment:
        cluster_name: cluster_0
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 10001
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          common_tls_context:
            tls_certificates:
              certificate_chain: { "filename": "cert.pem" }
              private_key: { "filename": "key.pem" }

    - name: cluster_1
      connect_timeout: 5s
      http2_protocol_options:
        allow_connect: true
      load_assignment:
        cluster_name: cluster_1
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 10002

server envoy:

static_resources:
  listeners:
  - name: listener_2
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 10002
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
                - "*"
              routes:
                - match:
                    connect_matcher:
                      {}
                  route:
                    cluster: cluster_2
                    upgrade_configs:
                      - upgrade_type: CONNECT
                        connect_config:
                          {}
          http_filters:
          - name: envoy.filters.http.router
          http2_protocol_options:
            allow_connect: true
          upgrade_configs:
            - upgrade_type: CONNECT

  - name: listener_3
    address:
      socket_address:
        protocol: TCP
        address: 127.0.0.1
        port_value: 10003
    filter_chains:
    - filters:
      - name: tcp
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: tcp_stats
          cluster: "cluster_3"
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
              certificate_chain: { "filename": "cert.pem" }
              private_key: { "filename": "key.pem" }

  clusters:
  - name: cluster_2
    connect_timeout: 2s
    load_assignment:
      cluster_name: cluster_2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10003

  - name: cluster_3
    connect_timeout: 2s
    load_assignment:
      cluster_name: cluster_3
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 7777