Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tls minversion not being respected in ClientTrafficPolicy #3060

Closed
jhouston1604 opened this issue Mar 29, 2024 · 16 comments
Closed

tls minversion not being respected in ClientTrafficPolicy #3060

jhouston1604 opened this issue Mar 29, 2024 · 16 comments
Labels

Comments

@jhouston1604
Copy link

Description:

Hi! thanks for all your work on this, really enjoying it. I'm trying to force the minVersion of TLS to 1.3 on our https listener for compliance reasons however it doesn't seem to honor the setting and is still showing TLSv1.0-TlS1.3. Or as always it might be user error. Thanks for taking the time to read this!

Repro steps:

  1. Envoy gateway installed in envoy-public namespace (routing works, security policies work etc)
  2. Configure ClientTrafficPolicy to this file:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: force-tls-min
  namespace: envoy-public
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: envoy-public
    namespace: envoy-public
  tls:
    minVersion: "1.3"
  1. Verify envoy gateway picked it up
    Screenshot 2024-03-29 at 12 15 44 PM
  2. Run scan via nmap
$  sudo nmap --script ssl-enum-ciphers -p 443 $REDACTED
Starting Nmap 7.94 ( https://nmap.org ) at 2024-03-29 12:09 EDT
Nmap scan report for $REDACTED
Host is up (0.0067s latency).
Other addresses for $REDACTED

PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers:
|   TLSv1.0:
|     ciphers:
|       TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (ecdh_x25519) - A
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.1:
|     ciphers:
|       TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (ecdh_x25519) - A
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256-draft (ecdh_x25519) - A
|     compressors:
|       NULL
|     cipher preference: client
|   TLSv1.3:
|     ciphers:
|       TLS_AKE_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_AKE_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_AKE_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|     cipher preference: client
|_  least strength: A

Environment:
Tested in 1.0 release

@liorokman
Copy link
Contributor

@jhouston1604 could you check the status of the ClientTrafficPolicy resource after you apply it? Was the policy itself accepted or rejected?

If it was accepted, could you share the Envoy Proxy listener configuration? You can find it with the egctl command.

egctl config envoy-proxy listener

@jhouston1604
Copy link
Author

hi @liorokman Screenshot attached of the policy being accepted, however getting this error running this command
image


egctl config envoy-proxy listener envoy-envoy-public-envoy-public-bc6fdf58-7fd697c8dc-72b7h -n envoy-public
error: proto: (line 3452:13): unknown field "forward_not_matching_preflights"

Note: i ran command with those options since it wasnt in the default envoy namespace. Thanks again for looking into this

@liorokman
Copy link
Contributor

liorokman commented Mar 29, 2024

@jhouston1604 I'm not able to reproduce the issue in my test environment.

Could you provide some more information?

  1. Exactly how you installed Envoy Gateway? Any special configuration?
  2. The output of kubectl describe deployment on the Envoy-Gateway deployment?
  3. The output of egctl version ?
  4. The deployment descriptors for the Gateway, the HTTPRoute, the ClientTrafficPolicy?
  5. Could you check if there are any errors in the Envoy Proxy pod's log?

@liorokman
Copy link
Contributor

error: proto: (line 3452:13): unknown field "forward_not_matching_preflights"

This seems to be related to CORS configuration in Envoy Proxy. Do you have any CORS related configuration?

@arkodg
Copy link
Contributor

arkodg commented Apr 1, 2024

error: proto: (line 3452:13): unknown field "forward_not_matching_preflights"

This seems to be related to CORS configuration in Envoy Proxy. Do you have any CORS related configuration?

@jhouston1604 this is tied to a mismatch b/w client and server version, if you install the latest egctl it should work fine

@arkodg
Copy link
Contributor

arkodg commented Apr 1, 2024

@liorokman can you help ? should be easy to repro with the config provided

@liorokman
Copy link
Contributor

@arkodg like I wrote - I could not reproduce the issue with the provided information. I need more details.

@jhouston1604
Copy link
Author

Hi @liorokman here are the items you requested, thanks again!

  1. Exactly how you installed Envoy Gateway? Any special configuration?
    I install via the helm commands below. The special configuration is i have two envoy-gateway deployments. One is just an internet facing lb and one is an internal lb, thats the only difference (besides the naming)
    - helm upgrade -i envoy-public oci://docker.io/envoyproxy/gateway-helm --version v1.0.0 -n envoy-public -f values-public.yaml
    - helm upgrade -i envoy-private oci://docker.io/envoyproxy/gateway-helm --version v1.0.0 -n envoy-private -f values-private.yaml
  2. The output of kubectl describe deployment on the Envoy-Gateway deployment?
$ kubectl describe deployments.apps -n envoy-public envoy-gateway
Name:                   envoy-gateway
Namespace:              envoy-public
CreationTimestamp:      Tue, 06 Feb 2024 12:14:03 -0500
Labels:                 app.kubernetes.io/instance=envoy-public
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=gateway-helm
                        app.kubernetes.io/version=v1.0.0
                        control-plane=envoy-gateway
                        helm.sh/chart=gateway-helm-v1.0.0
Annotations:            deployment.kubernetes.io/revision: 6
                        meta.helm.sh/release-name: envoy-public
                        meta.helm.sh/release-namespace: envoy-public
Selector:               app.kubernetes.io/instance=envoy-public,app.kubernetes.io/name=gateway-helm,control-plane=envoy-gateway
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=envoy-public
                    app.kubernetes.io/instance=envoy-public
                    app.kubernetes.io/name=gateway-helm
                    control-plane=envoy-gateway
  Service Account:  envoy-gateway
  Containers:
   envoy-gateway:
    Image:       docker.io/envoyproxy/gateway:v1.0.0
    Ports:       18000/TCP, 18001/TCP, 19001/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Args:
      server
      --config-path=/config/envoy-gateway.yaml
    Limits:
      cpu:     500m
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ENVOY_GATEWAY_NAMESPACE:     (v1:metadata.namespace)
      KUBERNETES_CLUSTER_DOMAIN:  cluster.local
    Mounts:
      /certs from certs (ro)
      /config from envoy-gateway-config (ro)
  Volumes:
   envoy-gateway-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      envoy-gateway-config
    Optional:  false
   certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  envoy-gateway
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  envoy-gateway-54464c8f57 (0/0 replicas created), envoy-gateway-8f7b45c5c (0/0 replicas created), envoy-gateway-7b487697b7 (0/0 replicas created), envoy-gateway-65d45ccdcd (0/0 replicas created)
NewReplicaSet:   envoy-gateway-7d6dcd75c7 (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  32m   deployment-controller  Scaled up replica set envoy-gateway-7d6dcd75c7 to 1 from 0
  Normal  ScalingReplicaSet  31m   deployment-controller  Scaled down replica set envoy-gateway-65d45ccdcd to 2 from 3
  Normal  ScalingReplicaSet  31m   deployment-controller  Scaled up replica set envoy-gateway-7d6dcd75c7 to 2 from 1
  Normal  ScalingReplicaSet  30m   deployment-controller  Scaled down replica set envoy-gateway-65d45ccdcd to 1 from 2
  Normal  ScalingReplicaSet  30m   deployment-controller  Scaled up replica set envoy-gateway-7d6dcd75c7 to 3 from 2
  Normal  ScalingReplicaSet  30m   deployment-controller  Scaled down replica set envoy-gateway-65d45ccdcd to 0 from 1
  1. The output of egctl version ?
$ egctl version
client: v0.6.0
server:
- Name: envoy-gateway-865f8499b5-2f8wt
  Namespace: envoy-internal
  envoyGatewayVersion: v1.0.0
  envoyProxyVersion: distroless-v1.29.2
  gatewayAPIVersion: v1.0.0
  gitCommitID: 72c0cc757c7c4703dfc14345dbd995240af663d7
- Name: envoy-gateway-865f8499b5-ghblh
  Namespace: envoy-internal
  envoyGatewayVersion: v1.0.0
  envoyProxyVersion: distroless-v1.29.2
  gatewayAPIVersion: v1.0.0
  gitCommitID: 72c0cc757c7c4703dfc14345dbd995240af663d7
- Name: envoy-gateway-865f8499b5-gmhw9
  Namespace: envoy-internal
  envoyGatewayVersion: v1.0.0
  envoyProxyVersion: distroless-v1.29.2
  gatewayAPIVersion: v1.0.0
  gitCommitID: 72c0cc757c7c4703dfc14345dbd995240af663d7
- Name: envoy-gateway-7d6dcd75c7-8cv7x
  Namespace: envoy-public
  envoyGatewayVersion: v1.0.0
  envoyProxyVersion: distroless-v1.29.2
  gatewayAPIVersion: v1.0.0
  gitCommitID: 72c0cc757c7c4703dfc14345dbd995240af663d7
- Name: envoy-gateway-7d6dcd75c7-9w7kx
  Namespace: envoy-public
  envoyGatewayVersion: v1.0.0
  envoyProxyVersion: distroless-v1.29.2
  gatewayAPIVersion: v1.0.0
  gitCommitID: 72c0cc757c7c4703dfc14345dbd995240af663d7
- Name: envoy-gateway-7d6dcd75c7-x7gvf
  Namespace: envoy-public
  envoyGatewayVersion: v1.0.0
  envoyProxyVersion: distroless-v1.29.2
  gatewayAPIVersion: v1.0.0
  gitCommitID: 72c0cc757c7c4703dfc14345dbd995240af663d7
  1. The deployment descriptors for the Gateway, the HTTPRoute, the ClientTrafficPolicy?
 Gateway:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-public
  namespace: envoy-public
spec:
  controllerName: gateway.envoyproxy.io/public-gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: custom-proxy-config
    namespace: envoy-public
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-public
  namespace: envoy-public
spec:
  gatewayClassName: envoy-public
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTP
      port: 443
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-public
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 3
        pod:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - envoy-public
                  topologyKey: "kubernetes.io/hostname"
          labels: 
            app: envoy-public
        
      envoyService:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-name: REDACTED
          service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-subnets: REDACTED
          service.beta.kubernetes.io/aws-load-balancer-type: nlb
HTTPRoute:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  annotations:
    meta.helm.sh/release-name: REDACTED
    meta.helm.sh/release-namespace: REDACTED
  creationTimestamp: 2024-03-20T13:12:42Z
  generation: 1
  labels:
    app.kubernetes.io/instance: REDACTED
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: REDACTED
    app.kubernetes.io/version: 0.0.1
    helm.sh/chart: REDACTED-0.1.0
  name: REDACTED
  namespace: REDACTED
  resourceVersion: "174249636"
  uid: a457993e-390b-4865-92d0-69a39dfa1b8d
spec:
  hostnames:
    - REDACTED
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: envoy-public
      namespace: envoy-public
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: REDACTED
          port: 80
          weight: 1
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              replacePrefixMatch: /api
              type: ReplacePrefixMatch
      matches:
        - path:
            type: PathPrefix
            value: /REDACTED/REDACTED
status:
  parents:
    - conditions:
        - lastTransitionTime: 2024-03-20T13:12:42Z
          message: Route is accepted
          observedGeneration: 1
          reason: Accepted
          status: "True"
          type: Accepted
        - lastTransitionTime: 2024-03-20T13:12:42Z
          message: Resolved all the Object references for the Route
          observedGeneration: 1
          reason: ResolvedRefs
          status: "True"
          type: ResolvedRefs
      controllerName: gateway.envoyproxy.io/public-gatewayclass-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: envoy-public
        namespace: envoy-public

ClientTrafficPolicy:
image

SeucirtyPolicy for the HTTP route posted:

cors:
  allowHeaders:
    - "*"
  allowMethods:
    - GET
    - POST
    - OPTIONS
  allowOrigins:
    - https://*.REDACTED.io
    - https://*.REDACTED.io
    - https://*.REDACTED.net
  exposeHeaders:
    - "*"
jwt:
  providers:
    - claimToHeaders:
        - claim: REDACTED_requester
          header: REDACTED_requester
        - claim: permissions
          header: permissions
        - claim: REDACTED_roles
          header: REDACTED_roles
        - claim: REDACTED_organization
          header: REDACTED_organization
      extractFrom:
        headers:
          - name: Authorization
            valuePrefix: "Bearer "
      name: REDACTED
      remoteJWKS:
        uri: https://REDACTED.us.REDACTED.com/.well-known/jwks.json
targetRef:
  group: gateway.networking.k8s.io
  kind: HTTPRoute
  name: REDACTED-REDACTED
  1. Could you check if there are any errors in the Envoy Proxy pod's log?
    no errors just this warning:
    [2024-04-01 12:10:25.607][1][warning][main] [source/server/server.cc:910] There is no configured limit to the number of allowed active downstream connections. Configure a limit in envoy.resource_monitors.downstream_connections resource monitor.
    Also just to add, routing, jwt configuration, cors configurations all working great in the configs i posted above. Is this a setting that has to be there during install? The clienttrafficpolicy i did apply sometime afterwards

@liorokman
Copy link
Contributor

liorokman commented Apr 1, 2024

@jhouston1604 thanks for all the additional information.

Two things jump out to me:

  1. You're using egctl version 0.6.0 against a 1.0.0 server. That egctl is compiled against an older version of the XDS protobufs, which might explain the error you got from egctl. Could you upgrade to the 1.0.0 version of egctl to verify?

    $ egctl version
    client: v0.6.0

  2. You are running envoy gateway with a replica size of 3. In version 1.0.0, the Kubernetes provider doesn't enable leader election.

    Selector: app.kubernetes.io/instance=envoy-public,app.kubernetes.io/name=gateway-helm,control-plane=envoy-gateway
    Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
    StrategyType: RollingUpdate

    This means two things:

    1. There is contention around the status updates for gateway resources
    2. There's a possible race condition between Envoy Proxy and the XDS server created in each of the replicas.

There's a PR making the rounds right now ( #2694 ) that makes it safe to use more than one replica for Envoy Gateway, but it hasn't merged yet.

Could you see if the issue reproduces for you when you're running Envoy Gateway with a replica size of 1 ?

@jhouston1604
Copy link
Author

@liorokman interesting, I will upgrade my client and and scale down my replica size to one right now, sec

@jhouston1604
Copy link
Author

@liorokman I did as requested, and i also recreated all the pods just incase. no change. I did however see two errors in the gateway logs. not sure if related
image
image

|2024-04-01T13:45:54.288Z	DPANIC	provider	kubernetes/predicates.go:128	odd number of arguments passed as key-value pairs for logging	{"runner": "provider", "ignored key": "gateway.envoyproxy.io/public-gatewayclass-controller"}
│ 2024-04-01T13:45:55.204Z    ERROR    infrastructure    runner/runner.go:70    failed to create new infra    {"runner": "infrastructure", "error": "failed to create or update service envoy-public/envoy-envoy-public-envoy-public-bc6fdf58: for Update: Service \"envoy-envoy-public-envoy-public-bc6fdf58\" is invalid: spec.loadBalancerClass: Invalid │
│  value: \"null\": may not change once set"}                                                                                                                                                                                                                                                                                                               │
│ 2024-04-01T13:45:55.204Z    ERROR    watchable    message/watchutil.go:56    observed an error    {"runner": "infrastructure", "error": "failed to create or update service envoy-public/envoy-envoy-public-envoy-public-bc6fdf58: for Update: Service \"envoy-envoy-public-envoy-public-bc6fdf58\" is invalid: spec.loadBalancerClass: Invalid value: \" │
│ null\": may not change once set"}

@liorokman
Copy link
Contributor

@jhouston1604 I don't think the errors you saw in the log are relevant - they seem to be from the infrastructure controller.

For the first issue, does the egctl config envoy-proxy listener $POD_NAME -n envoy-public still return the same error? If not - could you post the listener configuration that this command returns?

@jhouston1604
Copy link
Author

@liorokman no errors, let me redact some information in here and i'll post shortly!

@jhouston1604
Copy link
Author

@liorokman config attached!
listener-config.txt

@liorokman
Copy link
Contributor

@jhouston1604

Looking at the listener configuration, none of those listeners are configured to use TLS.

Your gateway is defined like this:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-public
  namespace: envoy-public
spec:
  gatewayClassName: envoy-public
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTP
      port: 443
      allowedRoutes:
        namespaces:
          from: All

Simply using port 443 doesn't transform the listener to a TLS enabled listener. You need to add a TLS section at the very least:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-public
  namespace: envoy-public
spec:
  gatewayClassName: envoy-public
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTPS # The protocol needs to be HTTPS and not HTTP
      port: 443
      allowedRoutes:
        namespaces:
          from: All
      tls: # This section is missing in the configuration files you listed above
         certificateRefs:  # The place where the server X.509 certificate can be found
         - group: "" 
            kind: Secret
            name: example-cert
          mode: Terminate  

Since TLS is not configured for any of the listeners, limiting the supported TLS version to 1.3 in a ClientTrafficPolicy doesn't really make any sense here.

@jhouston1604
Copy link
Author

oh my goodness @liorokman your so right. I"m so used to using cert manager completely spaced that our certs are in cloudflare. Thanks for all your time and help with this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants