Security policy from different route interferes authentication #2507

sadovnikov · 2024-01-26T14:31:55Z

Description:

Reusing the same Gateway, we created two pairs of HTTPRoute with matching SecurityPolicy, which use identical IDP settings for OIDC. The HTTP Routes are on different hosts. Both hosts are CNAMEs to the same A-record, with the IP address of the Gateway service

hostname	path	http route	security policy
reference-apps-twin.platform-staging.gcp.internal.xxx.org	/cbdp-k8s-sample-eg	cbdp-k8s-sample-web-eg	cbdp-k8s-sample-web-eg
reference-apps.platform-staging.gcp.internal.xxx.org	/cbdp-k8s-sample-pr-195	cbdp-k8s-sample-web-pr-195	cbdp-k8s-sample-web-pr-195

Opening of https://reference-apps.platform-staging.gcp.internal.xxx.org/cbdp-k8s-sample-pr-195 in an incognito tab goes through the login process and the application page gets opened.

However, the opening of https://reference-apps-twin.platform-staging.gcp.internal.xxx.org/cbdp-k8s-sample-eg/ in a new incognito tab gives "too many redirects" after the login page. This link starts working when the cbdp-k8s-sample-web-pr-195 Security Policy, which belongs to a different HTTP Route, is deleted.

Repro steps:

the envoy-proxy-min-err.json is the Envoy configuration with both Security Policies present
the envoy-proxy-min-ok.json is the Envoy configuration without the cbdp-k8s-sample-web-pr-195 Security Policy
YAML of HTTP Routes and Security Policies will be added in the issue comment

envoy-proxy-min-err.json
envoy-proxy-min-ok.json

Environment:

Envoy Gateway is deployed using the v0.0.0-latest Helm chart.

dependencies:
- name: gateway-helm
  version: v0.0.0-latest
  repository: oci://docker.io/envoyproxy
  alias: eg

Container

Containers:
  envoy:
    Container ID:  containerd://b8648147d345bcf50d6f68aab09149ece2737c29d72fb0c6fb18115cb946699d
    Image:         envoyproxy/envoy-dev:latest
    Image ID:      docker.io/envoyproxy/envoy-dev@sha256:9d9493167f9b35e0047ce1463df8b679bb5847f1d2b7bbb9d8fb432b08e97bab

Logs:

attached envoy-log.txt is the envoy proxy log from "too many redirects"

envoy-log.txt

The text was updated successfully, but these errors were encountered:

sadovnikov · 2024-01-26T14:32:55Z

HTTP Routes

apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata:
    labels:
      app.kubernetes.io/instance: reference-apps-spring-boot-eg
    name: cbdp-k8s-sample-web-eg
    namespace: reference-apps
  spec:
    hostnames:
    - reference-apps-twin.platform-staging.gcp.internal.xxx.org
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: shared-host-default
      namespace: envoy-gateway-system
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: cbdp-k8s-sample-web-eg
        port: 80
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /cbdp-k8s-sample-eg
      timeouts:
        request: 10s
- apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata:
    annotations:
      meta.helm.sh/release-name: cbdp-k8s-sample-pr-195-preemptible
      meta.helm.sh/release-namespace: reference-apps
    labels:
      app.kubernetes.io/managed-by: Helm
    name: cbdp-k8s-sample-web-pr-195
    namespace: reference-apps
  spec:
    hostnames:
    - reference-apps.platform-staging.gcp.internal.xxx.org
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: shared-host-default
      namespace: envoy-gateway-system
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: cbdp-k8s-sample-web-pr-195
        port: 80
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /cbdp-k8s-sample-pr-195
      timeouts:
        request: 10s
kind: List

Security Policies

apiVersion: v1
items:
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    labels:
      app.kubernetes.io/instance: reference-apps-spring-boot-eg
    name: cbdp-k8s-sample-web-eg
    namespace: reference-apps
  spec:
    oidc:
      clientID: d8c0c5a5-5b45-4566-9fb2-367e62c397ee
      clientSecret:
        group: ""
        kind: Secret
        name: oauth2-secret-eg
      provider:
        issuer: https://login.microsoftonline.com/a87b6d3d-d85e-4d9b-8704-6aed76a49444/v2.0
      scopes:
      - api://d8c0c5a5-5b45-4566-9fb2-367e62c397ee/oidc
      - offline_access
    targetRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: cbdp-k8s-sample-web-eg
      namespace: reference-apps
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    annotations:
      meta.helm.sh/release-name: cbdp-k8s-sample-pr-195-preemptible
      meta.helm.sh/release-namespace: reference-apps
    labels:
      app.kubernetes.io/managed-by: Helm
    name: cbdp-k8s-sample-web-pr-195
    namespace: reference-apps
  spec:
    oidc:
      clientID: d8c0c5a5-5b45-4566-9fb2-367e62c397ee
      clientSecret:
        group: ""
        kind: Secret
        name: oauth2-secret-eg
      provider:
        issuer: https://login.microsoftonline.com/a87b6d3d-d85e-4d9b-8704-6aed76a49444/v2.0
      scopes:
      - api://d8c0c5a5-5b45-4566-9fb2-367e62c397ee/oidc
      - offline_access
    targetRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: cbdp-k8s-sample-web-pr-195
      namespace: reference-apps
kind: List

akhenakh · 2024-02-07T16:16:55Z

I can confirm it happened on our installations as well, while applying an ODIC security policy to an isolated httproute (hostname + path prefix)

  hostnames:
  - xxxxx.aaaa.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: envoy-gateway-system
  rules:
  - backendRefs:
    - kind: Service
      name: auth-generator-api
      port: 8085
    matches:
    - path:
        type: PathPrefix
        value: /

the OIDC redirection triggered on some others HTTPRoute

for example a route supposed to match /v1/users triggered the OIDC redirect with the url /v1/users/ (trailing slash)rather than the usual 404.

When removing the security policy on xxxxx.aaaa.com, the expected 404 on zzzzzz.aaaa.com reappeared.

Maybe it should be another ticket, since the title is matching but not the description, the security policy with the OIDC has been applied to another route that had nothing to do with it, not protected by OIDC at all.

arkodg · 2024-02-08T23:08:21Z

assigning this to @zhaohuabing since he's been debugging this and has a solution, most likely the API will need to be revisited to include a user defined hmac secret (similar to #2567)

zhaohuabing · 2024-02-09T07:36:30Z

Reusing the same Gateway, we created two pairs of HTTPRoute with matching SecurityPolicy, which use identical IDP settings for OIDC. The HTTP Routes are on different hosts.

@sadovnikov I believe you can't use identical OIDC settings for two different HTTPRoutes because the callback url on the IDP side can only point to one HTTPRoute, and you want to make sure the callback url is caught by the targeting HTTPRoute.

Instead, you should have something like the below:

apiVersion: v1
items:
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    labels:
      app.kubernetes.io/instance: reference-apps-spring-boot-eg
    name: cbdp-k8s-sample-web-eg
    namespace: reference-apps
  spec:
    oidc:
      clientID: ${client-id-1}
      clientSecret:
        group: ""
        kind: Secret
        name: oauth2-secret-eg-1
      provider:
        issuer: https://login.microsoftonline.com/a87b6d3d-d85e-4d9b-8704-6aed76a49444/v2.0
      scopes:
      - api://d8c0c5a5-5b45-4566-9fb2-367e62c397ee/oidc
      - offline_access
      redirectURL: https://reference-apps-twin.platform-staging.gcp.internal.xxx.org/cbdp-k8s-sample-eg/oauth2/callback
    targetRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: cbdp-k8s-sample-web-eg
      namespace: reference-apps
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    annotations:
      meta.helm.sh/release-name: cbdp-k8s-sample-pr-195-preemptible
      meta.helm.sh/release-namespace: reference-apps
    labels:
      app.kubernetes.io/managed-by: Helm
    name: cbdp-k8s-sample-web-pr-195
    namespace: reference-apps
  spec:
    oidc:
      clientID: ${client-id-2}
      clientSecret:
        group: ""
        kind: Secret
        name: oauth2-secret-eg-2
      provider:
        issuer: https://login.microsoftonline.com/a87b6d3d-d85e-4d9b-8704-6aed76a49444/v2.0
      scopes:
      - api://d8c0c5a5-5b45-4566-9fb2-367e62c397ee/oidc
      - offline_access
      redirectURL: https://reference-apps.platform-staging.gcp.internal.xxx.org/cbdp-k8s-sample-pr-195/oauth2/callback
    targetRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: cbdp-k8s-sample-web-pr-195
      namespace: reference-apps

zhaohuabing · 2024-02-10T00:52:04Z

I can confirm it happened on our installations as well, while applying an ODIC security policy to an isolated httproute (hostname + path prefix)
  hostnames:
  - xxxxx.aaaa.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: envoy-gateway-system
  rules:
  - backendRefs:
    - kind: Service
      name: auth-generator-api
      port: 8085
    matches:
    - path:
        type: PathPrefix
        value: /
the OIDC redirection triggered on some others HTTPRoute

for example a route supposed to match /v1/users triggered the OIDC redirect with the url /v1/users/ (trailing slash)rather than the usual 404.

When removing the security policy on xxxxx.aaaa.com, the expected 404 on zzzzzz.aaaa.com reappeared.

Maybe it should be another ticket, since the title is matching but not the description, the security policy with the OIDC has been applied to another route that had nothing to do with it, not protected by OIDC at all.

I reproduced this in my test env. Somehow a request that misses all the routes still hits the filters in the HCM filter chain even though these filters are disabled in the routeConfig. It seems a bug in the Envoy upstream. A workaround is adding a catching-all route at the bottom of the routes and returning a 404 direct response.

https://github.com/zhaohuabing/playground/blob/8e73a6573922aeac91550574da91e62d0708c61c/envoy/per-route-oauth2-oidc/envoy.yaml#L109-L142

zhaohuabing · 2024-02-15T03:51:38Z

This has been fixed in EG with #2586. Reopen it to track the upstream issue (which I haven't created yet).

arkodg · 2024-02-27T22:22:21Z

fixed with #2663

sadovnikov added the triage label Jan 26, 2024

arkodg assigned zhaohuabing Feb 8, 2024

arkodg added kind/bug Something isn't working road-to-ga and removed triage labels Feb 8, 2024

github-project-automation bot added this to Envoy Gateway: The Road to GA Feb 8, 2024

github-project-automation bot moved this to Todo in Envoy Gateway: The Road to GA Feb 8, 2024

arkodg added this to the v1.0.0-rc1 milestone Feb 8, 2024

zhaohuabing mentioned this issue Feb 12, 2024

add a catch-all route if needed #2586

Merged

zhaohuabing closed this as completed in #2586 Feb 15, 2024

github-project-automation bot moved this from Todo to Done in Envoy Gateway: The Road to GA Feb 15, 2024

zhaohuabing reopened this Feb 15, 2024

arkodg closed this as completed Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Security policy from different route interferes authentication #2507

Security policy from different route interferes authentication #2507

sadovnikov commented Jan 26, 2024

sadovnikov commented Jan 26, 2024

akhenakh commented Feb 7, 2024 •

edited

Loading

arkodg commented Feb 8, 2024

zhaohuabing commented Feb 9, 2024 •

edited

Loading

zhaohuabing commented Feb 10, 2024 •

edited

Loading

zhaohuabing commented Feb 15, 2024

arkodg commented Feb 27, 2024

Security policy from different route interferes authentication #2507

Security policy from different route interferes authentication #2507

Comments

sadovnikov commented Jan 26, 2024

sadovnikov commented Jan 26, 2024

akhenakh commented Feb 7, 2024 • edited Loading

arkodg commented Feb 8, 2024

zhaohuabing commented Feb 9, 2024 • edited Loading

zhaohuabing commented Feb 10, 2024 • edited Loading

zhaohuabing commented Feb 15, 2024

arkodg commented Feb 27, 2024

akhenakh commented Feb 7, 2024 •

edited

Loading

zhaohuabing commented Feb 9, 2024 •

edited

Loading

zhaohuabing commented Feb 10, 2024 •

edited

Loading