Support Routing based on JWT Claims

[arkodg]

Description:

Describe the desired behavior, what scenario it enables and how it
would be used.

Users would like to route to a specific backend by matching on JWT claims

Looks like a popular issue raised in upstream as well kubernetes-sigs/gateway-api#920 but was closed due to lack of ownership

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[arkodg]

• Some limitations we have from an Upstream API perspective - there is no first class field for claims within the matches field within HTTPRoute, this would be made it very easy for the user to author this intent
• Envoy Gateway has a feature claimToHeaders https://gateway.envoyproxy.io/v0.6.0/api/extension_types/#claimtoheader
  to convert JWT claims to HTTP Headers which can be used for header matching, but from an implementation perspective JWT filter is applied at the route level, post route match. To support above use case, we could
  • automagically move the jwt filter to the hcm level if claimToHeaders is set
  • add a new field under jwt called useForRouting which also moves the jwt filter to the listener level

[arkodg]

there's a clear_route_cache option in the JWT filter config https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/jwt_authn/v3/config.proto#extensions-filters-http-jwt-authn-v3-jwtprovider which clears the route decision and recomputes it, but based on the info

Clears route cache in order to allow JWT token to correctly affect routing decisions. Filter clears all cached routes when:

The field is set to true.
At least one claim_to_headers header is added to the request OR if payload_in_metadata is set.

it doesnt look like it needs to be set

[tmsnan]

@arkodg, Its comments may be ambiguous, requiring clear_route_cache to be set.
https://github.com/envoyproxy/envoy/blob/1f1c32083150dced241336a6e2734e05bcead4b8/source/extensions/filters/http/jwt_authn/authenticator.cc#L380C1-L382C4

[arkodg]

nice find @tmsnan, does that logic need to be updated to

provider.clear_route_cache() || .......

?

[tmsnan]

No changes needed @arkodg. Recalculating routes may cause performance issues. In some scenarios, only header or metadata information needs to be added without requiring rerouting, such as ratelimit-based JWT claims.
Envoy comment needs to be updated.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[zetaab]

I added this to my jwt path and trying to figure out why it is not working:

% egctl config envoy-proxy listener -l gateway.envoyproxy.io/owning-gateway-name=internal > listener
error: proto: (line 2899:18): unknown field "clear_route_cache"

worth of adding new issue or?

[zirain]

I added this to my jwt path and trying to figure out why it is not working:

% egctl config envoy-proxy listener -l gateway.envoyproxy.io/owning-gateway-name=internal > listener
error: proto: (line 2899:18): unknown field "clear_route_cache"

worth of adding new issue or?

are you using the latest of egctl?

[zetaab]

sorry, forgot to compile newest egctl. With newest egctl I can dump the configuration. However, the JWT based routing does not work with the PR which added clear_route_cache to JWT.

configuration:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echoserver-int
  namespace: echoserver
spec:
  parentRefs:
  - name: internal
    namespace: envoy-gateway-system
    sectionName: https
  hostnames:
  - foobar.com
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: echoserver
      port: 80
      weight: 1
    matches:    
    - path:
        type: PathPrefix
        value: /
      headers:
      - name: x-sub
        type: Exact
        value: ca28333f-177f-4b7a-90c3-83951bc0eda4
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: echoserver-int-oidc
  namespace: echoserver
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: echoserver-int
  oidc:
    provider:
      issuer: https://cognito-idp.eu-central-1.amazonaws.com/xxx
    clientID: yy
    scopes:
    - openid
    - email
    - profile
    clientSecret:
      group: ""
      kind: Secret
      name: my-cognito-client-secret
  jwt:
    providers:
    - name: cognito
      recomputeRoute: true
      claimToHeaders:
      - claim: sub
        header: x-sub
      - claim: cognito:groups
        header: x-groups
      - claim: email
        header: x-email
      - claim: name
        header: x-name
      remoteJWKS:
        uri: https://cognito-idp.eu-central-1.amazonaws.com/xxx/.well-known/jwks.json

my application behind httproute is echoserver, if I remove that headers filter. I can see the x-sub and that same value in the response. Also if I remove securitypolicy and just use -H 'x-sub: ca28333f-177f-4b7a-90c3-83951bc0eda4' as curl parameters, the routing will work. So as I see it: new recomputeRoute parameter does not work like it was assumed. Is this tested and verified that it should work?

[arkodg]

@zetaab plan on wrapping up a few higher priority work items for v1 rc before adding e2e and docs for this feature, should hopefully complete it by next week.
for your case you're missing a catchall route (e.g. a route rule for a / prefix match with a direct response of 404), that the route can match on first, convert the claim to header, and then recompute the route and match on your route rule with the x-sub match. We were initially planning on adding the catchall rule by default within EG, but based on community feedback, we're going to document and ask users to do it