envoyproxy needs restart before securitypolicy takes action #2521

zetaab · 2024-01-28T22:10:59Z

Description:
I am trying to add oidc+jwt authentication to single HTTPRoute. However, after the configuration is in place, it does not take action before the envoyproxy is restarted.

Repro steps:

first create httproute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echoserver-ext
  namespace: echoserver
spec:
  parentRefs:
  - name: external
    namespace: envoy-gateway-system-external
    sectionName: https
  hostnames:
  - eg-ext.example.com
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: echoserver
      port: 80
      weight: 1
    matches:    
    - path:
        type: PathPrefix
        value: /

% curl https://eg-ext.example.com -I
HTTP/2 200

Works like should. Then apply securitypolicy:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: echoserver-ext-oidc
  namespace: echoserver
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: echoserver-ext
  oidc:
    provider:
      issuer: https://xxx
    clientID: yy
    scopes:
    - openid
    - email
    - profile
    - groups
    clientSecret:
      group: ""
      kind: Secret
      name: my-app-client-secret
  jwt:
    providers:
    - name: kaas
      claimToHeaders:
      - claim: email
        header: x-email
      - claim: groups
        header: x-groups
      - claim: name
        header: x-name
      - claim: sub
        header: x-sub
      remoteJWKS:
        uri: https://xxx/.well-known/jwks.json

% kubectl apply -f external/app.yaml -n echoserver
httproute.gateway.networking.k8s.io/echoserver-ext configured
securitypolicy.gateway.envoyproxy.io/echoserver-ext-oidc created
 % kubectl get securitypolicy -n echoserver
NAME                  STATUS     AGE
echoserver-ext-oidc   Accepted   7s

Still target working without authentication?

% curl https://eg-ext.example.com -I
HTTP/2 200

Restarting envoy and trying again:

% kubectl delete pod -l gateway.envoyproxy.io/owning-gateway-name=external  -n envoy-gateway-system-external 
pod "envoy-envoy-gateway-system-external-external-ad2cc5d0-dd7b5ghtl" deleted
% curl https://eg-ext.example.com -I
HTTP/2 302

Works like should. Why the restart is needed?

Environment:
kube 1.29.1

Logs:

The text was updated successfully, but these errors were encountered:

tmsnan · 2024-01-29T02:28:40Z

@zetaab, can you post the configuration of envoy's listener and route when it does not take effect?
egctl config envoy-proxy listener
egctl config envoy-proxy route

zetaab · 2024-01-29T05:26:35Z

@tmsnan took few attempts to reproduce

routes https://gist.github.com/zetaab/63a22a8d4db2a2a9c0edaf995d4abee8
listeners https://gist.github.com/zetaab/51279b863eb90ed17b29a5075f56602f

in listeners the interesting information is perhaps errorstate

          "errorState": {
            "details": "OAuth2 filter: unknown cluster 'oidc_example_com_443' in config. Please specify which cluster to direct OAuth requests to.",

egctl config envoy-proxy cluster -n envoy-gateway-system-external > cluster.json also provided https://gist.github.com/zetaab/fbbf54b992ee48e59cae36f3965f671c

looks like the cluster nowadays exists, but I am thinking is the config modification order wrong or something? The cluster needs to be applied first and after that oauth2 config itself. If the order is wrong (race-condition?) it can end to error state? For me this is happening something like 10-20% of cases.

I am running this process in loop (manually):

create httproute
verify that new route works (curl <address> -I)
apply securitypolicies
check does route still answer with 200 or 302 (curl <address> -I)
if 4) is answering 302 (like should) remove securitypolicy and httproute and go back to 1)

this is visible also in prometheus metrics

# TYPE envoy_listener_downstream_listener_filter_error counter
envoy_listener_downstream_listener_filter_error{envoy_listener_address="0.0.0.0_10080"} 0
envoy_listener_downstream_listener_filter_error{envoy_listener_address="0.0.0.0_10443"} 1
envoy_listener_downstream_listener_filter_error{envoy_listener_address="0.0.0.0_19001"} 0

arkodg · 2024-01-29T17:40:14Z

cc @zhaohuabing

zetaab · 2024-02-16T07:55:19Z

this issue is quite easy to reproduce with small golang code https://gist.github.com/zetaab/814fe46949c8739007f314463b5d8607

and manifests: route.yaml https://gist.github.com/zetaab/75d87db3afd666619f496a8d20e25dcc
and sec.yaml https://gist.github.com/zetaab/5af54e735f316625156678acfa6e6b80

zetaab · 2024-02-16T07:58:00Z

output from execution:

% go run main.go
httproute.gateway.networking.k8s.io/echoserver-ext created
2024/02/16 09:55:53 fetching code
securitypolicy.gateway.envoyproxy.io/echoserver-ext-oidc created
2024/02/16 09:55:54 fetching code
2024/02/16 09:55:54 fetching code
2024/02/16 09:55:55 fetching code
2024/02/16 09:55:55 fetching code
2024/02/16 09:55:56 fetching code
2024/02/16 09:55:56 fetching code
2024/02/16 09:55:57 fetching code
2024/02/16 09:55:57 fetching code
2024/02/16 09:55:58 fetching code
2024/02/16 09:55:58 fetching code

% egctl config envoy-proxy listener -n envoy-gateway-system |grep yy    
            "details": "OAuth2 filter: unknown cluster 'yy_foobar_com_443' in config. Please specify which cluster to direct OAuth requests to.",
                                "authorizationEndpoint": "https://yy.foobar.com/oauth2/authorize",
                                  "cluster": "yy_foobar_com_443",
                                  "uri": "https://yy.foobar.com/oauth2/token"
                                "httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/yy": {
                                      "cluster": "yy_foobar_com_443",
                                      "uri": "https://yy.foobar.com/.well-known/jwks.json"
                                  "providerName": "httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/yy"
            "details": "OAuth2 filter: unknown cluster 'yy_foobar_com_443' in config. Please specify which cluster to direct OAuth requests to.",
                                "authorizationEndpoint": "https://yy.foobar.com/oauth2/authorize",
                                  "cluster": "yy_foobar_com_443",
                                  "uri": "https://yy.foobar.com/oauth2/token"
                                "httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/yy": {
                                      "cluster": "yy_foobar_com_443",
                                      "uri": "https://yy.foobar.com/.well-known/jwks.json"
                                  "providerName": "httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/yy"

I tried to modify https://github.com/envoyproxy/gateway/blob/main/internal/xds/translator/translator.go#L304 to be executed before https://github.com/envoyproxy/gateway/blob/main/internal/xds/translator/translator.go#L162 but it does not help either. I am thinking can this be race condition bug in envoygateway itself?

If I check egctl config envoy-proxy cluster -n envoy-gateway-system the configuration is there. These are just added in incorrect order and it does not "refresh" the listeners after that

zetaab · 2024-02-16T08:15:52Z

the logs from envoy proxy itself

[2024-02-16 08:08:44.425][1][debug][config] [source/extensions/config_subscription/grpc/new_grpc_mux_impl.cc:92] Received DeltaDiscoveryResponse for type.googleapis.com/envoy.config.listener.v3.Listener at version 115
[2024-02-16 08:08:44.425][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:542] begin add/update listener: name=envoy-gateway-system-external/external/https hash=6130871143298778987
[2024-02-16 08:08:44.425][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:574] use in place update filter chain update path for listener name=envoy-gateway-system-external/external/https hash=6130871143298778987
[2024-02-16 08:08:44.426][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:137]   filter #0:
[2024-02-16 08:08:44.426][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:138]     name: envoy.filters.listener.proxy_protocol
[2024-02-16 08:08:44.426][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:168]   config: {"@type":"type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol"}
[2024-02-16 08:08:44.426][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:137]   filter #1:
[2024-02-16 08:08:44.426][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:138]     name: envoy.filters.listener.tls_inspector
[2024-02-16 08:08:44.426][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:168]   config: {"@type":"type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"}
[2024-02-16 08:08:44.426][1][debug][init] [source/common/init/manager_impl.cc:24] added shared target SdsApi envoy-gateway-system-external/default-tls to init manager Listener-local-init-manager envoy-gateway-system-external/external/https 6130871143298778987
[2024-02-16 08:08:44.427][1][debug][config] [source/extensions/transport_sockets/tls/ssl_socket.cc:465] Secret is updated.
[2024-02-16 08:08:44.427][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:89]   filter #0:
[2024-02-16 08:08:44.427][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:102]     name: envoy.filters.network.http_connection_manager
[2024-02-16 08:08:44.427][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:105]   config: {"@type":"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager","stat_prefix":"https","rds":{"config_source":{"ads":{},"resource_api_version":"V3"},"route_config_name":"envoy-gateway-system-external/external/https"},"http_filters":[{"name":"envoy.filters.http.oauth2_httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com","typed_config":{"@type":"type.googleapis.com/envoy.extensions.filters.http.oauth2.v3.OAuth2","config":{"token_endpoint":{"uri":"https://yy.example.com/oauth2/token","cluster":"yy_example_com_443","timeout":"10s"},"authorization_endpoint":"https://yy.example.com/oauth2/authorize","credentials":{"client_id":"8g5wl57tcxTVVa5AK7Pwd8PC","token_secret":{"name":"httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/oauth2/client_secret","sds_config":{"ads":{},"resource_api_version":"V3"}},"hmac_secret":{"name":"httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/oauth2/hmac_secret","sds_config":{"ads":{},"resource_api_version":"V3"}}},"redirect_uri":"%REQ(x-forwarded-proto)%://%REQ(:authority)%/oauth2/callback","redirect_path_matcher":{"path":{"exact":"/oauth2/callback"}},"signout_path":{"path":{"exact":"/logout"}},"forward_bearer_token":true,"auth_scopes":["openid","email","profile","groups"],"auth_type":"BASIC_AUTH"}}},{"name":"envoy.filters.http.jwt_authn","typed_config":{"@type":"type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication","providers":{"httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/kaas":{"remote_jwks":{"http_uri":{"uri":"https://yy.example.com/.well-known/jwks.json","cluster":"yy_example_com_443","timeout":"10s"},"cache_duration":"300s","async_fetch":{},"retry_policy":{}},"forward":true,"claim_to_headers":[{"header_name":"x-email","claim_name":"email"},{"header_name":"x-groups","claim_name":"groups"},{"header_name":"x-name","claim_name":"name"},{"header_name":"x-sub","claim_name":"sub"}]}},"requirement_map":{"httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com":{"provider_name":"httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/kaas"}}}},{"name":"envoy.filters.http.router","typed_config":{"@type":"type.googleapis.com/envoy.extensions.filters.http.router.v3.Router","suppress_envoy_headers":true}}],"http2_protocol_options":{"max_concurrent_streams":100,"initial_stream_window_size":65536,"initial_connection_window_size":1048576},"access_log":[{"name":"envoy.access_loggers.file","typed_config":{"@type":"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog","path":"/dev/stdout","log_format":{"text_format_source":{"inline_string":"{\"start_time\":\"%START_TIME%\",\"method\":\"%REQ(:METHOD)%\",\"x-envoy-origin-path\":\"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\"protocol\":\"%PROTOCOL%\",\"response_code\":\"%RESPONSE_CODE%\",\"response_flags\":\"%RESPONSE_FLAGS%\",\"response_code_details\":\"%RESPONSE_CODE_DETAILS%\",\"connection_termination_details\":\"%CONNECTION_TERMINATION_DETAILS%\",\"upstream_transport_failure_reason\":\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\"bytes_received\":\"%BYTES_RECEIVED%\",\"bytes_sent\":\"%BYTES_SENT%\",\"duration\":\"%DURATION%\",\"x-envoy-upstream-service-time\":\"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\"x-forwarded-for\":\"%REQ(X-FORWARDED-FOR)%\",\"user-agent\":\"%REQ(USER-AGENT)%\",\"x-request-id\":\"%REQ(X-REQUEST-ID)%\",\":authority\":\"%REQ(:AUTHORITY)%\",\"upstream_host\":\"%UPSTREAM_HOST%\",\"upstream_cluster\":\"%UPSTREAM_CLUSTER%\",\"upstream_local_address\":\"%UPSTREAM_LOCAL_ADDRESS%\",\"downstream_local_address\":\"%DOWNSTREAM_LOCAL_ADDRESS%\",\"downstream_remote_address\":\"%DOWNSTREAM_REMOTE_ADDRESS%\",\"requested_server_name\":\"%REQUESTED_SERVER_NAME%\",\"route_name\":\"%ROUTE_NAME%\"}\n"}}}}],"use_remote_address":true,"upgrade_configs":[{"upgrade_type":"websocket"}],"normalize_path":true,"merge_slashes":true,"server_header_transformation":"PASS_THROUGH","common_http_protocol_options":{"headers_with_underscores_action":"REJECT_REQUEST"},"path_with_escaped_slashes_action":"UNESCAPE_AND_REDIRECT"}
[2024-02-16 08:08:44.428][1][debug][init] [source/common/init/manager_impl.cc:24] added shared target RdsRouteConfigSubscription RDS init envoy-gateway-system-external/external/https to init manager Listener-local-init-manager envoy-gateway-system-external/external/https 6130871143298778987
[2024-02-16 08:08:44.428][1][debug][config] [./source/common/http/filter_chain_helper.h:111]     http filter #0
[2024-02-16 08:08:44.428][1][debug][init] [source/common/init/manager_impl.cc:24] added shared target SdsApi httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/oauth2/client_secret to init manager Listener-local-init-manager envoy-gateway-system-external/external/https 6130871143298778987
[2024-02-16 08:08:44.428][1][debug][init] [source/common/init/manager_impl.cc:24] added shared target SdsApi httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/oauth2/hmac_secret to init manager Listener-local-init-manager envoy-gateway-system-external/external/https 6130871143298778987
[2024-02-16 08:08:44.429][1][debug][secret] [./source/common/secret/secret_manager_impl.h:130] Unregister secret provider. hash key: 683430673790048586.httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/oauth2/hmac_secret
[2024-02-16 08:08:44.429][1][debug][init] [source/common/init/target_impl.cc:68] shared target SdsApi httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/oauth2/hmac_secret destroyed
[2024-02-16 08:08:44.429][1][debug][secret] [./source/common/secret/secret_manager_impl.h:130] Unregister secret provider. hash key: 683430673790048586.httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/oauth2/client_secret
[2024-02-16 08:08:44.429][1][debug][init] [source/common/init/target_impl.cc:68] shared target SdsApi httproute/echoserver/echoserver-ext/rule/0/match/0/foobar_com/oauth2/client_secret destroyed
[2024-02-16 08:08:44.429][1][debug][init] [source/common/init/watcher_impl.cc:31] Listener-local-init-watcher envoy-gateway-system-external/external/https destroyed
[2024-02-16 08:08:44.429][1][debug][init] [source/common/init/watcher_impl.cc:31] init manager Listener-local-init-manager envoy-gateway-system-external/external/https 6130871143298778987 destroyed
[2024-02-16 08:08:44.429][1][debug][init] [source/common/init/target_impl.cc:34] target  destroyed
[2024-02-16 08:08:44.429][1][warning][config] [source/extensions/config_subscription/grpc/delta_subscription_state.cc:269] delta config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) envoy-gateway-system-external/external/https: OAuth2 filter: unknown cluster 'yy_example_com_443' in config. Please specify which cluster to direct OAuth requests to.

[2024-02-16 08:08:44.429][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) envoy-gateway-system-external/external/https: OAuth2 filter: unknown cluster 'yy_example_com_443' in config. Please specify which cluster to direct OAuth requests to.

[2024-02-16 08:08:44.429][1][debug][config] [source/extensions/config_subscription/grpc/new_grpc_mux_impl.cc:92] Received DeltaDiscoveryResponse for type.googleapis.com/envoy.config.route.v3.RouteConfiguration at version 115
[2024-02-16 08:08:44.429][1][debug][rds] [source/common/rds/rds_route_config_subscription.cc:91] rds: loading new configuration: config_name=envoy-gateway-system-external/external/https hash=560963842472378334
[2024-02-16 08:08:44.429][1][debug][config] [source/extensions/config_subscription/grpc/delta_subscription_state.cc:262] Delta config for type.googleapis.com/envoy.config.route.v3.RouteConfiguration accepted with 1 resources added, 0 removed
[2024-02-16 08:08:44.430][1][debug][config] [source/extensions/config_subscription/grpc/new_grpc_mux_impl.cc:92] Received DeltaDiscoveryResponse for type.googleapis.com/envoy.config.cluster.v3.Cluster at version 115
[2024-02-16 08:08:44.430][1][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 1 cluster(s), remove 0 cluster(s)
[2024-02-16 08:08:44.437][1][debug][config] [./source/common/http/filter_chain_helper.h:111]     upstream http filter #0
[2024-02-16 08:08:44.437][1][debug][config] [./source/common/http/filter_chain_helper.h:158]       name: envoy.filters.http.upstream_codec
[2024-02-16 08:08:44.437][1][debug][config] [./source/common/http/filter_chain_helper.h:161]     config: {"@type":"type.googleapis.com/envoy.extensions.filters.http.upstream_codec.v3.UpstreamCodec"}
[2024-02-16 08:08:44.438][1][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:825] add/update cluster yy_example_com_443 starting warming
[2024-02-16 08:08:44.438][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:365] dns resolution for yy.example.com started
[2024-02-16 08:08:44.438][1][debug][upstream] [source/common/upstream/cds_api_helper.cc:51] cds: add/update cluster 'yy_example_com_443'

we can clearly see that the order of the calls are incorrect. Clusters should be updated before listeners

arkodg · 2024-02-16T17:54:21Z

thanks for sharing the logs @zetaab ! this really helps
@zhaohuabing wasn't the xds resource ordering issue solved with envoyproxy/go-control-plane#801 & envoyproxy/go-control-plane#752 ?

zetaab · 2024-02-16T21:30:41Z

@arkodg I tried to look this little bit more. Now I am somewhere in https://github.com/envoyproxy/go-control-plane/blob/main/pkg/cache/v3/simple.go#L301-L304

the issue is that cache.ads is not enabled (debugging says that value for that variable is false) and based on that text when ads is not enabled the order is not guaranteed. That might be the issue.

zetaab added the triage label Jan 28, 2024

tmsnan added the kind/bug Something isn't working label Jan 29, 2024

arkodg added the road-to-ga label Feb 16, 2024

github-project-automation bot added this to Envoy Gateway: The Road to GA Feb 16, 2024

github-project-automation bot moved this to Todo in Envoy Gateway: The Road to GA Feb 16, 2024

arkodg added this to the v1.0.0-rc1 milestone Feb 16, 2024

zetaab mentioned this issue Feb 16, 2024

use ADS cache to ensure the rule order #2634

Merged

arkodg closed this as completed in #2634 Feb 17, 2024

github-project-automation bot moved this from Todo to Done in Envoy Gateway: The Road to GA Feb 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

envoyproxy needs restart before securitypolicy takes action #2521

envoyproxy needs restart before securitypolicy takes action #2521

zetaab commented Jan 28, 2024 •

edited

Loading

tmsnan commented Jan 29, 2024 •

edited

Loading

zetaab commented Jan 29, 2024 •

edited

Loading

arkodg commented Jan 29, 2024

zetaab commented Feb 16, 2024

zetaab commented Feb 16, 2024 •

edited

Loading

zetaab commented Feb 16, 2024 •

edited

Loading

arkodg commented Feb 16, 2024 •

edited

Loading

zetaab commented Feb 16, 2024 •

edited

Loading

envoyproxy needs restart before securitypolicy takes action #2521

envoyproxy needs restart before securitypolicy takes action #2521

Comments

zetaab commented Jan 28, 2024 • edited Loading

tmsnan commented Jan 29, 2024 • edited Loading

zetaab commented Jan 29, 2024 • edited Loading

arkodg commented Jan 29, 2024

zetaab commented Feb 16, 2024

zetaab commented Feb 16, 2024 • edited Loading

zetaab commented Feb 16, 2024 • edited Loading

arkodg commented Feb 16, 2024 • edited Loading

zetaab commented Feb 16, 2024 • edited Loading

zetaab commented Jan 28, 2024 •

edited

Loading

tmsnan commented Jan 29, 2024 •

edited

Loading

zetaab commented Jan 29, 2024 •

edited

Loading

zetaab commented Feb 16, 2024 •

edited

Loading

zetaab commented Feb 16, 2024 •

edited

Loading

arkodg commented Feb 16, 2024 •

edited

Loading

zetaab commented Feb 16, 2024 •

edited

Loading