Feature Request: External Authorization Support

[bcelenza]

If you want to see App Mesh implement this idea, please upvote with a 👍.

Tell us about your request

Once a client has been identified (e.g. via a certificate, request token, or IP address), it would be useful to perform an authorization check to further restrict the scope of access to a given service. For example, restricting certain identified clients to only specific HTTP verbs and paths.

Envoy supports authorizing a client using an external authorization provider, which allows you to bring an existing solution in (such as Open Policy Agent), or write your own.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
After identifying a caller, there is a need to further restrict access to an endpoint such that only certain actions may be performed. Authorization can be performed on HTTP for example by only allowing GET requests on specific paths.

Being able to authorize a client through a separate application provides a flexible interface that can be customized to fit many needs.

Are you currently working around this issue?
This issue can be worked around by implementing authorization in the application code, or some other middle hop, but represents additional complexity that doesn't fit the primary benefits and uses of a service mesh.

[abhijitherekar]

hi @bcelenza , is there a open PR for this? Would be really interested to check this feature.
Please, let me know.

[lavignes]

Hey @abhijitherekar. We're still currently researching how we'll implement this feature in App Mesh. We'll update this issue when we have an API proposal and will happily accept feedback / answer any questions you post here.

[bcelenza]

Hey folks!

We're working on this feature, and looking for your input. 5 easy questions below -- please feel free to answer any that pertain to your needs.

1. Do you require authorization capabilities for traffic coming from outside your mesh, within your mesh, or both?
2. What types of identities do you need to perform an authorization decision on (e.g. X.509 certificate, OAuth bearer token, JWT), and where within inside or outside of the mesh do you use them?
3. For services within the mesh, do your authorization policies vary greatly from service to service, or are they largely the same?
4. Is the component making the authorization decision a local process (i.e. sidecar) or a remote service? If remote, is the service within or outside of the mesh configuration?
5. Do you plan on using an existing solution that integrates with Envoy's external authorization API (e.g. Open Policy Agent)? If not, what solution would you ideally like to use?

Thanks in advance!

[RyanFrench]

Hi Brian.

My team is currently considering migrating to AppMesh, but we would require this feature before we could use the Virtual Gateway for ingress to the mesh. Our requirement is for an auth subrequest for ingress traffic, similar to how the NGINX auth subrequest works. To provide this capability we have been looking at running our own Envoy or NGINX service that has an Envoy sidecar so that it can proxy requests to services within the mesh.

1. We would only require this for traffic coming from outside the mesh.
2. Our authentication and authorisation is done via a JWT contained in the Authorization header of the request. Authentication is done via a single service that exists within the mesh, authorization is handled by each individual microservice in the mesh.
3. Our authorization policies are quite different for services within the mesh, however, each service handles it themselves.
4. Authorization is handled by each service, authentication is handled by a single service that exists within the mesh.
5. Our solution is a custom-built microservice which receives a copy of the Authorization header from the original request and returns either a 200 or 401 HTTP response. If a 200 response is returned then the request continues, otherwise the request is rejected.

[bcelenza]

Thanks for the feedback @RyanFrench!

Can you tell me a little bit more about how you authorize a caller once you’ve authenticated them? For example, do you do authorization based on HTTP verb and path, or is it more complex than that?

Additionally, have you looked into Open Policy Agent for both the authN and authZ functionality? I’m curious how it compares to your existing implementation.

[RyanFrench]

Once the call is authenticated we have very simple information within the JWT that points to the permissions of the caller for each resource they are allowed access to. This typically gives read/write access to a specific resource ID, and the individual service will determine if the service is allowed to read or modify the resource based on what the path does.

We haven't looked into the Open Policy Agent. It does look interesting although I think it's a little bit different in terms of the implementation. Our implementation is done in code, not via configuration.

[jlpettersson]

I would like to have two instances of my app, e.g. two different kubernetes Deployment and then declare JWT-rules, e.g. users with this "claim" are routed to version A and users without that JWT-claim can be routed to the other instance, version B.

In this way, I can configure what users are belonging to an early-test-group, so I can have early features in one Deployment, but only stable features in the other Deployment.

Yes, it would be good if I can use OpenPolicyAgent - but I have not understood if I an achieve this setup yet.

[bcelenza]

Hey @jlpettersson, neat idea. Because Open Policy Agent typically runs on the destination service, I don't think routing based on a JWT claim would work at a Virtual Service -> Virtual Router level in App Mesh (these resources actually result in client configuration).

For your use case, here's what I think would work:

1. Create a Virtual Gateway which will be responsible for routing requests to version A and version B.
2. Integrate that Gateway with OPA. OPA will validate the JWT, and append a header to the request to identify the claim(s).
3. Using a Gateway Route, route the request to the appropriate Virtual Service given the header added by OPA.

Does this sound about like what you're looking for?

Alternatively, we're also looking into native support for JWT (#81) at a later date, which could be an alternative way to do what you're describing (likely using a Virtual Gateway again).

[rajal-amzn]

Hello,
Here is our proposal for enabling authorization of incoming traffic. We propose to enable authorization integration on an AppMesh endpoint’s (Virtual Node or Virtual Gateway) listener using Envoy proxy's external authorization capabilities.

Customers can configure External Authorization using a local sidecar agent that supports Envoy's GRPC Client API. The Authorization sidecar agent could be an Open Policy Agent or customers could bring their own sidecar.

API Design

# On a VirtualNode / Virtual Gateway    
listeners:
- portMapping:
    port: 443
    protocol: http
  # (*NEW*) (*Optional*) Specifies authorization settings for this listener.
  authorization:
    external:
     # (*NEW*) (*Required*) Specifies where to send the authZ request.
     destination:
       local:
         # (*NEW*) (*Required*) Specifies listening port of the authz service
         port: 8181

     # (*NEW*) (*Required*) Specifies external authorization API selection and settings.
     api:
       # (*NEW*) (*Required*)
       xds:
         # (*Required*) Specifies which authorization API and associated options the destination understands.
         # Valid values: V2, V3
         transportVersion: V2

     # (*NEW*) (*Optional*) Specifies timeout for the authorization decision.
     timeout:
       value: 500
       unit: ms

     # (*NEW*) (*Optional*) Specifies how to handle failed calls to the authZ service.
     failure:
       # (*NEW*) (*Optional*) Specifies whether to allow the request when the authZ service fails.
       allow: NONE
       
       # (*NEW*) (*Optional*) Specifies the action to take when a failed call occurs.
       action:
         # (*NEW*) (*Optional*) Specifies the status to return to the client if the AuthZ request fails
         # Valid values: INTERNAL_SERVER_ERROR (500), FORBIDDEN (403)
         returnStatus: INTERNAL_SERVER_ERROR

We'd love to hear your feedback on this proposal to see whether it fit your use cases in the comments.

[danielpops]

@rajal-amzn hello,

I have a scenario where I'm using Envoy and using the ext_authz filter to talk to an opa instance on the host. In our setup we have envoy authenticating end-users via envoy.filters.http.jwt_authn (https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/filter/http/jwt_authn/v2alpha/config.proto). To propagate the verified_jwt payload information from the filter's payload_in_metadata through to the ext_authz filter, we specify metadata_context_namespaces in our ext_authz config.

Our config looks like:

[...]
        http_filters:
        - name: envoy.filters.http.jwt_authn
          typed_config:
            "@type": type.googleapis.com/envoy.config.filter.http.jwt_authn.v2alpha.JwtAuthentication
            providers:
                okta:
                    payload_in_metadata: verified_jwt
                    issuer: "https://yourOktaDomain.okta.com"
                    remote_jwks:
                        cache_duration: 1800s
                        http_uri:
                            uri: https://yourOktaDomain.okta.com/oauth2/v1/keys
                            cluster: jwks_cluster
                            timeout: 60s
            rules:
              - match:
                  prefix: /
                requires:
                  requires_any:
                    requirements:
                      - provider_name: okta
                      - allow_missing_or_failed: {}
        - name: envoy.ext_authz
          typed_config:
            "@type": type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz
            failure_mode_allow: false
            grpc_service:
              envoy_grpc:
                cluster_name: opa_grpc
              timeout: 0.5s
            metadata_context_namespaces:
            - envoy.filters.http.jwt_authn

Will that scenario be covered by your proposal?

[dfezzie]

Hey @danielpops. In this initial iteration of external authorization support I don't believe we would support the metadata_context_namespaces since we do not currently support using the JWT AuthN filter. It definitely should be considered as part of that work though.

[Gurpartap]

One of the use case is to verify external signed http requests.

It requires comparison of the digest value from header with a generated value using the request body. Subsequently, this digest is used to verify the request signature.

Envoy's ext_authz, by default, does not include the request body in the CheckRequest parameter. The with_request_body: { … } option is required for CheckRequest parameter to include the request body.

I hope there'll be an option to enable this with app mesh's extauthz implementation.

https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_authz/v3/ext_authz.proto

- name: envoy.filters.http.ext_authz
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
    transport_api_version: V3
    grpc_service:
      envoy_grpc:
        cluster_name: api-gateway-ext-authz
      timeout: 0.25s
    with_request_body: # 👈
      max_request_bytes: 10240  # 👈
      allow_partial_message: false  # 👈
    failure_mode_allow: false
    include_peer_certificate: true

[danielpops]

When using client certificates to authenticate to the mesh, how will the client information be presented to the external authz interface? Will AppMesh support Envoy's X-Forwarded-Client-Cert?

[CrawX]

@danielpops I'm currently testing App Mesh for some deployments and it does seem AppMesh doesn't set forward_client_cert_details so x-forwarded-client-cert is not being set by envoy.

Sadly, this is limiting the usefulness of mTLS in App Mesh a lot... Any workarounds?

[caleygoff-invitae]

@CrawX @danielpops Thanks for pointing this out. I am also going down the route myself of trying to put Envoy with an opa sidecar in an Appmesh mutated podspec to sit in front of the Appmesh Envoy sidecar for ext_authz stuffs. I also noticed in the opa-agent the X-Forwarded-Client-Cert wasn't set in the input.attributes and was wondering if I was doing something wrong. 🤔

[CrawX]

This would be an immensely useful feature to be able to activate this flag somehow. I'm running Fargate on EKS and App Mesh is the only option (seeing as Istio etc are not supported right now).

The only way I could see this working is to run a patched version of envoy that has a different default for this flag baked in. But I'm relying on IRSA for permissions which is only available in the (closed-source) default envoy image as #182 is not merged upstream yet. So no way to reproduce this build.

Not sure how to proceed yet, not having mTLS (incl. X-Forwarded-Client-Cert) takes one of the massive upsides of AppMesh away from me.

[jalaziz]

Late to the party, but our answers to your questions:

Hey folks!

We're working on this feature, and looking for your input. 5 easy questions below -- please feel free to answer any that pertain to your needs.

1. Do you require authorization capabilities for traffic coming from outside your mesh, within your mesh, or both?

Primarily from outside the mesh.

2. What types of identities do you need to perform an authorization decision on (e.g. X.509 certificate, OAuth bearer token, JWT), and where within inside or outside of the mesh do you use them?

JWT, arbitrary Authorization headers, and some query string use cases. We basically need full ExtAuthz support. We're using these outside the mesh.

3. For services within the mesh, do your authorization policies vary greatly from service to service, or are they largely the same?

Currently, they're the same across a large subset of services. There are services, however, that don't require the auth policies. The auth policies apply to anything that has external ingress through the mesh.

4. Is the component making the authorization decision a local process (i.e. sidecar) or a remote service? If remote, is the service within or outside of the mesh configuration?

The external auth service is deployed as a k8s daemonset. A sidecar approach is feasible, just not as efficient.

5. Do you plan on using an existing solution that integrates with Envoy's external authorization API (e.g. Open Policy Agent)? If not, what solution would you ideally like to use?

We're rolling our own solution. In our particular case, we need to do request signing validation so we need access to the request body as well.

[gurudatta-carbon]

hello, any timelines for this feature ?
it's been 2 yrs

[dushyant8858]

[Request to AppMesh team]

Can you please provide an update on this LONG OPEN ISSUE (~ 2Y 3M)?

Would really like to see how AppMesh team is planning to implement access control? Any progress being done? ETA?

Any AWS or 3rd party document/blog we can refer to see/learn how to implement/integrate AWS AppMesh/Envoy with external Authorization?

@shwetasahuit

[johnpekcan]

Any updates on this issue? @shsahu @suniltheta ? Thanks a ton!

[harjotgill]

Upvoting. This feature will allow the Aperture Flow Control platform to work with AWS AppMesh. Aperture provides advanced flow control features such as prioritized load shedding (adaptive concurrency limiting) and distributed rate limiting. It leverages external authz protocol to make per-request decisions. See documentation and how it integrates with Istio Mesh.

[vnid]

Any update/roadmap on AppMesh to support cross cutting concerns such as external Auth/Rate limit etc?

[herrhound]

@vnid, it's not on our immediate roadmap. We continue collecting customer feedback that will help us prioritize external Auth and other feature requests.

[buffdaddy570]

More answer to the questionnaire above:

Do you require authorization capabilities for traffic coming from outside your mesh, within your mesh, or both?

Right now for traffic from outside the mesh, though I'm sure we'll have use cases for traffic within the mesh in future.

What types of identities do you need to perform an authorization decision on (e.g. X.509 certificate, OAuth bearer token, JWT), and where within inside or outside of the mesh do you use them?

Primarily JWT, but it's likely that X.509 would come in the picture in the near future.

For services within the mesh, do your authorization policies vary greatly from service to service, or are they largely the same?
Varying greatly between services.

Is the component making the authorization decision a local process (i.e. sidecar) or a remote service? If remote, is the service within or outside of the mesh configuration?

Initial idea is as a sidecar, however as daemonset if that is more cost effective or if it improves latency.

Do you plan on using an existing solution that integrates with Envoy's external authorization API (e.g. Open Policy Agent)? If not, what solution would you ideally like to use?

Open Policy Agent is the front runner right now.

[bmihaescu]

Hi team, any plans to support AWS VPC Lattice integration?