Support GKE Workload Identity

[hermanbanken]

Proposal

https://keda.sh/docs/1.4/concepts/authentication/#pod-authentication-providers lists multiple Pod Identity providers, but none is available for GCP/GKE clusters. My proposal is to add "google" as an option in podIdentity.provider.

Use-Case

Running a Google Cloud PubSub Queue scaler on GKE without hardcoding credentials.

Anything else?

Some people are having the desire for the same feature: https://stackoverflow.com/questions/67486922/how-to-setup-keda-authentication-with-gcp-pubsub-when-deployments-are-using-work.

I tried to figure out how the Azure integration works, but I have too little knowledge of the Azure platform to port PR #275 to GKE's Workload Identity. If "podIdentity" is used, will the operator then start a pod to retrieve the identity from? If that is the case, then the configuration probably needs to include a setting which serviceAccountName needs to be bound to said pod. In GKE Workload Identity the ServiceAccount that is assigned to the pod will be assigned a token that can be used for remote IAM-authorized requests.

[tomkerkhove]

That's a fair and nice request!

I tried to figure out how the Azure integration works, but I have too little knowledge of the Azure platform to port PR #275 to GKE's Workload Identity. If "podIdentity" is used, will the operator then start a pod to retrieve the identity from? If that is the case, then the configuration probably needs to include a setting which serviceAccountName needs to be bound to said pod. In GKE Workload Identity the ServiceAccount that is assigned to the pod will be assigned a token that can be used for remote IAM-authorized requests.

For Azure, the identity is assigned to KEDA itself since that is the client to get the metrics with. I don't know how GKE works, do you have a link to the corresponding docs?

[hermanbanken]

The documentation is at https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity, but I'd say that the most important thing to know is that you configure the cluster (addon) to inject different tokens for Kubernetes ServiceAccounts.

I don't know exactly at which time the injection happens. It could be both the file available in the ServiceAccount mount or the data in the Metadata API.

However, any Google client library will be able to handle this automatically.

If one correctly provisions the SA (out of scope for this feature, but maybe not for the docs) then "it just works" if the client libraries are used to query the subscription from Cloud PubSub, or any other granted Google service for that matter.

[arschles]

@hermanbanken in the case when workload identity is injected, do you see a valid way to get the Google-standard GOOGLE_APPLICATION_CREDENTIALS environment variable to show up in the pod? that might be wishful thinking, but it would make this relatively easy from an implementation perspective.

[coderanger]

There is no env var, it works via the magic metadata servers. GKE just captures those requests and annotates them with the post it originated from. There is no way to inherit the creds from another pod, you have to grant Keda's pods the appropriate access themselves.

[arschles]

yea, figured as much. oh well

[glindstedt]

Started looking into keda but ran into this roadblock, so chiming in here with our use case, although it might be a bit niche. We use Identity-Aware Proxy extensively for all our self-hosted services and have our prometheus service behind it. So in order for us to be able to use the prometheus scaler without exposing it in a different way keda would need to support Workload Identity (with custom audience) for generic HTTP requests. Example from google docs: https://cloud.google.com/iap/docs/authentication-howto#iap_make_request-go

The configuration would need to be a bit different from the podIdentity used for the pubsub scaler, as you'd need to provide the audience for the specific IAP as well. But assuming the configuration could be provided it could be used when creating the client here: https://github.com/kedacore/keda/blob/main/pkg/util/http.go#L35

But currently the prometheus scaler replaces the Transport here https://github.com/kedacore/keda/blob/main/pkg/scalers/prometheus_scaler.go#L91
The roundtripper would need to be chained instead of replaced. Another potential issue is if someone decides to use IAP auth as well as prometheus bearer auth mode, since both use the Authorization header. IAP does support the Proxy-Authorization header though but I don't think it is the default. I found this example on how to use it though googleapis/google-api-go-client#1020

This doesn't seem like a "quick fix" thing though, might look into replicating the prometheus scaler functionality in an external scaler where we can control the http client.

[hermanbanken]

Already fixed in #2225. Thanks 🙏