Better segregation of permissions when using managed identities

[stephaneey]

Proposal

Today, when using TriggerAuthentication with podIdentity, Keda is using the operator's associated managed identity to authenticate against the monitored stores (ie: Storage Account, Message Brokers, etc.). When using Keda at scale, this results is granting way too many permissions to a single identity. Keda is used by multiple workloads of different nature with different data confidentiality levels. The current implementation is not in line with the least privilege principle.

What about modifying the current TriggerAuthentication CRD and add a client_id field to let users specify a workload-specific identity to be used. To take a concrete example, in azure_aad_podidentity.go, Azure's MSI endpoint is declared as follows:

const (
	msiURL = "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=%s"
)

What about adding the optional client_id parameter to the query string (as documented here https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token). This could work the following way:

• If any identity is specified at the level of the TriggerAuthentication resource, you use it to perform the call
• If no identity is specified at the level of the TriggerAuthentication, you fallback on the operator's associated identity

I might be over simplifying and this may not work with all Cloud providers but this would allow us to better segregate permissions.

Use-Case

Achieve the same level of granularity as with the secret/config map approach but using managed identities.

Anything else?

No response

[tomkerkhove]

That seems like a doable thing - Are you open to contributing this?

I would use the following approach:

podIdentity:
  provider: azure # Optional. Default: false
  identityId: <id> # Optional. Allows end-users to specific a user-assigned identity different from the default KEDA identity.

[stephaneey]

Exactly :). That would be great!

[stephaneey]

I will check more in detail to see if I can contribute :)

[tomkerkhove]

Any thoughts on this @zroubalik @JorTurFer? Don't think there should be an issue but just double-checking.

[JorTurFer]

From high level seems doable, and it could help to reduce the amount of permissions under the same identity.

Anyway, we should maintain also current behavior, something like: if other identity is set, use it, if not, use the current approach.

[stephaneey]

I started looking into this and I validated the approach through a quick hack just to make sure it would work and yes, it seems doable. I have crafted the high level architecture & config to get it working. I'll dive into it over the weekend.

[stephaneey]

Hello @tomkerkhove @JorTurFer

I got it working in my environment (AKS 1.21.7) but I did touch a serious number of files....and I face one limitation with Azure Service Bus because of the TokenProvider interface which is forcing GetToken to pass a target audience only, so no way to pass a client id on top of it.

However, Azure Service Bus works as usual with its default support of TriggerAuthentication. Before making the proposal, I did not realize that only Azure Storage & Service Bus did support TriggerAuthentication and with the SB limitation, this proposal would only be suitable for Azure Storage (at this stage), so I'm not sure it would add much value.

Overall, here is what I did:

where all the operator-related identities must reference a single selector in their AzureIdentityBinding CRD, which must match the aadpodidbinding label defined in the operator (.-set podIdentity.activeDirectory.identity=autoscaler-aad-identity when installing KEDA). Thanks to this, the operator is able to leverage any identity associated to the selector. If nothing is specified in the TriggerAuthentication CRD, the call to NMI does not include any clientid, which makes NMI fallback on the first identity available in ascending order...

This means that this is only usable if workload-related identities have a name that is "bigger" than the default operator identity. Any "lower" name would be picked by NMI if not explicit identity is specified in the TriggerAuthentication resource.

Let me know your thoughts.
Thanks

[tomkerkhove]

However, Azure Service Bus works as usual with its default support of TriggerAuthentication. Before making the proposal, I did not realize that only Azure Storage & Service Bus did support TriggerAuthentication and with the SB limitation, this proposal would only be suitable for Azure Storage (at this stage), so I'm not sure it would add much value.

This is only for now, though, as we should end up supporting all of the Azure scalers with managed identity (and I thought we already did?). Feel free to open issues for them

where all the operator-related identities must reference a single selector in their AzureIdentityBinding CRD, which must match the aadpodidbinding label defined in the operator (.-set podIdentity.activeDirectory.identity=autoscaler-aad-identity when installing KEDA). Thanks to this, the operator is able to leverage any identity associated to the selector. If nothing is specified in the TriggerAuthentication CRD, the call to NMI does not include any clientid, which makes NMI fallback on the first identity available in ascending order...

This means that this is only usable if workload-related identities have a name that is "bigger" than the default operator identity. Any "lower" name would be picked by NMI if not explicit identity is specified in the TriggerAuthentication resource.

Do you have any sample YAMLs of what it looks like? Overall I think this looks OK.

[stephaneey]

Hi @tomkerkhove

Sorry for my late reply but I've been rather busy :). The below YAML corresponds to the following scenario:

• The data sources are 2 Azure Storage Accounts with their own queue
• 1 handling function (deployed as two separate deployments) using its associated managed identity for both storage accounts. The target account and managed identity to be used are passed as K8s secrets
• 2 scaled objects to bind the storage accounts with their handlers
• 2 TriggerAuthentication to specify identities which should be used by the scaler.

#SECRETS USED BY THE HANDLERS TO DEQUEUE MESSAGES USING MI. 
#THERE ARE IN FACT NO SECRET VALUE IN THESE SECRETS
---
data:
  AzureWebJobsStorage: VXNlRGV2ZWxvcG1lbnRTdG9yYWdlPXRydWU=
  MyStorageConnection__credential: bWFuYWdlZGlkZW50aXR5
  MyStorageConnection__clientId: MTViYTFhMjAtN2YxMi00NTQ2LTliY2QtOWQ5OWQ3NTVlNmUw
  MyStorageConnection__serviceUri: aHR0cHM6Ly9zZXlzaXRlY29yZWtlZGEucXVldWUuY29yZS53aW5kb3dzLm5ldC8=
  FUNCTIONS_WORKER_RUNTIME: ZG90bmV0
apiVersion: v1
kind: Secret
metadata:
  name: kedaqueuesecrethandler1
---
data:
  AzureWebJobsStorage: VXNlRGV2ZWxvcG1lbnRTdG9yYWdlPXRydWU=
  MyStorageConnection__credential: bWFuYWdlZGlkZW50aXR5
  MyStorageConnection__clientId: MDljODJhZmQtYzBiYS00YmVkLTk3ODMtY2M3YzFhNWJmNzVl
  MyStorageConnection__serviceUri: aHR0cHM6Ly9zZXlzaXRlY29yZWtlZGEyLnF1ZXVlLmNvcmUud2luZG93cy5uZXQv
  FUNCTIONS_WORKER_RUNTIME: ZG90bmV0
apiVersion: v1
kind: Secret
metadata:
  name: kedaqueuesecrethandler2
---
#SCALER IDENTITIES, BINDINGS TO BE PLACED WITHIN KEDA-SYSTEM
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: keda-scaler-storage1
  namespace: keda-system
  annotations:
    aadpodidentity.k8s.io/Behavior: namespaced
spec:
  type: 0
  resourceID: /subscriptions/f63a908a------------a1fd-1eadaee67ffc/resourceGroups/azlab/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kedastorage1
  clientID: a5ebf646-440a-485d-9527-203dddf1333f
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: keda-scaler-storage2
  namespace: keda-system
  annotations:
    aadpodidentity.k8s.io/Behavior: namespaced
spec:
  type: 0
  resourceID: /subscriptions/f63a908a------------a1fd-1eadaee67ffc/resourceGroups/azlab/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kedastorage2
  clientID: 07ad808f-105d-4f72-8dde-a8ec6daaeed8
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: keda-scaler-storage1-binding  
  namespace: keda-system
spec:
  azureIdentity: keda-scaler-storage1
  selector: autoscaler-aad-identity
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: keda-scaler-storage2-binding  
  namespace: keda-system
spec:
  azureIdentity: keda-scaler-storage2
  selector: autoscaler-aad-identity
---
#TRIGGERAUTH TO BE PLACED IN WORKLOAD NAMESPACE
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: kedastorage1ta
spec:
  podIdentity:
    provider: azure
    identityId: a5ebf646-440a-485d-9527-203dddf1333f   
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: kedastorage2ta
spec:
  podIdentity:
    provider: azure
    identityId: 07ad808f-105d-4f72-8dde-a8ec6daaeed8
---
#HANDLER IDENTITIES & BINDINGS
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: kedastoragehandler1
  annotations:
    aadpodidentity.k8s.io/Behavior: namespaced
spec:
  type: 0
  resourceID: /subscriptions/f63a908a------------a1fd-1eadaee67ffc/resourceGroups/azlab/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kedastoragehandler1
  clientID: 15ba1a20-7f12-4546-9bcd-9d99d755e6e0
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: kedastoragehandler2 
  annotations:
    aadpodidentity.k8s.io/Behavior: namespaced
spec:
  type: 0
  resourceID: /subscriptions/f63a908a------------a1fd-1eadaee67ffc/resourceGroups/azlab/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kedastoragehandler2
  clientID: 09c82afd-c0ba-4bed-9783-cc7c1a5bf75e
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: kedastoragehandler1-binding  
spec:
  azureIdentity: kedastoragehandler1
  selector: kedastoragehandler1
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: kedastoragehandler2-binding  
spec:
  azureIdentity: kedastoragehandler2
  selector: kedastoragehandler2
---
#DEPLOYMENTS - HANDLERS
apiVersion: apps/v1
kind: Deployment
metadata:
  name: handler1
  labels:
    app: handler1    
spec:
  selector:
    matchLabels:
      app: handler1
  template:
    metadata:
      labels:
        app: handler1
        aadpodidbinding: kedastoragehandler1
    spec:
      containers:
      - name: miqueuereader
        image: stephaneey/miqueuereader
        #image: stephaneey/msifunc
        env:
        - name: AzureFunctionsJobHost__functions__0
          value: Function1
        envFrom:
        - secretRef:
            name: kedaqueuesecrethandler1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: handler2
  labels:
    app: handler2    
spec:
  selector:
    matchLabels:
      app: handler2
  template:
    metadata:
      labels:
        app: handler2
        aadpodidbinding: kedastoragehandler2
    spec:
      containers:
      - name: miqueuereader
        image: stephaneey/miqueuereader
        #image: stephaneey/msifunc
        env:
        - name: AzureFunctionsJobHost__functions__0
          value: Function1
        envFrom:
        - secretRef:
            name: kedaqueuesecrethandler2
---
#SCALEDOBJECTS
apiVersion: keda.sh/v1alpha1 
kind: ScaledObject
metadata:
  name: handler1scaler
spec:
  scaleTargetRef:
    name: handler1  
  maxReplicaCount: 10
  cooldownPeriod: 60
  triggers:
  - type: azure-queue
    metadata:
      type: queueTrigger     
      queueName: myqueue-items
      accountName: seysitecorekeda
      name: myQueueItem
      queueLength: '5'    
    authenticationRef:
      name: kedastorage1ta
---
apiVersion: keda.sh/v1alpha1 
kind: ScaledObject
metadata:
  name: handler2scaler
spec:
  scaleTargetRef:
    name: handler2  
  maxReplicaCount: 10
  cooldownPeriod: 60
  triggers:
  - type: azure-queue
    metadata:
      type: queueTrigger     
      queueName: myqueue-items
      accountName: seysitecorekeda2
      name: myQueueItem
      queueLength: '5'    
    authenticationRef:
      name: kedastorage2ta
---

I have deleted all the resources so no issue to "reveal" sensitive information here, I have just obfuscated my subscription ID. The above YAML represents. Here is a diagram that illustrates the above YAML:

To make things simple, I did not include a separate triggerauthentication block without the explicit MI to be used but it works fine (fallback scenario on the diagram).

If we go further, I'll post a complete example, including the infrastructure as code bits to deploy the corresponding Azure Components.

Best Regards

[tomkerkhove]

This looks good to me, what do you think @JorTurFer?

[tomkerkhove]

Are you willing to contribute this @stephaneey?

[stephaneey]

Hi @tomkerkhove

Yes of course but I was waiting for @JorTurFer 's feedback. Do you want me to start the pull request process?

Thanks

[tomkerkhove]

I think it looks pretty accurate, let's start the feature and doc PR as I don't expect big changes - Sounds good?

[stephaneey]

Ok, I'll try to move on with this over the weekend

[tomkerkhove]

No rush, next week or later is fine as I was checking who would pick it up

[JorTurFer]

Sorry for the delay. This issue was missing, maybe I marked as readed by error... 😔
The idea sounds good ♥️

[tomkerkhove]

The initial work was done by @stephaneey and is available in the azure-identity-seperation branch but we would need somebody to continue the work since he cannot finish it.

@raorugan @v-shenoy Is this something your team can pick up? This is currently just for Pod Identity (if I'm not mistaken) but would be nice if we could support this for workload identity as well.

[wsugarman]

I have been lurking on this issue and PR, but it would be great to see this merged. I have been waiting to add KEDA scalers, but I am reluctant to add an AAD identity that has permissions for every app's resources on our cluster.

[tomkerkhove]

It's definitely coming but we just need somebody to complete the work so it's a matter of time and should definitely make it in to 2.8.

[v-shenoy]

I will try to take this up after #2907 gets merged.

[tomkerkhove]

Thanks!

[wsugarman]

Any update on when this will be picked up @v-shenoy?

[v-shenoy]

I will start working on this, and will make sure it's a part of the next release. Not sure about the release date. @kedacore/keda-core-contributors can help you there.

[tomkerkhove]

The next release will be at early August so we have time:
https://github.com/kedacore/keda/blob/main/ROADMAP.md#upcoming-release-cycles

[v-shenoy]

@wsugarman This has been implemented.

[wsugarman]

@v-shenoy That's wonderful news! Thank you very much

[AnthonyDewhirst]

I know this is closed. But I am banging my head trying to get this to work.
Is there any chance or a sample for this to live alongside https://github.com/kedacore/sample-dotnet-worker-servicebus-queue/blob/main/pod-identity.md that can include the above.
I don't want to add the fallback option if possible as I don't want an msi with super permissions. So I have been trying to get this working without the fallback.
If I add the scaledObject and Trigger, my app stops working altogether and gets 403 when connecting to SB and the keda-operator gets constant timeouts (awaiting headers).
So would love to see a sample. So I can first see what is different from what I am doing

[v-shenoy]

Could you share the steps that you performed, and any logs with the errors?

[JorTurFer]

Please, share also the AAD pod identity logs, this case usually is produced because something sin't correctly assigned in pod identity side and in that case, those pods are who have the errors in the logs

[tomkerkhove]

We are discussing this offline and will open a discussion so we can move the conversation there. The main problem is that owner/manage permissions where lacking for the identity used by KEDA.

[JorTurFer]

so, do we need to update the sample? Could you open an issue for that?

[AnthonyDewhirst]

No, I missed the Big obvious Note on the documentation which states the permission needed (hangs head), I am still working out now how I think I want this to work, but will open a discussion as per Toms advice (much appreciated).

One clarification I would like if we could though:
Inside the TriggerAuthentication, the value for spec: podIdentity: identityId, what should this be? The clientId from AzureIdentity: spec: clientID from the same namespace or something else?
When you set up keda using helm and set podIdentity.activeDirectory.identity=app-autoscaler, the value "app-autoscaler" is a selector value, and so the two values don't seem to be aligned even though they are for setting 'similar' settings (unless I am really missing something, which is quite likely)

[JorTurFer]

Inside the TriggerAuthentication, the value for spec: podIdentity: identityId, what should this be? The clientId from AzureIdentity: spec: clientID from the same namespace or something else?

This value is optional and will override the value set during helm installation. This is explained in the Authentication Provider section for AAD Pod Identity

[JorTurFer]

But I think @v-shenoy can explain this better "as the father" of the feature :)

[AnthonyDewhirst]

Yes, this appears to be the part that isn't working for me. I have read this and it's just not clear to me. I wanted to make sure I was using the correct value here, before I raise the issue.
If someone can tell me what value to put here, I will test it, then can provide logs and examples in the other forum.
Appreciate everyone's patience with me on this

[tomkerkhove]

Can we please move this to a discussion given this is unrelated to this issue please? :)

identityId allows you to use another identity than the one that was assigned to KEDA during installation.

[AnthonyDewhirst]

I have created #3776