Wrong role is using instead of s3-csi-driver-sa

[touch-dev-null]

/kind bug

What happened?
Followed by instructions: https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html

Created:
a) Created policy
b) AWS Role - AmazonEKS_S3_CSI_DriverRole:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::XXX:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/YYY"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/YYY:aud": "sts.amazonaws.com",
                    "oidc.eks.us-east-1.amazonaws.com/id/YYY:sub": "system:serviceaccount:kube-system:s3-csi-*"
                }
            }
        }
    ]
}

c) Installed Add On via AWS Web Console

I am able to see new pods "s3-csi-node-..." and it looks like it is working, but I have an issue during run my test pod:

Warning  FailedMount  105s (x12 over 10m)  kubelet            MountVolume.SetUp failed for volume "s3-pv-<masked-bucket-name>" : rpc error: code = Internal desc = Could not mount "<masked-bucket-name>" at "/var/lib/kubelet/pods/cfbcbfd9-95c6-46e5-9809-c7cd5032c56e/volumes/kubernetes.io~csi/s3-pv-<masked-bucket-name>/mount": Mount failed: Failed to start service output: Error: Failed to create S3 client  Caused by:     0: initial ListObjectsV2 failed for bucket <masked-bucket-name> in region us-east-1     1: Client error     2: Forbidden: User: arn:aws:sts::XXX:assumed-role/EKSNodeDevRole/i-07b8b3358b26ae9ab is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::<masked-bucket-name>" because no identity-based policy allows the s3:ListBucket action Error: Failed to create mount process

Most important thing here - "User: arn:aws:sts::XXX:assumed-role/EKSNodeDevRole". Why it try to use this role instead of "AmazonEKS_S3_CSI_DriverRole" ?

Daemonset:

Name:           s3-csi-node
Selector:       app=s3-csi-node,app.kubernetes.io/instance=aws-mountpoint-s3-csi-driver,app.kubernetes.io/name=aws-mountpoint-s3-csi-driver
Node-Selector:  kubernetes.io/os=linux
Labels:         app.kubernetes.io/component=csi-driver
                app.kubernetes.io/instance=aws-mountpoint-s3-csi-driver
                app.kubernetes.io/managed-by=EKS
                app.kubernetes.io/name=aws-mountpoint-s3-csi-driver
Annotations:    deprecated.daemonset.template.generation: 4
Desired Number of Nodes Scheduled: 5
Current Number of Nodes Scheduled: 5
Number of Nodes Scheduled with Up-to-date Pods: 5
Number of Nodes Scheduled with Available Pods: 5
Number of Nodes Misscheduled: 0
Pods Status:  5 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=s3-csi-node
                    app.kubernetes.io/component=csi-driver
                    app.kubernetes.io/instance=aws-mountpoint-s3-csi-driver
                    app.kubernetes.io/managed-by=EKS
                    app.kubernetes.io/name=aws-mountpoint-s3-csi-driver
  Annotations:      kubectl.kubernetes.io/restartedAt: 2024-11-22T10:46:14-05:00
  Service Account:  s3-csi-driver-sa
  Init Containers:

Serviceaccount:

Name:                s3-csi-driver-sa
Namespace:           kube-system
Labels:              app.kubernetes.io/component=csi-driver
                     app.kubernetes.io/instance=aws-mountpoint-s3-csi-driver
                     app.kubernetes.io/managed-by=EKS
                     app.kubernetes.io/name=aws-mountpoint-s3-csi-driver
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::XXX:role/AmazonEKS_S3_CSI_DriverRole
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

My test pod:

apiVersion: v1
kind: Pod
metadata:
  name: s3-app-new
  namespace: kube-system
spec:
  serviceAccountName: s3-csi-driver-sa
...

All "tests" currently in one namespace - kube-system

Also, I can see in AWS WEB Console "Last activity": - (none) for AmazonEKS_S3_CSI_DriverRole

So, for some reason - this "plugin" is using my Cluster Node role instead of AmazonEKS_S3_CSI_DriverRole.

Any suggestions?

Environment

• Kubernetes version (use kubectl version): 1.30
• Driver version:
• v1.8.1-eksbuild.1
• v1.9.0-eksbuild.1

[unexge]

Hey @touch-dev-null, thanks for reporting the issue.

Could you please try updating your trust policy's subject field to specify the full service account name instead of using *:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::XXX:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/YYY"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/YYY:aud": "sts.amazonaws.com",
-                    "oidc.eks.us-east-1.amazonaws.com/id/YYY:sub": "system:serviceaccount:kube-system:s3-csi-*"
+                    "oidc.eks.us-east-1.amazonaws.com/id/YYY:sub": "system:serviceaccount:kube-system:s3-csi-driver-sa"
                }
            }
        }
    ]
}

I think the documentation states that you need to change condition from StringEquals to StringLike but it's not very clear and easy to miss. We'll update the documentation to make it clearer.

[touch-dev-null]

@unexge
Hah, interesting. I tried this solution (I found it here: #173) but it was not helped. Had same situation. Note: I tried to update role after I install and configure everything. Of course - to make sure - I recreated pv/pvc after that, restarted all s3-csi-node-xxx and re-deploy my pod.

Now, I uninstalled plugin and tried to install it with helm. Same result. Helm was uninstalled.

Now I got your answer, updated role again with full name, install plugin - and now it is working =)

Anyway, thank you for response, and yes, it looks like it is good idea to update doc to use full name! Thank you!

[jiaeenie]

The documentation has been updated to clarify the confusion caused. This issue is now resolved and closed.