Helm - Updating Spark config to force default credentials chain for AWS

Based on the following kustomize change: - 0e2dd72 Updating Spark config to force default credentials chain for AWS (flyteorg#674) Signed-off-by: Sören Brunk <[email protected]>
sbrunk · Apr 12, 2021 · 9701cf8 · 9701cf8
1 parent 9309778
commit 9701cf8
Showing 1 changed file with 4 additions and 0 deletions.
diff --git a/helm/values-eks.yaml b/helm/values-eks.yaml
@@ -504,6 +504,10 @@ configmap:
     plugins:
       spark:
         spark-config-default:
+          - # We override the default credentials chain provider for Hadoop so that
+            # it can use the serviceAccount based IAM role or ec2 metadata based.
+            # This is more in line with how AWS works
+          - spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
           - spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: "2"
           - spark.kubernetes.allocation.batch.size: "50"
           - spark.hadoop.fs.s3a.acl.default: "BucketOwnerFullControl"