Velero cannot authenticate to s3 buckets using IAM integrated service-accounts

[geofffranks]

What steps did you take and what happened:
We're trying to set up velero to talk to S3 using the IAM role/policy given to our velero service account, following the instructions here: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html on our EKS cluster.

What did you expect to happen:
Velero would be able to start and talk to S3. Instead, it is failing.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero

time="2019-10-15T19:51:14Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:405"
time="2019-10-15T19:51:14Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:412"
An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>
  No backups taken, velero won't launch as it can't talk to the S3 bucket

Anything else you would like to add:
Tried this in both velero 1.0.0 and velero 1.1.0, using the following installation params:

./velero install --no-secret --bucket our-s3-bucket --provider aws --backup-location-config region=us-east-1

Environment:

• Velero version (use velero version):

./velero version
Client:
	Version: v1.1.0
	Git commit: a357f21aec6b39a8244dd23e469cc4519f1fe608

• Velero features (use velero client config get features):

./velero client config get features
features: <NOT SET>

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.7-eks-e9b1d0", GitCommit:"e9b1d0551216e1e8ace5ee4ca50161df34325ec2", GitTreeState:"clean", BuildDate:"2019-09-21T08:33:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:
  EKS v1.14
• Cloud provider or hardware configuration:
  AWS
• OS (e.g. from /etc/os-release):
  Amazon Linux 2

[skriss]

@geofffranks take a look at #1962 for some context - the TLDR is that we should be able to support this in v1.2 since we've updated the SDK. v1.2 is scheduled for release around the end of the month.

[skriss]

Closing this out as this should be resolved in the upcoming v1.2 release - feel free to reach out again as needed!

[IlyaNakhaichuk]

I have a similar problem. Please tell me how can I fix this? I try with version 1.2, it still fails authorization. Maybe there are some other ways to put a backup in s3 using a velero?

[ashish-amarnath]

@IlyaNakhaichuk Can you please share how you are installing Velero? and also confirm that the credentials file you are using is in the expected format? You can find more instructions here https://github.com/vmware-tanzu/velero-plugin-for-aws#option-1-set-permissions-with-an-iam-user