S3 IAM access via AWS Kubernetes IRSA does not work with 2.2.2

[galen-reltio]

Describe the bug
I deploy tempo in AWS eks kubernetes using the tempo-distributed helm chart. I use s3 storage. For kubenetes pod perissions I use IRSA to authorize the kubernetes service account to access S3. Chart version 1.6.3 (which upgrades from 2.2.1 to 2.2.2) results in tempo failing to start with the message:

evel=error ts=2023-09-06T16:19:17.002371309Z caller=main.go:111 msg="error running Tempo" err="failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on grafana-tempo-qa-XXXX-us-east-1: Access Denied"

Rolling back to helm chart release 1.6.2 returns tempo to normal functionality.

To Reproduce

1. Start from tempo 2.2.1 deployed via helm chart 1.6.2 in AWS EKS kubernetes with the tempo service account authorized to access S3 stroge via IRSA.
2. Upgrade to tempo 2.2.2 deployed via helm chart 1.6.3; workloads now fail to start due to permissions issues.

Expected behavior
Tempo should start and access s3 storage. No permissions issues should be encountered.

Environment:

• Infrastructure: AWS EKS Kubernetes with IRSA
• Deployment tool: helm

Additional Context
I highly suspect this is caused by PR 2760 which fixed issue 2743

[joe-elliott]

Thanks for filing an issue. We are tracking it here: #2888

We are looking for confirmation that the PR fixes AWS auth after which we will merge and cut a 2.2.3. If you are able to test the images linked on the branch please do 🙏

I'm going to close this issue b/c it seems like a dupe of the above, but please reopen if I misunderstand your issue!