Storage S3 - AWS node role IMDSv1 does not work after upgrade to v2.2

[coufalja]

Describe the bug
Tempo does not start with "transparent" AWS credentials (node role). It seems to not take the credentials/role into account at all.

To Reproduce
Steps to reproduce the behavior:

1. Start from tempo-distributed chart
2. Use node role credentials to connect to S3
3. Fail with "err":"failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on <bucket name>: Access Denied"

Expected behavior
Tempo connects and starts.

Environment:

• Kubernetes
• Helm chart tempo-distributed >=1.5.3

Additional Context

• Just downgrading image version while using the very same config fixes the problem, so must be code or image related issue.
• I also though about missing some niche permission in the attached policy so I temporarily granted * to the node role but no joy.

[joe-elliott]

I'm not seeing much changes to the s3 backend.

Prefix Support: a55da81

Standardized TLS config: e809ae607a

Updated minio: e9898effad

The first two seem innocuous. WDYT? The minio changes are here:

minio/minio-go@v7.0.23...v7.0.52

If it's possible to test image versions immediately before some of these changes it could help us narrow down the issue. We could also try just upgrading minio again and see if it fixes it.

[coufalja]

So it is the minio upgrade. Image grafana/tempo:main-9996433-amd64 is working as expected, the docker.io/grafana/tempo:main-e9898ef-amd64 fails.

[coufalja]

Tried bumping to github.com/minio/minio-go/v7 v7.0.61 and building from the main branch but ended up with the same error.

[joe-elliott]

Nothing really stands out in the minio changelog. Can you file an issue here:

https://github.com/minio/minio-go

Detailing your auth setup and see if they have insight into what may have changed?

[coufalja]

I have found the culprit: Use 1s timeout for fetching imdsv2 token introduced in v7.0.24. Reverting that and building custom tempo image with custom minio-go fixes the issue.

[coufalja]

Seems to me as issue in Tempo that surfaced in due to 2 bugfixes made in minio lib: minio/minio-go#1626 and minio/minio-go#1682 I fixed the bug title as it affects the IMDSv1 only. It seems that the same issue appeared and was fixed some time ago in Cortex cortexproject/cortex#4897

[coufalja]

I backported the Cortex fix in #2760

[joe-elliott]

Excellent research and fix, @coufalja. Thank you.