Enforcing v2 of AWS IDMS causes Access Denied error

[mateuszwieloch]

Thanos, Prometheus and Golang version used:

name: "thanosio/thanos",
version: "0.13.0",
tag: "v0.13.0",
checksum: "sha256:743e5a7df61d67a16fd3e27fae4cae2af335cfe36f8b210240b2c14e9839b2a7"

name: "prom/prometheus",
version: "2.19.2",
tag: "v2.19.2",
checksum: "sha256:cd134bd4fca0f60ff8b4c679cebe5c5c5cf5e2da5f4886b2ae933da821915f92"

Note: I've also tried with the latest stable version of Thanos (0.14) but the same error appeared.

Object Storage Provider:

What happened:
Following AWS security recommendations I have enforced usage of v2 IDMS on EC2 instances of Thanos Compactor. This has caused the following error to appear in logs:

{
    "caller": "runutil.go:98",
    "err": "BaseFetcher: iter bucket: Access Denied",
    "level": "error",
    "msg": "function failed. Retrying in next tick",
    "ts": "2020-08-28T20:05:24.151455425Z"
}

which seems to be coming from fetcher.go file, which in turn seems to be using Minio library. (I wonder if upgrading Minio could help at all).

Also note that when I remove the policy, error reliably disappears, so it shouldn't be related with permissions to access a bucket.

What you expected to happen:
No errors.

How to reproduce it (as minimally and precisely as possible):
Enforce v2 of IDMS on Compactor EC2 instance, which includes two steps:

• set metadata_options.http_tokens to "required"
• add an IAM instance policy to Compactor Amazon EC2 instance to enforce usage of v2 of IDMS

{
  "Sid": "RequireAllEc2RolesToUseV2",
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "NumericLessThan": {
      "ec2:RoleDelivery": "2.0"
    }
  }
}

Full logs to relevant components: N/A
Anything else we need to know: N/A

[bwplotka]

Looks like we need to investigate / find out how to enable v2 IDMS with minio library. Help wanted! (:

[wjam]

I've looked at the minIO documentation, and it seems like it doesn't have support for IDMSv2.

Is there a reason for using minIO rather than the AWS SDK?

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[markmsmith]

stale bot, we still need this.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[markmsmith]

Still needed.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[markmsmith]

Still needed.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[markmsmith]

👍 Still needed

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[markmsmith]

I think this is still to be resolved. I'm not sure why the stale bot ignored my comment.
Any ideas @bwplotka?

[saputkin]

Is it still to be resolved?
Enforcing IMDSv2 can be a great addition, as it's the recommended way by AWS

[markmsmith]

I believe this is still needed. I don't know why the issue was closed.

[felipewnp]

@bwplotka Can you un-close this?

It's definitely still needed..

[wjam]

Looks like Minio now supports this IDMS v2 (minio/minio-go#1489) and so probably just needs a Thanos-side code change?