Vault tries to revoke AWS IAM users that don't exist

[zBart]

Describe the bug
When we were looking at our CloudTrail logs, we noticed a large amount of errors being generated by the vault user. It seems that vault tries to revoke an IAM user that doesn't exist every minute, here is the log from CloudTrail:

{
    "eventVersion": "1.02",
    "userIdentity": {
      "type": "IAMUser",
      "principalId": "XXXXXXXXXXXXXXXXXXXXX",
      "arn": "arn:aws:iam::our-aws-account-id:user/vault",
      "accountId": "our-aws-account-id",
      "accessKeyId": "XXXXXXXXXXXXXXXXXXXXX",
      "userName": "vault"
    },
    "eventTime": "2018-08-27T10:53:25Z",
    "eventSource": "iam.amazonaws.com",
    "eventName": "ListGroupsForUser",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "xxx.xxx.xxx.xxx",
    "userAgent": "aws-sdk-go/1.12.74 (go1.10.3; linux; amd64)",
    "errorCode": "NoSuchEntityException",
    "errorMessage": "The user with name vault-userpass-aVaultUsername-role-aVaultRoleName-1794 cannot be found.",
    "requestParameters": "{\"userName\": \"vault-userpass-aVaultUsername-role-aVaultRoleName-1794\", \"maxItems\": 1000}",
    "responseElements": "null",
    "requestID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "eventID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "eventType": "AwsApiCall",
    "recipientAccountId": "our-aws-account-id"
  }

We think this happens when Vault fails to create the IAM user in the first place (although I suppose it can also happen if someone removes the IAM user manually). Because we only use STS tokens, we never gave Vault the rights to create IAM users. However when using the interface, clicking on a role immediately creates an IAM user. Vault then tries to revoke this, but fails.

To Reproduce
Steps to reproduce the behavior:

1. Setup Vault with AWS, without giving it IAM permissions to create actual users
2. Create a role for AWS within vault
3. Try to generate a account through the web interface (might also work through CLI)
   You'll get this error:
   Error creating IAM user: AccessDenied: User: arn:aws:iam::our-aws-account-id:user/vault is not authorized to perform: iam:CreateUser on resource: arn:aws:iam::[...]
4. Afterwards, check the CloudTrail logs and see that Vault tries to revoke the user every minute

Expected behavior
Vault acknowledges that the account does not exist (for example by checking the response the IAM API gives) and stops trying to remove the account.

Or alternatively have a way that you can stop Vault from trying to revoke the accounts.

Environment:

• Vault Server Version (retrieve with vault status):

Key             Value
---             -----
Seal Type       shamir
Sealed          false
Total Shares    5
Threshold       3
Version         0.10.3
Cluster Name    vault-cluster-63d461f0
Cluster ID      xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
HA Enabled      true
HA Cluster      https://our-domain
HA Mode         active

• Vault CLI Version (retrieve with vault version):

Vault v0.10.3 ('c69ae68faf2bf7fc1d78e3ec62655696a07454c7')

• Server Operating System/Architecture:
  n/a

Vault server configuration file(s):
n/a

[chrishoffman]

Vault's job is to manage the lease and to keep on trying to revoke it. If the credential was not fully created or manually deleted, you can force revoke the lease to tell vault to "forget" the credential.

See https://www.vaultproject.io/api/system/leases.html#revoke-force.

[zBart]

While I find it odd that Vault sees the account as created when the API call performing the create returns an error, I understand the philosophy.

I did the revoke as per the documentation:

$ curl -v --header "X-Vault-Token: xxxxxx-xxxx-xxxx-xxxxxxxxxxxx" --request PUT https://our-vault-domain/v1/sys/leases/revoke-force/aws/creds
> PUT /v1/sys/leases/revoke-force/aws/creds HTTP/1.1
> Host: our-vault-domain
> User-Agent: curl/7.47.0
> Accept: */*
> X-Vault-Token: xxxxxx-xxxx-xxxx-xxxxxxxxxxxx
>
< HTTP/1.1 204 No Content
< Date: Mon, 27 Aug 2018 13:41:33 GMT
< Content-Type: application/json
< Connection: keep-alive
< Cache-Control: no-store

But this did not seem to have fixed the issue. We still see Vault trying to remove the users.

Am I revoking the wrong namespace?

[zBart]

Hey @chrishoffman I was checking the source code trying to figure out why Vault is still trying to revoke the user and noticed that the GCP secrets engine does seem to check the API response when revoking access keys:

https://github.com/hashicorp/vault-plugin-secrets-gcp/blob/master/plugin/secrets_service_account_key.go#L163

And the access token revoking also has some logic attached to it:
https://github.com/hashicorp/vault-plugin-secrets-gcp/blob/master/plugin/secrets_access_token.go#L104

Since GCP does this, maybe it would be a good addition for the AWS engine as well?

[joelthompson]

@chrishoffman I think the issue is that Vault writes a WAL entry before creating the user, and then the user creation fails as does the rollback. I think we should, upon user creation failure, try to delete the WAL entry, and further, detect the NoSuchEntity error and report that as a successful rollback, since a failure to mark a user as revoked would cause Vault to attempt to continue revoking it indefinitely. Should be a relatively simple fix.

[joelthompson]

I've proposed what I believe will fix this in #5202

[Huper7777]

[email protected]

[Huper7777]

build #70700006