Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retry on transient failures during AWS IAM auth login attempts #8727

Merged
merged 2 commits into from
Apr 13, 2020
Merged

Retry on transient failures during AWS IAM auth login attempts #8727

merged 2 commits into from
Apr 13, 2020

Conversation

tyrannosaurus-becks
Copy link
Contributor

Fixes #8721

This PR causes Vault to retry if AWS returns an occasional 500-family or connection error. The approach taken here using the retryable package is the same as was recently taken when writing a lightweight Kubernetes Go client for the service registration feature.

I do view this as a bug because our documentation here would lead folks to believe that this call would already be retried. However, I do not think we should put it out as a patch, and rather should include it in the next beta, just to err on the side of safety because even small AWS changes can have a big impact.

Acceptance test output 
tbex@pop-os:~/go/src/github.com/hashicorp/vault/builtin/credential/aws$ VAULT_ACC=1 go test -v ./...
=== RUN   TestBackend_E2E_Initialize
2020-04-10T16:22:57.838-0700 [DEBUG] storage.cache: creating LRU cache: size=0
2020-04-10T16:22:57.838-0700 [ERROR] core: no seal config found, can't determine if legacy or new-style shamir
2020-04-10T16:22:57.838-0700 [ERROR] core: no seal config found, can't determine if legacy or new-style shamir
2020-04-10T16:22:57.838-0700 [ERROR] core: no seal config found, can't determine if legacy or new-style shamir
2020-04-10T16:22:57.838-0700 [INFO]  core: security barrier not initialized
2020-04-10T16:22:57.839-0700 [INFO]  core: security barrier initialized: stored=1 shares=3 threshold=3
2020-04-10T16:22:57.839-0700 [DEBUG] core: cluster name not found/set, generating new
2020-04-10T16:22:57.839-0700 [DEBUG] core: cluster name set: name=vault-cluster-6f643874
2020-04-10T16:22:57.839-0700 [DEBUG] core: cluster ID not found, generating new
2020-04-10T16:22:57.839-0700 [DEBUG] core: cluster ID set: id=b6128f0a-f11e-62ca-0682-2a44c7f93c11
2020-04-10T16:22:57.839-0700 [DEBUG] core: generating cluster private key
2020-04-10T16:22:57.849-0700 [DEBUG] core: generating local cluster certificate
2020-04-10T16:22:57.858-0700 [INFO]  core: post-unseal setup starting
2020-04-10T16:22:57.858-0700 [DEBUG] core: clearing forwarding clients
2020-04-10T16:22:57.858-0700 [DEBUG] core: done clearing forwarding clients
2020-04-10T16:22:57.858-0700 [DEBUG] core: persisting feature flags
2020-04-10T16:22:57.866-0700 [INFO]  core: loaded wrapping token key
2020-04-10T16:22:57.867-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-04-10T16:22:57.867-0700 [INFO]  core: no mounts; adding default mount table
2020-04-10T16:22:57.867-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-04-10T16:22:57.867-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-04-10T16:22:57.867-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-04-10T16:22:57.869-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-04-10T16:22:57.869-0700 [INFO]  core: restoring leases
2020-04-10T16:22:57.869-0700 [INFO]  rollback: starting rollback manager
2020-04-10T16:22:57.869-0700 [DEBUG] expiration: collecting leases
2020-04-10T16:22:57.869-0700 [DEBUG] expiration: leases collected: num_existing=0
2020-04-10T16:22:57.870-0700 [INFO]  expiration: lease restore complete
2020-04-10T16:22:57.870-0700 [DEBUG] identity: loading entities
2020-04-10T16:22:57.870-0700 [DEBUG] identity: entities collected: num_existing=0
2020-04-10T16:22:57.870-0700 [INFO]  identity: entities restored
2020-04-10T16:22:57.870-0700 [DEBUG] identity: identity loading groups
2020-04-10T16:22:57.870-0700 [DEBUG] identity: groups collected: num_existing=0
2020-04-10T16:22:57.870-0700 [INFO]  identity: groups restored
2020-04-10T16:22:57.870-0700 [INFO]  core: post-unseal setup complete
2020-04-10T16:22:57.870-0700 [INFO]  core: root token generated
2020-04-10T16:22:57.870-0700 [INFO]  core: pre-seal teardown starting
2020-04-10T16:22:57.870-0700 [DEBUG] expiration: stop triggered
2020-04-10T16:22:57.870-0700 [DEBUG] expiration: finished stopping
2020-04-10T16:22:57.870-0700 [INFO]  rollback: stopping rollback manager
2020-04-10T16:22:57.870-0700 [INFO]  core: pre-seal teardown complete
2020-04-10T16:22:57.870-0700 [DEBUG] core: unseal key supplied
2020-04-10T16:22:57.871-0700 [DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=ad452547-87a0-0539-26d4-3eb3a308c384
2020-04-10T16:22:57.871-0700 [DEBUG] core: unseal key supplied
2020-04-10T16:22:57.871-0700 [DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=ad452547-87a0-0539-26d4-3eb3a308c384
2020-04-10T16:22:57.871-0700 [DEBUG] core: unseal key supplied
2020-04-10T16:22:57.871-0700 [DEBUG] core: starting cluster listeners
2020-04-10T16:22:57.871-0700 [INFO]  core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:0
2020-04-10T16:22:57.871-0700 [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:46531
2020-04-10T16:22:57.871-0700 [INFO]  core: vault is unsealed
2020-04-10T16:22:57.871-0700 [INFO]  core: entering standby mode
2020-04-10T16:22:57.871-0700 [INFO]  core: acquired lock, enabling active operation
2020-04-10T16:22:57.871-0700 [DEBUG] core: generating cluster private key
2020-04-10T16:22:57.879-0700 [DEBUG] core: generating local cluster certificate
2020-04-10T16:22:57.887-0700 [INFO]  core: post-unseal setup starting
2020-04-10T16:22:57.887-0700 [DEBUG] core: clearing forwarding clients
2020-04-10T16:22:57.887-0700 [DEBUG] core: done clearing forwarding clients
2020-04-10T16:22:57.887-0700 [DEBUG] core: persisting feature flags
2020-04-10T16:22:57.887-0700 [INFO]  core: loaded wrapping token key
2020-04-10T16:22:57.887-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-04-10T16:22:57.888-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-04-10T16:22:57.888-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-04-10T16:22:57.888-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-04-10T16:22:57.889-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-04-10T16:22:57.889-0700 [INFO]  core: restoring leases
2020-04-10T16:22:57.889-0700 [DEBUG] identity: loading entities
2020-04-10T16:22:57.889-0700 [INFO]  rollback: starting rollback manager
2020-04-10T16:22:57.889-0700 [DEBUG] identity: entities collected: num_existing=0
2020-04-10T16:22:57.889-0700 [DEBUG] expiration: collecting leases
2020-04-10T16:22:57.889-0700 [DEBUG] expiration: leases collected: num_existing=0
2020-04-10T16:22:57.889-0700 [INFO]  identity: entities restored
2020-04-10T16:22:57.889-0700 [DEBUG] identity: identity loading groups
2020-04-10T16:22:57.889-0700 [DEBUG] identity: groups collected: num_existing=0
2020-04-10T16:22:57.889-0700 [INFO]  identity: groups restored
2020-04-10T16:22:57.889-0700 [DEBUG] core: request forwarding setup function
2020-04-10T16:22:57.889-0700 [DEBUG] core: clearing forwarding clients
2020-04-10T16:22:57.889-0700 [DEBUG] core: done clearing forwarding clients
2020-04-10T16:22:57.889-0700 [INFO]  expiration: lease restore complete
2020-04-10T16:22:57.889-0700 [DEBUG] core: leaving request forwarding setup function
2020-04-10T16:22:57.889-0700 [INFO]  core: post-unseal setup complete
2020-04-10T16:22:57.890-0700 [INFO]  core: successful mount: namespace= path=secret/ type=kv
2020-04-10T16:22:57.891-0700 [INFO]  core: enabled audit backend: path=noop/ type=noop
2020-04-10T16:22:57.893-0700 [DEBUG] auth.aws.auth_aws_8232e80e.initialize: starting initialization
2020-04-10T16:22:57.893-0700 [INFO]  core: enabled credential backend: path=aws/ type=aws
2020-04-10T16:22:57.894-0700 [INFO]  auth.aws.auth_aws_8232e80e.initialize: an upgrade was performed during initialization
2020-04-10T16:22:58.903-0700 [INFO]  core: marked as sealed
2020-04-10T16:22:58.903-0700 [DEBUG] core: clearing forwarding clients
2020-04-10T16:22:58.903-0700 [DEBUG] core: done clearing forwarding clients
2020-04-10T16:22:58.903-0700 [DEBUG] core: finished triggering standbyStopCh for runStandby
2020-04-10T16:22:58.903-0700 [DEBUG] core: shutting down periodic key rotation checker
2020-04-10T16:22:58.903-0700 [DEBUG] core: shutting down periodic leader refresh
2020-04-10T16:22:58.903-0700 [DEBUG] core: shutting down leader elections
2020-04-10T16:22:58.903-0700 [INFO]  core: pre-seal teardown starting
2020-04-10T16:22:59.403-0700 [DEBUG] expiration: stop triggered
2020-04-10T16:22:59.403-0700 [DEBUG] expiration: finished stopping
2020-04-10T16:22:59.403-0700 [INFO]  rollback: stopping rollback manager
2020-04-10T16:22:59.403-0700 [INFO]  core: pre-seal teardown complete
2020-04-10T16:22:59.403-0700 [DEBUG] core: runStandby done
2020-04-10T16:22:59.403-0700 [INFO]  core: stopping cluster listeners
2020-04-10T16:22:59.403-0700 [INFO]  core.cluster-listener: forwarding rpc listeners stopped
2020-04-10T16:22:59.871-0700 [INFO]  core.cluster-listener: rpc listeners successfully shut down
2020-04-10T16:22:59.871-0700 [INFO]  core: cluster listeners successfully shut down
2020-04-10T16:22:59.871-0700 [DEBUG] core: sealing barrier
2020-04-10T16:22:59.871-0700 [INFO]  core: vault is sealed
2020-04-10T16:22:59.871-0700 [DEBUG] core: unseal key supplied
2020-04-10T16:22:59.872-0700 [DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=3059b22f-4ef8-1f70-e35d-2067ebc9c32e
2020-04-10T16:22:59.872-0700 [DEBUG] core: unseal key supplied
2020-04-10T16:22:59.872-0700 [DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=3059b22f-4ef8-1f70-e35d-2067ebc9c32e
2020-04-10T16:22:59.872-0700 [DEBUG] core: unseal key supplied
2020-04-10T16:22:59.872-0700 [DEBUG] core: starting cluster listeners
2020-04-10T16:22:59.872-0700 [INFO]  core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:46531
2020-04-10T16:22:59.872-0700 [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:46531
2020-04-10T16:22:59.872-0700 [INFO]  core: vault is unsealed
2020-04-10T16:22:59.872-0700 [INFO]  core: entering standby mode
2020-04-10T16:22:59.872-0700 [INFO]  core: acquired lock, enabling active operation
2020-04-10T16:22:59.872-0700 [DEBUG] core: generating cluster private key
2020-04-10T16:22:59.883-0700 [DEBUG] core: generating local cluster certificate
2020-04-10T16:22:59.892-0700 [INFO]  core: post-unseal setup starting
2020-04-10T16:22:59.892-0700 [DEBUG] core: clearing forwarding clients
2020-04-10T16:22:59.892-0700 [DEBUG] core: done clearing forwarding clients
2020-04-10T16:22:59.892-0700 [DEBUG] core: persisting feature flags
2020-04-10T16:22:59.892-0700 [INFO]  core: loaded wrapping token key
2020-04-10T16:22:59.892-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-04-10T16:22:59.892-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-04-10T16:22:59.892-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-04-10T16:22:59.892-0700 [INFO]  core: successfully mounted backend: type=kv path=secret/
2020-04-10T16:22:59.892-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-04-10T16:22:59.893-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-04-10T16:22:59.893-0700 [INFO]  core: successfully enabled credential backend: type=aws path=aws/
2020-04-10T16:22:59.893-0700 [INFO]  core: restoring leases
2020-04-10T16:22:59.893-0700 [INFO]  rollback: starting rollback manager
2020-04-10T16:22:59.893-0700 [DEBUG] identity: loading entities
2020-04-10T16:22:59.893-0700 [DEBUG] identity: entities collected: num_existing=0
2020-04-10T16:22:59.893-0700 [DEBUG] expiration: collecting leases
2020-04-10T16:22:59.893-0700 [DEBUG] expiration: leases collected: num_existing=0
2020-04-10T16:22:59.893-0700 [INFO]  identity: entities restored
2020-04-10T16:22:59.893-0700 [DEBUG] identity: identity loading groups
2020-04-10T16:22:59.893-0700 [DEBUG] identity: groups collected: num_existing=0
2020-04-10T16:22:59.893-0700 [INFO]  identity: groups restored
2020-04-10T16:22:59.893-0700 [DEBUG] core: request forwarding setup function
2020-04-10T16:22:59.893-0700 [DEBUG] core: clearing forwarding clients
2020-04-10T16:22:59.893-0700 [DEBUG] core: done clearing forwarding clients
2020-04-10T16:22:59.893-0700 [DEBUG] core: leaving request forwarding setup function
2020-04-10T16:22:59.893-0700 [INFO]  expiration: lease restore complete
2020-04-10T16:22:59.893-0700 [DEBUG] auth.aws.auth_aws_8232e80e.initialize: starting initialization
2020-04-10T16:22:59.893-0700 [INFO]  core: post-unseal setup complete
2020-04-10T16:22:59.893-0700 [INFO]  auth.aws.auth_aws_8232e80e.initialize: an upgrade was performed during initialization
2020-04-10T16:23:02.894-0700 [INFO]  TestBackend_E2E_Initialize: cleaning up vault cluster
--- PASS: TestBackend_E2E_Initialize (7.04s)
=== RUN   TestBackend_CreateParseVerifyRoleTag
--- PASS: TestBackend_CreateParseVerifyRoleTag (0.00s)
=== RUN   TestBackend_prepareRoleTagPlaintextValue
--- PASS: TestBackend_prepareRoleTagPlaintextValue (0.00s)
=== RUN   TestBackend_CreateRoleTagNonce
--- PASS: TestBackend_CreateRoleTagNonce (0.00s)
=== RUN   TestBackend_ConfigTidyIdentities
--- PASS: TestBackend_ConfigTidyIdentities (0.00s)
=== RUN   TestBackend_ConfigTidyRoleTags
--- PASS: TestBackend_ConfigTidyRoleTags (0.00s)
=== RUN   TestBackend_TidyIdentities
--- PASS: TestBackend_TidyIdentities (1.00s)
=== RUN   TestBackend_TidyRoleTags
--- PASS: TestBackend_TidyRoleTags (1.00s)
=== RUN   TestBackend_ConfigClient
2020-04-10T16:23:06.882-0700 [DEBUG] storage.cache: creating LRU cache: size=0
2020-04-10T16:23:06.882-0700 [ERROR] core: no seal config found, can't determine if legacy or new-style shamir
2020-04-10T16:23:06.882-0700 [INFO]  core: security barrier not initialized
2020-04-10T16:23:06.882-0700 [INFO]  core: security barrier initialized: stored=1 shares=1 threshold=1
2020-04-10T16:23:06.882-0700 [DEBUG] core: cluster name not found/set, generating new
2020-04-10T16:23:06.882-0700 [DEBUG] core: cluster name set: name=vault-cluster-8804b3a0
2020-04-10T16:23:06.882-0700 [DEBUG] core: cluster ID not found, generating new
2020-04-10T16:23:06.882-0700 [DEBUG] core: cluster ID set: id=cac24a2b-2f0b-9f01-853b-a57807e09948
2020-04-10T16:23:06.882-0700 [INFO]  core: post-unseal setup starting
2020-04-10T16:23:06.883-0700 [DEBUG] core: clearing forwarding clients
2020-04-10T16:23:06.883-0700 [DEBUG] core: done clearing forwarding clients
2020-04-10T16:23:06.883-0700 [DEBUG] core: persisting feature flags
2020-04-10T16:23:06.905-0700 [INFO]  core: loaded wrapping token key
2020-04-10T16:23:06.905-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-04-10T16:23:06.905-0700 [INFO]  core: no mounts; adding default mount table
2020-04-10T16:23:06.907-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-04-10T16:23:06.907-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-04-10T16:23:06.908-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-04-10T16:23:06.910-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-04-10T16:23:06.910-0700 [INFO]  core: restoring leases
2020-04-10T16:23:06.910-0700 [INFO]  rollback: starting rollback manager
2020-04-10T16:23:06.910-0700 [DEBUG] expiration: collecting leases
2020-04-10T16:23:06.911-0700 [DEBUG] expiration: leases collected: num_existing=0
2020-04-10T16:23:06.911-0700 [INFO]  expiration: lease restore complete
2020-04-10T16:23:06.912-0700 [DEBUG] identity: loading entities
2020-04-10T16:23:06.912-0700 [DEBUG] identity: entities collected: num_existing=0
2020-04-10T16:23:06.912-0700 [INFO]  identity: entities restored
2020-04-10T16:23:06.912-0700 [DEBUG] identity: identity loading groups
2020-04-10T16:23:06.912-0700 [DEBUG] identity: groups collected: num_existing=0
2020-04-10T16:23:06.912-0700 [INFO]  identity: groups restored
2020-04-10T16:23:06.912-0700 [INFO]  core: post-unseal setup complete
2020-04-10T16:23:06.912-0700 [INFO]  core: root token generated
2020-04-10T16:23:06.912-0700 [INFO]  core: pre-seal teardown starting
2020-04-10T16:23:06.912-0700 [DEBUG] expiration: stop triggered
2020-04-10T16:23:06.912-0700 [DEBUG] expiration: finished stopping
2020-04-10T16:23:06.912-0700 [INFO]  rollback: stopping rollback manager
2020-04-10T16:23:06.912-0700 [INFO]  core: pre-seal teardown complete
2020-04-10T16:23:06.912-0700 [DEBUG] core: unseal key supplied
2020-04-10T16:23:06.913-0700 [INFO]  core: clustering disabled, not starting listeners
2020-04-10T16:23:06.913-0700 [INFO]  core: post-unseal setup starting
2020-04-10T16:23:06.913-0700 [DEBUG] core: clearing forwarding clients
2020-04-10T16:23:06.913-0700 [DEBUG] core: done clearing forwarding clients
2020-04-10T16:23:06.913-0700 [DEBUG] core: persisting feature flags
2020-04-10T16:23:06.913-0700 [INFO]  core: loaded wrapping token key
2020-04-10T16:23:06.913-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-04-10T16:23:06.913-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-04-10T16:23:06.913-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-04-10T16:23:06.913-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-04-10T16:23:06.914-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-04-10T16:23:06.914-0700 [INFO]  core: restoring leases
2020-04-10T16:23:06.914-0700 [INFO]  rollback: starting rollback manager
2020-04-10T16:23:06.914-0700 [DEBUG] identity: loading entities
2020-04-10T16:23:06.914-0700 [DEBUG] identity: entities collected: num_existing=0
2020-04-10T16:23:06.914-0700 [DEBUG] expiration: collecting leases
2020-04-10T16:23:06.914-0700 [DEBUG] expiration: leases collected: num_existing=0
2020-04-10T16:23:06.914-0700 [INFO]  identity: entities restored
2020-04-10T16:23:06.914-0700 [DEBUG] identity: identity loading groups
2020-04-10T16:23:06.914-0700 [INFO]  expiration: lease restore complete
2020-04-10T16:23:06.914-0700 [DEBUG] identity: groups collected: num_existing=0
2020-04-10T16:23:06.914-0700 [INFO]  identity: groups restored
2020-04-10T16:23:06.914-0700 [INFO]  core: post-unseal setup complete
2020-04-10T16:23:06.914-0700 [INFO]  core: vault is unsealed
2020-04-10T16:23:06.916-0700 [DEBUG] initialize: starting initialization
2020-04-10T16:23:06.916-0700 [INFO]  core: enabled credential backend: path=mnt/ type=test
2020-04-10T16:23:06.916-0700 [INFO]  initialize: an upgrade was performed during initialization
2020-04-10T16:23:06.917-0700 [WARN]  Executing test step: step_number=1
2020-04-10T16:23:06.917-0700 [WARN]  Executing test step: step_number=2
2020-04-10T16:23:06.917-0700 [WARN]  Executing test step: step_number=3
2020-04-10T16:23:06.917-0700 [WARN]  Executing test step: step_number=4
2020-04-10T16:23:06.917-0700 [WARN]  Requesting RollbackOperation
--- PASS: TestBackend_ConfigClient (0.04s)
=== RUN   TestBackend_pathConfigCertificate
--- PASS: TestBackend_pathConfigCertificate (0.00s)
=== RUN   TestBackend_parseAndVerifyRoleTagValue
--- PASS: TestBackend_parseAndVerifyRoleTagValue (0.00s)
=== RUN   TestBackend_PathRoleTag
--- PASS: TestBackend_PathRoleTag (0.00s)
=== RUN   TestBackend_PathBlacklistRoleTag
--- PASS: TestBackend_PathBlacklistRoleTag (0.00s)
=== RUN   TestBackendAcc_LoginWithInstanceIdentityDocAndWhitelistIdentity
--- SKIP: TestBackendAcc_LoginWithInstanceIdentityDocAndWhitelistIdentity (0.00s)
    backend_test.go:1043: env var TEST_AWS_EC2_PKCS7 not set, skipping test
=== RUN   TestBackend_pathStsConfig
--- PASS: TestBackend_pathStsConfig (0.00s)
=== RUN   TestBackendAcc_LoginWithCallerIdentity
--- SKIP: TestBackendAcc_LoginWithCallerIdentity (0.00s)
    backend_test.go:1477: env var TEST_AWS_ACCESS_KEY_ID not set, skipping test
=== RUN   TestGeneratePartitionToRegionMap
--- PASS: TestGeneratePartitionToRegionMap (0.00s)
=== RUN   TestBackend_pathConfigClient
--- PASS: TestBackend_pathConfigClient (0.00s)
=== RUN   TestBackend_pathConfigIdentity
--- PASS: TestBackend_pathConfigIdentity (0.00s)
=== RUN   TestBackend_pathLogin_getCallerIdentityResponse
--- PASS: TestBackend_pathLogin_getCallerIdentityResponse (0.00s)
=== RUN   TestBackend_pathLogin_parseIamArn
--- PASS: TestBackend_pathLogin_parseIamArn (0.00s)
=== RUN   TestBackend_validateVaultHeaderValue
--- PASS: TestBackend_validateVaultHeaderValue (0.00s)
=== RUN   TestBackend_pathLogin_IAMHeaders
=== RUN   TestBackend_pathLogin_IAMHeaders/Default
=== RUN   TestBackend_pathLogin_IAMHeaders/Map-complete
=== RUN   TestBackend_pathLogin_IAMHeaders/Map-incomplete
=== RUN   TestBackend_pathLogin_IAMHeaders/JSON-complete
=== RUN   TestBackend_pathLogin_IAMHeaders/JSON-incomplete
=== RUN   TestBackend_pathLogin_IAMHeaders/Base64-complete
=== RUN   TestBackend_pathLogin_IAMHeaders/Base64-incomplete-missing-header
=== RUN   TestBackend_pathLogin_IAMHeaders/Base64-incomplete-missing-auth-sig
--- PASS: TestBackend_pathLogin_IAMHeaders (0.01s)
    --- PASS: TestBackend_pathLogin_IAMHeaders/Default (0.00s)
    --- PASS: TestBackend_pathLogin_IAMHeaders/Map-complete (0.00s)
    --- PASS: TestBackend_pathLogin_IAMHeaders/Map-incomplete (0.00s)
    --- PASS: TestBackend_pathLogin_IAMHeaders/JSON-complete (0.00s)
    --- PASS: TestBackend_pathLogin_IAMHeaders/JSON-incomplete (0.00s)
    --- PASS: TestBackend_pathLogin_IAMHeaders/Base64-complete (0.00s)
    --- PASS: TestBackend_pathLogin_IAMHeaders/Base64-incomplete-missing-header (0.00s)
    --- PASS: TestBackend_pathLogin_IAMHeaders/Base64-incomplete-missing-auth-sig (0.00s)
=== RUN   TestBackend_pathRoleEc2
--- PASS: TestBackend_pathRoleEc2 (0.00s)
=== RUN   Test_enableIamIDResolution
--- PASS: Test_enableIamIDResolution (0.00s)
=== RUN   TestBackend_pathIam
--- PASS: TestBackend_pathIam (0.00s)
=== RUN   TestBackend_pathRoleMixedTypes
2020-04-10T16:23:06.930-0700 [DEBUG] no stsRole found for 123456789012
2020-04-10T16:23:06.930-0700 [DEBUG] no cached client for region us-east-1 and stsRole 
--- PASS: TestBackend_pathRoleMixedTypes (0.94s)
=== RUN   TestAwsEc2_RoleCrud
--- PASS: TestAwsEc2_RoleCrud (0.00s)
=== RUN   TestAwsEc2_RoleDurationSeconds
--- PASS: TestAwsEc2_RoleDurationSeconds (0.00s)
=== RUN   TestRoleEntryUpgradeV
--- PASS: TestRoleEntryUpgradeV (0.00s)
=== RUN   TestRoleInitialize
--- PASS: TestRoleInitialize (0.00s)
=== RUN   TestAwsVersion
--- PASS: TestAwsVersion (0.00s)
=== RUN   TestRoleResolutionWithSTSEndpointConfigured
--- SKIP: TestRoleResolutionWithSTSEndpointConfigured (0.00s)
    path_role_test.go:1008: skipping because AWS_ASSUMABLE_ROLE_ARN is unset
PASS
ok  	github.com/hashicorp/vault/builtin/credential/aws	10.056s
?   	github.com/hashicorp/vault/builtin/credential/aws/cmd/aws	[no test files]
Output from manual testing IRL with ephemeral Vault test instance 
tbex@pop-os:~/go/src/github.com/hashicorp/vault$ vault auth enable aws
Success! Enabled aws auth method at: aws/
tbex@pop-os:~/go/src/github.com/hashicorp/vault$ vault write auth/aws/role/dev-role-iam auth_type=iam bound_iam_principal_arn=<redacted> policies=prod,dev max_ttl=500h
Success! Data written to: auth/aws/role/dev-role-iam
tbex@pop-os:~/go/src/github.com/hashicorp/vault$ vault login -method=aws role=dev-role-iam
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

WARNING! The following warnings were returned from Vault:

  * TTL of "768h" exceeded the effective max_ttl of "500h"; TTL value is
  capped accordingly

Key                                Value
---                                -----
token                              s.0MF93tE6a8C3DI8xeI8Gfjnn
token_accessor                     hM64JXIMW5aRo59IjwqKfCYN
token_duration                     500h
token_renewable                    true
token_policies                     ["default" "dev" "prod"]
identity_policies                  []
policies                           ["default" "dev" "prod"]
token_meta_account_id              047646243813
token_meta_canonical_arn           arn:aws:iam::047646243813:user/tbex
token_meta_client_arn              arn:aws:iam::047646243813:user/tbex
token_meta_client_user_id          AIDAIFHUGH4BQQNND2ZBI
token_meta_inferred_aws_region     n/a
token_meta_role_id                 1a9c066d-898a-4961-71cc-c1fad25734cd
token_meta_auth_type               iam
token_meta_inferred_entity_id      n/a
token_meta_inferred_entity_type    n/a

@tyrannosaurus-becks tyrannosaurus-becks added auth/aws bug Used to indicate a potential bug labels Apr 10, 2020
@tyrannosaurus-becks tyrannosaurus-becks added this to the 1.5 milestone Apr 10, 2020
@tyrannosaurus-becks tyrannosaurus-becks requested a review from a team April 10, 2020 23:30
@tyrannosaurus-becks tyrannosaurus-becks merged commit 78ed5f3 into master Apr 13, 2020
@tyrannosaurus-becks tyrannosaurus-becks deleted the retry-failed-aws-login-attempts branch April 13, 2020 21:28
mikegrass pushed a commit to mikegrass/vault that referenced this pull request Jul 9, 2020
@tyrannosaurus-becks @mikegrass
Backport 1.4.4: Retry on transient failures during AWS IAM auth login…
fbc27e3 
… attempts (hashicorp#8727)

* use retryer for failed aws auth attempts

* fixes from testing
Valarissa pushed a commit that referenced this pull request Jul 29, 2020
@tyrannosaurus-becks
Retry on transient failures during AWS IAM auth login attempts (#8727)
2bfb303 
* use retryer for failed aws auth attempts

* fixes from testing
@Valarissa Valarissa mentioned this pull request Jul 29, 2020
Closed
Valarissa pushed a commit that referenced this pull request Jul 29, 2020
@tyrannosaurus-becks
Retry on transient failures during AWS IAM auth login attempts (#8727) (
4238560 
#9630)

* use retryer for failed aws auth attempts

* fixes from testing

Co-authored-by: Becca Petrin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@calvn calvn calvn approved these changes

Assignees
No one assigned
Labels
auth/aws bug Used to indicate a potential bug
Projects
None yet
Milestone
1.5
Development

Successfully merging this pull request may close these issues.

AWS IAM auth method doesn't retry sts:GetCallerIdentity request if it fails
2 participants
@tyrannosaurus-becks @calvn