c7n-org MFA issues #1956
Comments
|
wow, thats interesting, thanks for the bug report. there's a --debug flag on org to go single threaded through multiple accounts (it also drops into a debugger on error). it sounds like your just running this from an ec2 instance so instance role/profile as initial credential source? does it go haywire from the get go, or after some time? |
|
It goes haywire from the get go. It is running on an ec2 instance, but per the boto3 documentation, it should look at the credential file in .aws first, and that's what it does when only run one at a time. I'm going to try the --debug flag and see what it kicks out. |
|
are you putting static credentials in ~/.aws or are you doing a login with mfa.. perhaps it worth coming up a level, how do you do automate requiring sts w/ mfa automation on a cron execution?. typical is to just setup instance profile/arn for credential source sans the mfa. |
|
We're not yet able to automate sts w/ mfa. So far cloud custodian runs with temporary credentials from whoever is currently developing policies. We use a python wrapper around the aws-adfs package because it supports MFA from the CLI. This talks to an ADFS server which authenticates our credentials against a CAS server. From the exchange between ADFS and AWS, unique access keys (per account) are granted along with a single session token (the adfs_auth token). Our python writes these unique keys to the .aws credentials and config folder as shown below: .aws/config [prd-account-1] [prd-account-2] .aws/credentials [prd-account-1] [prd-account-2] [ . . . ] [all] ` |
|
It seems that c7n-org only gets credentials out of the [default] section of the credentials and config files. Does this mean that c7n-org will only work if you have cross-account roles set up? |
|
yes, its not setup to lookup a profile for each account, the creds found for the cli execution (sdk defaults, --profile, --assume cli options) are expected to be able to sts role assume to the other roles. it seems reasonable as an enhancement to allow for the profile option to be specified on each account. |
|
My current workaround: renaming the profile for each account to default one at a time. While it is tedious, it is much faster than needing to accept push notifications to my phone every minute. My end goal would be to have something like this automated so it can run nightly on a cron job. I'm testing a possible workaround by setting up an instance profile that has access to assume cross-account roles, but this still hasn't worked. |
|
for tools/c7n_guardian (uses c7n-org under the hood) i ended up adding in support for profile as part of the account config, should be easy to up-port it to c7n-org |
|
ported over to c7n-org that should allow for the mfa case if you active credentials against those profiles. basically in c7n-org config instead of role specify profile key for each account. |
|
Hi kaplit i need help on the same issue, i want to use profiles name in the c7n-org command will it possible |
When run against one single account at a time, c7n-org does great:
accounts.yml:
accounts:
name: XXXXXXXXXXXX
regions:
role: arn:aws:iam::XXXXXXXXXX:role/AccountAdministrator
tags:
c7n-org has trouble dealing with multiple accounts that require MFA:
ClientError: An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,200: custodian.output:ERROR Error while executing policy Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/c7n/policy.py", line 305, in run resources = self.policy.resource_manager.resources() File "/usr/local/lib/python2.7/site-packages/c7n/query.py", line 397, in resources resources = self.augment(self.source.resources(query)) File "/usr/local/lib/python2.7/site-packages/c7n/query.py", line 210, in resources return self.query.filter(self.manager, **query) File "/usr/local/lib/python2.7/site-packages/c7n/query.py", line 74, in filter client = local_session(self.session_factory).client( File "/usr/local/lib/python2.7/site-packages/c7n/utils.py", line 250, in local_session s = factory() File "/usr/local/lib/python2.7/site-packages/c7n/credentials.py", line 40, in __call__ region or self.region, self.external_id) File "/usr/local/lib/python2.7/site-packages/c7n/credentials.py", line 87, in assumed_session metadata=refresh(), File "/usr/local/lib/python2.7/site-packages/c7n/credentials.py", line 78, in refresh session.client('sts').assume_role, **parameters)['Credentials'] File "/usr/local/lib/python2.7/site-packages/c7n/utils.py", line 347, in _retry return func(*args, **kw) File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call raise error_class(parsed_response, operation_name) ClientError: An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,202: c7n_org:ERROR Exception running policy:elb-tag-compliance account:XXXXXXXXXXXXX region:us-east-1 error:An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,204: c7n_org:ERROR Exception running policy:elb-tag-compliance account:XXXXXXXXXXXXXXX region:us-west-2 error:An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,205: c7n_org:ERROR Exception running policy:elb-tag-compliance account:XXXXXXXXXXXXXXXX region:us-west-2 error:An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired 2018-01-19 16:59:46,248: c7n_org:INFO Policy resource counts Counter()When run against all 17 accounts, it eats up all the available memory on my server, causes a kernel panic, and kicks me out of my ssh session.
The text was updated successfully, but these errors were encountered: