s3 remote: DVC asks MFA code for each file on push/pull

[shcheklein]

Context:

https://discordapp.com/channels/485586884165107732/485596304961962003/644313703038189568

Might be related:

boto/boto3#1179

[shcheklein]

Temporary workaround:

https://discordapp.com/channels/485586884165107732/485596304961962003/644334026311598124

[TimothyHyndman]

To get the above workaround working, I had to remove the mfa_serial = ... line from the profile I was using in ~/.aws/config

[efiop]

Might be solved by caching the session #2473 (comment) , but need to confirm.

[Suor]

Looked it up. This probably won't be solved by threadsafe caching s3 prop. I am surprised MFA work with dvc at all though.

Need to ask the user how MFA device is set.

[efiop]

Another user is experiencing this same issue. https://discordapp.com/channels/485586884165107732/485596304961962003/656143281163599885

[efiop]

We could solve it the same way we do it with passphrase-s for ssh, by using ask_password config option, but maybe we could auto-detect this somehow. Need to research this.

[efiop]

From https://docs.aws.amazon.com/cli/latest/topic/config-vars.html

If you specify an mfa_serial, then the first time an AssumeRole call is made, you will be prompted to enter the MFA code. Subsequent commands will use the cached temporary credentials. However, when the temporary credentials expire, you will be re-prompted for another MFA code.

need to take a closer look.

[efiop]

Might be related https://github.com/mixja/boto3-session-cache

[efiop]

Workaround for now: https://discordapp.com/channels/485586884165107732/485596304961962003/656166503971815445

[Suor]

It feels to me that we are not caching some boto session properly, thus preventing it from caching an access token.

[Suor]

@efiop your related lib says it is not needed anymore:

You no longer need this package as of botocore version 1.8.14, which now includes the JSON file cache structure traditionally used by the AWS CLI

[drisspg]

Hey, my organization recently switched to MFA style access control for our s3 buckets. While trying to run dvc push i get the prompt for MFA code 33 times and once I enter the code it another prompt is appended after the TQDM progress bar.

I was wondering if the workaround is still the only solution to make this work?
@efiop Let me know that the workaround is currently the only known solution however switching to s3fs is in progress and may help with this issue

[efiop]

@drisspg Thanks for the suggestion! We've been migrating to fsspec this year and we've fully switched to https://github.com/dask/s3fs as a backend a few months back. @drisspg if you can, it would be great if you could confirm that this works for you now natively. 🙏

[drisspg]

I am unable to get this to work.

I set the env var: export AWS_PROFILE=****
Then call dvc push since this will run a s3 read over the bucket.

It prompts for my MFA code multiple times and says that it is not valid:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: MultiFactorAuthentication failed, unable to validate MFA code.  Please verify your MFA serial number is valid and associated with this user.
Refreshing temporary credentials failed during mandatory refresh period.

I get prompted multiple times to enter mfa code. When I use the source script the MFA code is working correctly so the mfa token shouldn't be wrong.

[efiop]

@drisspg Thanks for trying it out! Reopening. Looks like we need to take a closer look.

IIRC, currently you need to generate session_token yourself and then use it along with access_key_id and secret_access_key in dvc config.

How do you normally use it with things like awscli?

[drisspg]

I have my credentials set in the ~/.aws/credentials file and use different profiles aws --profile Prd ***

Is there further documentation somewhere about getting dvc to work with s3 and mfa I can try?

[efiop]

@drisspg Not really, usually those are all the options you need to configure. We are not able to reproduce so far, looks like it might be something with your environment, not sure 🙁 Closing for now. Happy to take another look if you could produce a reproducible example.

[pmrowla]

The setup described in the "Using temporary credentials with named profiles" section here should allow you to use profiles + MFA https://aws.amazon.com/premiumsupport/knowledge-center/authenticate-mfa-cli/

Essentially you need to use aws sts get-session-token (which prompts you for your MFA token) to populate the values for the appropriate profile in .aws/credentials, and then you just use that profile as normal in DVC