s3: provide a way to use key id and key secret directly

[Suor]

For now the only way to pass credentials to S3 is using credentials file and the env var. This is very inconvenient when trying to use dvc as a library. For now I hacked it like:

def pull(repo):
    # In memory config update to avoid validation
    repo.config["core"]["remote"] = "vdefault"
    repo.config["remote"]["vdefault"] = {"url": ..., "key_id": ..., "key_secret": ...}

    fix_s3()
    repo.pull()


from funcy import once, monkey, cached_property, wrap_prop  # noqa

@once
def fix_s3():
    import threading
    from dvc.remote.s3 import S3RemoteTree

    @monkey(S3RemoteTree, "s3")
    @wrap_prop(threading.Lock())
    @cached_property
    def s3(self):
        import boto3

        session = boto3.session.Session(
            profile_name=self.profile,
            region_name=self.region,
            # Pass these two from config to session
            aws_access_key_id=self.config.get("key_id"),
            aws_secret_access_key=self.config.get("key_secret"),
        )

        return session.client("s3", endpoint_url=self.endpoint_url, use_ssl=self.use_ssl)

Would be nice to have it inside dvc.remote.S3RemoteTree.

[efiop]

@Suor Have you considered changing the env for that call?

Also, I suppose your request is only about internal api, right?

[Suor]

Changing env for the process will leak it to other threads and also may leak to subsequent calls. Starting a new process is a solution, but a heavy one.

I am using repo.cloud.pull(), but repo.pull() should work the same.

[Suor]

Ideally I want key_id/key_secret pair be added to config. Any reason you want to avoid it?

[efiop]

Ideally I want key_id/key_secret pair be added to config. Any reason you want to avoid it?

Same reason as with any plain-text passwords 🙂 We could make it api-only, but feels like a hack, still. I guess the only current nice way to do it is to just use a tempfile, dump keys into and and then use it as credentailpath in the config.

[Suor]

That would work too. However, if we want api to be usable we should stop doing things like that) Non-saveable config option is fine with me too. Hack or not.

[Suor]

And BTW, we already have it for oss, gdrive and ssh. Maybe it's ok to save password/key to local config. People are storing this anyway in some ~/.aws/credentials, which is a blessed way.

[efiop]

@Suor agreed. Let's do that.