Botocore does not make S3 Signed URLs with SigV4

[mikegrima]

Hello!

We noticed that botocore by default will generate S3 signed urls with HmacV1QueryAuth and not with SigV4. This is reproducible with the latest and greatest boto3 and botocore versions with the following code:

import boto3

s3_client = boto3.client("s3", region_name="us-east-1")

bucket = "somebucket"
key = "some/prefix/to/an/object"

print(s3_client.generate_presigned_url("get_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=3600))

When digging deep into botocore, we found that in the code: https://github.com/boto/botocore/blob/develop/botocore/signers.py#L183-L195

        handler, response = self._event_emitter.emit_until_response(
            'choose-signer.{0}.{1}'.format(
                self._service_id.hyphenize(), operation_name),
            signing_name=self._signing_name, region_name=self._region_name,
            signature_version=signature_version, context=context)

        if response is not None:
            signature_version = response
            # The suffix needs to be checked again in case we get an improper
            # signature version from choose-signer.
            if signature_version is not botocore.UNSIGNED and not \
                    signature_version.endswith(suffix):
                signature_version += suffix

the signature_version initially starts as s3v4-query, (which does properly map to S3SigV4QueryAuth

botocore/botocore/auth.py

Line 856 in 30206ab

's3v4-query': S3SigV4QueryAuth,

). The response that is returned is not None and the response object has the result set to s3-signer. This replaces the correct v4 version with one that maps to the very old HmacV1QueryAuth signer.

When digging even deeper, we noticed that one of the handlers has a function that is called _default_s3_presign_to_sigv2, and as the name suggests, sets the pre-sign signature to v2 (or at least not v4): https://github.com/boto/botocore/blob/develop/botocore/client.py#L260-L276

Many large AWS customers migrated away from using SigV2 for S3 calls and users that need to make use of S3 signed URLs are getting V2 signature URLs by default with boto3. In our case, we actually added deny statements to our S3 buckets to prevent anything other than SigV4 calls from being made to S3. This results in Python boto3 users getting non-functional S3 signatures (our Java users don't experience this issue).

Other than us instructing our Python S3 Signed URL users to do their own signature logic: #1784 (comment) we would greatly appreciate if this would just work properly in botocore.

Thank you

[swetashre]

Thank you for your post. Botocore still uses Sigv2 for generating presigned url unless it has explicitly configured to use Sigv4 because it is a backward incompatible change to switch them from v2 to v4. For example if a user generates a presigned url using no region or incorrect region then that request can fail since region matters for Sigv4.

For all the normal HTTP requests, Sigv4 is used by default because here if the user uses the incorrect region then in this case SDK can detect the incorrect region and resend the requests using the correct region.

Hope it helps and please let us know if you have any concerns.

[mikegrima]

I think if there is a note in the documentation about the default signature version -- and an easy way to switch it to V4 -- this would alleviate my concerns.

My colleague made a workaround in one of our other projects where we can specify a custom botocore Config object: Netflix-Skunkworks/cloudaux#116 which unblocked our issue.

[swetashre]

@mikegrima - I agree we should update our documentation. We are tracking this documentation issue under this boto3 issue boto/boto3#2417. Let's track this issue under the linked github issue.
This is a way to change default signature version:

from botocore.config import Config

my_config = Config(
    signature_version = 'v4',
)

client = boto3.client('s3', config=my_config)

Here you can find some more information about how to switch it to v4.
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

I hope it helps and please let us know if you have any concerns.

[asyschikov]

@swetashre may I suggest reconsidering the decision to "still use Sigv2 for generating presigned url unless it has explicitly configured to use Sigv4"? At some point deprecation has to become permanent backwards incompatible and the time for Sigv2 has passed. The process of deprecation started in 2012-2013 and was supposed to finish in 2019 (got a bit delayed). https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/ At this point there is more harm in defaulting to Sigv2, because it is considered a security risk and certain tools that monitor AWS CloudTrail logs flag this as a problem.

[atishbits]

Minor update on code snippet from @swetashre.
Setting signature_version = 'v4' did not work. For some reason when it comes to S3 clients, boto3 expects to set this to signature_version = 's3v4'. Refer: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html?highlight=s3v4

Updated code snippet per this:

from botocore.config import Config

my_config = Config(
    signature_version = 's3v4',
)

client = boto3.client('s3', config=my_config)