boto3 Region: Frankfurt - botocore.exceptions.ClientError

[kresdjan]

I'm using boto3 / django-storages together with Heroku, but when I deploy the app I get this AWS4-HMAC-SHA256 error:

           raise error_class(parsed_response, operation_name)

       botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the PutObject operation: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

requirements.txt

boto==2.45.0
boto3==1.4.4
botocore==1.5.7
dj-database-url==0.4.1
Django==1.10.4
django-storages==1.5.2
docutils==0.13.1
gunicorn==19.6.0
jmespath==0.9.1
olefile==0.44
Pillow==4.0.0
psycopg2==2.6.2
python-dateutil==2.6.0
s3transfer==0.1.10
six==1.10.0
whitenoise==3.2

How do I fix this?

Full build log:

-----> Python app detected

     $ pip install -r requirements.txt

     $ python manage.py collectstatic --noinput

       Traceback (most recent call last):

         File "manage.py", line 10, in <module>

           execute_from_command_line(sys.argv)

         File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line

           utility.execute()

         File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 359, in execute

           self.fetch_command(subcommand).run_from_argv(self.argv)

         File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/base.py", line 294, in run_from_argv

           self.execute(*args, **cmd_options)

         File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/base.py", line 345, in execute

           output = self.handle(*args, **options)

         File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle

           collected = self.collect()

         File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect

           handler(path, prefixed_path, storage)

         File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 347, in copy_file

           self.storage.save(prefixed_path, source_file)

         File "/app/.heroku/python/lib/python3.6/site-packages/django/core/files/storage.py", line 54, in save

           return self._save(name, content)

         File "/app/.heroku/python/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 452, in _save

           self._save_content(obj, content, parameters=parameters)

         File "/app/.heroku/python/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 467, in _save_content

           obj.upload_fileobj(content, ExtraArgs=put_parameters)

         File "/app/.heroku/python/lib/python3.6/site-packages/boto3/s3/inject.py", line 509, in object_upload_fileobj

           ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)

         File "/app/.heroku/python/lib/python3.6/site-packages/boto3/s3/inject.py", line 427, in upload_fileobj

           return future.result()

         File "/app/.heroku/python/lib/python3.6/site-packages/s3transfer/futures.py", line 73, in result

           return self._coordinator.result()

         File "/app/.heroku/python/lib/python3.6/site-packages/s3transfer/futures.py", line 233, in result

           raise self._exception

         File "/app/.heroku/python/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__

           return self._execute_main(kwargs)

         File "/app/.heroku/python/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main

           return_value = self._main(**kwargs)

         File "/app/.heroku/python/lib/python3.6/site-packages/s3transfer/upload.py", line 679, in _main

           client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)

         File "/app/.heroku/python/lib/python3.6/site-packages/botocore/client.py", line 253, in _api_call

           return self._make_api_call(operation_name, kwargs)

         File "/app/.heroku/python/lib/python3.6/site-packages/botocore/client.py", line 543, in _make_api_call

           raise error_class(parsed_response, operation_name)

       botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the PutObject operation: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

 !     Error while running '$ python manage.py collectstatic --noinput'.

       See traceback above for details.

       You may need to update application code to resolve this error.

       Or, you can disable collectstatic for this application:

          $ heroku config:set DISABLE_COLLECTSTATIC=1

       https://devcenter.heroku.com/articles/django-assets

 !     Push rejected, failed to compile Python app.

 !     Push failed

[kresdjan]

If I change the bucket to US Standard instead of EU Frankfurt, I get no errors. The files are transferred to the bucket correctly.

Seems like a Europe only issue. Does anyone know how to solve this?

[stealthycoin]

Frankfurt only supports sigv4 region support list

For some reason that environment you have is defaulting to sigv2 (which US Standard understands)

You can force it to use sigv4 by setting a config variable:
aws configure set s3.signature_version s3v4 --profile {profile_name}

more info

[kresdjan]

I see, don't know why that is. I haven't changed a single value. I just installed boto3==1.4.4 with pip.
If I understand the documentation correctly, it should happen automatically.

s3
Set S3 specific configuration data. You typically will not need to set these values. Boto3 will automatically switching signature versions and addressing styles if necessary. This is a nested configuration value. See the Nested Configuration section for more information on the format. The sub config keys supported for s3 are:

• addressing_style: Specifies which addressing style to use. This controls if the bucket name is in the hostname or part of the URL. Value values are: path, virtual, and auto.
• signature_version: Which AWS signature version to use when signing requests. Value values are: s3 and s3v4.
  https://boto3.readthedocs.io/en/latest/guide/configuration.html#configuration

But I will and fix it.
Thanks

[shoad]

Guys, i'm having the same error using exactly this sample code from the docs. All libs are up to date.

My python code is:

AWS_ACCESS_KEY_ID = '...'
AWS_SECRET_ACCESS_KEY = '...'
AWS_STORAGE_BUCKET_NAME = 'my_bucket'
AWS_AUTO_CREATE_BUCKET = False
AWS_S3_REGION_NAME = 'eu-central-1'
AWS_QUERYSTRING_AUTH = True
AWS_S3_FILE_OVERWRITE = True
AWS_S3_HOST = 's3.eu-central-1.amazonaws.com'
AWS_S3_CUSTOM_DOMAIN = '{bucket}.s3.amazonaws.com'.format(bucket=AWS_STORAGE_BUCKET_NAME)
AWS_LOCATION = ''
AWS_S3_SIGNATURE_VERSION = 's3v4'
S3_USE_SIGV4 = True
os.environ['S3_USE_SIGV4'] = 'True'

#### TEST
from botocore.signers import CloudFrontSigner
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import padding

# First you create a cloudfront signer based on a normalized RSA signer::
def rsa_signer(message):
    with open('config/pk-cloudfront.pem', 'rb') as key_file:
        private_key = serialization.load_pem_private_key(
            key_file.read(),
            password=None,
            backend=default_backend()
        )
    signer = private_key.signer(padding.PKCS1v15(), hashes.SHA1())
    signer.update(message)
    return signer.finalize()

key_id = '...'
cf_signer = CloudFrontSigner(key_id, rsa_signer)

# To sign with a canned policy::
import datetime
signed_url = cf_signer.generate_presigned_url(
    '...', date_less_than=datetime.datetime(2020, 12, 1))

print(signed_url)

My ~/.aws/config is (after i ran aws configure set s3.signature_version s3v4 ):

[default]
output = json
region = eu-central-1
s3 =
    signature_version = s3v4

This post says that the issue can't be solved on the code side. Is there a solution?