[s3] Add support for signing CloudFront URLs

[terencehonles]

Fixes #456. Similar to #484 but uses AWS CloudFront signed URLS (See AWS docs)

This is probably the right way to approach the problem in #484, but if that solution works that might be an alternative (not sure how @mattjegan configured S3+CloundFront)

[jschneier]

Why even deal with multiple possible keys via AWS_CLOUDFRONT_KEYS?

[terencehonles]

The warning or having multiple keys? I put it that way so you can have the key definition in one place (your settings.py), but have multiple storages which reference different buckets or different accounts (you may be allowed to sign CloudFront URLs for a different project if they have delegated permissions).

Either way there needs to be a ("key_id", "key") tuple and I figured rather than store it as a list or two different settings it was more useful to just store it in a dict

[terencehonles]

Friendly reminder

[terencehonles]

Ping

[terencehonles]

@jschneier anymore thoughts about this PR?

[alexandernst]

Any news about this?

[dangelsaurus]

I'd love to see this soon!

[ericovis]

I was about to implement that myself, any news on this PR?

[thefedoration]

+1

[terencehonles]

For those that need this now you can install via an editable install in your requirements.txt w/ the following -e git+git://github.com/<username>/django-storages.git@support-signing-cloudfront-urls#egg=django-storages, you can use terencehonles as the username, but that will break when/if this PR gets merged and I delete the branch. You can clone my branch and reference your own repo, but the downside of an editable install is the package will pinned to this version unless you or I integrate any new changes from this repo.

[studybuffalo]

One other option that builds off the solution by @terencehonles: You can extended the S3Boto3Storage class and override the url method with the code in the pull request. You can then point the Django settings DEFAULT_FILE_STORAGE attribute to the custom class. This should work around the issues of pinning to the forked version.

[colin-byrne-1]

Does this repo have an active maintainer? Seems like there is 100% consensus to get this merged.

[jschneier]

It depends on how you define active I suppose. It comes in bursts for me anyway. I do intend to look more at this one, sure.

…

On Wednesday, November 6, 2019, cobyrne09 ***@***.***> wrote: Does this repo have an active maintainer? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#587?email_source=notifications&email_token=AAREDWH6WMIRFSLOER67SC3QSNLTZA5CNFSM4FTNW7OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDINRZY#issuecomment-550557927>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAREDWEG7DNOOKQP3U2J4GTQSNLTZANCNFSM4FTNW7OA> .

[jschneier]

Hi, this looks okay. I'd rather we just do a single AWS_CLOUDFRONT_KEY and people can subclass as they wish. Can you update to that & rebase at which point we can merge?

[terencehonles]

@jschneier yeah I'll update this PR, I was doing my normal updating forks, but w/o looking at this PR when updating last.

[DataGreed]

@terencehonles So, how do I make cloudfront signing work? What do I have to specify in settings and how do I get the url?

BTW, is there a way to set expiration time?

[DataGreed]

@jschneier could you please release this version to PIP so it can be used in projects as dependency?

[DataGreed]

@terencehonles what exactly should be supplied for AWS_CLOUDFRONT_KEY? Should it be a path to some kind of pem file or some secret key? Either way I get from_buffer() cannot return the address of a unicode object

File "/project/.env/lib/python3.7/site-packages/storages/backends/s3boto3.py" in get_default_settings
  369.             cloudfront_signer = self.get_cloudfront_signer(cloudfront_key_id, cloudfront_key)

File "/project/.env/lib/python3.7/site-packages/storages/backends/s3boto3.py" in get_cloudfront_signer
  357.         return _cloud_front_signer_from_pem(key_id, key)

File "/project/.env/lib/python3.7/site-packages/storages/backends/s3boto3.py" in _cloud_front_signer_from_pem
  54.             pem, password=None, backend=default_backend())

File "/project/.env/lib/python3.7/site-packages/cryptography/hazmat/primitives/serialization/base.py" in load_pem_private_key
  16.     return backend.load_pem_private_key(data, password)

File "/project/.env/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py" in load_pem_private_key
  1089.             password,

File "/project/.env/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py" in _load_key
  1282.         mem_bio = self._bytes_to_bio(data)

File "/project/.env/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py" in _bytes_to_bio
  473.         data_ptr = self._ffi.from_buffer(data)

Exception Type: TypeError at /objects/3/
Exception Value: from_buffer() cannot return the address of a unicode object

[DataGreed]

Okay, I guess something like this works:

with open(os.path.join(BASE_DIR, "keyfile.pem")) as aws_cert:
    AWS_CLOUDFRONT_KEY = aws_cert.read().encode('ascii')

[terencehonles]

@DataGreed glad you figured it out, I guess I probably should have seen if there was documentation I could have added to. Did you want to open a PR to potentially add your suggestion?

with open(os.path.join(BASE_DIR, "keyfile.pem")) as aws_cert:
    AWS_CLOUDFRONT_KEY = aws_cert.read().encode('ascii')

My keyfile is inline (actually an ENV VAR) instead of read via a file like your example, but I think what you wrote would get the point across. One comment is you can open your file in binary mode and you will get bytes and not have to .encode('ascii')

So:

with open(os.path.join(BASE_DIR, 'keyfile.pem'), 'rb') as cert:
    AWS_CLOUDFRONT_KEY = cert.read()

[DataGreed]

@terencehonles oh thanks! That makes more sense. Sure, I can try extend the docs if you won't be able to do it.

@terencehonles btw, is there any way to change the expiration time?

[Faisal-Manzer]

When will the patch version be published with the official package?

[terencehonles]

@terencehonles oh thanks! That makes more sense. Sure, I can try extend the docs if you won't be able to do it.

I might be able to get to it, but I'm encouraging you to also contribute to the project just like I did 😉

@terencehonles btw, is there any way to change the expiration time?

@DataGreed the expiration will be defined by the value you pass to the storage url function, if that isn't specified it defaults to AWS_QUERYSTRING_EXPIRE.

[SkiFamily]

Hi,

Has anybody got this to work ? I will extend the docs to hopefully save other peoples time once I manage to get the CloudFront signing working but at the moment I can't seem to get it going.

My Process:

1. Removed django-storages
2. Installed django-storages from source (installed directly from the master branch with pip)
3. modified my settings.py to include:

    AWS_CLOUDFRONT_KEY = os.environ.get('AWS_CLOUDFRONT_KEY', None)
    AWS_CLOUDFRONT_KEY_ID = os.environ.get('AWS_CLOUDFRONT_KEY_ID', None)

4. Generated a CloudFront Key Pair as specified in the docs
5. Updated my ENV vars with the corresponding values:

AWS_CLOUDFRONT_KEY=-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
AWS_CLOUDFRONT_KEY_ID=APK....

The Key file is inline as specified by @terencehonles but I still seem to get the buffer error @DataGreed mentioned

I'll keep playing around with it, and update this comment as I go.

EDIT 1

I have added

AWS_CLOUDFRONT_KEY = os.environ.get('AWS_CLOUDFRONT_KEY', None).encode('ascii')

Which fixes the buffer error but raises a new one:

  File "/app/.scalingo/python/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1324, in _load_key
    self._handle_key_loading_error()
  File "/app/.scalingo/python/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1382, in _handle_key_loading_error
    raise ValueError("Could not deserialize key data.")

Exception Type: ValueError at /
Exception Value: Could not deserialize key data.

EDIT 2:

The value of the key in my ENV VAR was wrong because the key contained an equal sign and my ENV parser considered it as another ENV VAR.
I have found a workaround to define the appropriate value un my ENV vars and everything seems to be working fine now.

I have made a PR to update the docs #900

[DataGreed]

@terencehonles could you please tell me if there is any way to make the specific URL never expire? Should I just pass a ridiculously high timestamp or None?

[terencehonles]

@DataGreed None will not work, but it really depends on your use case.

I pointed to the AWS docs in the PR description, and you can see this section Private Content Overview: Choosing Duration. A long duration may work for you, but if not, you will not be able to use a "Canned Policy" and you can see the differences here. This PR only implements a "Canned Policy" and you would have to alter the storage in order to provide a custom policy. You may even want to go as far as trying to validate that it is a valid policy (or is at least valid JSON)

[DataGreed]

@terencehonles than you so much for giving a detailed answer.

I think the longer expiration date will work for me - some of the uploaded files are public (like images) and should be used in open graph tags which facebook caches for some unspecified amount of time and some private file links should be valid for just a day.

In my case I was trying to make public links valid for longer period of time so opengraph images won't suddenly break. I guess if it will allow me to set 100 years, I am all set :D

I did not find any limitation description in Amazon docs, but one of the articles you posted above says

You can also distribute private content using a signed URL that is valid for a longer time, possibly for years.

I'll post a follow up telling if it was a success or not.

[jnovkovic]

@SkiFamily Hey man, how did you manage to go above equal (=) sign in env? I'm getting the same error as you, and not sure how to proceed.

Thanks.

[DataGreed]

I'll post a follow up telling if it was a success or not.

Forgot to follow up, my bad. It was a success – you can set 100 years as expiration time.
I've ended up just setting different expiration timeouts for private content and for public content.

[kumar-student]

Hi,

Has anybody got this to work ? I will extend the docs to hopefully save other peoples time once I manage to get the CloudFront signing working but at the moment I can't seem to get it going.

My Process:

1. Removed django-storages
2. Installed django-storages from source (installed directly from the master branch with pip)
3. modified my settings.py to include:

    AWS_CLOUDFRONT_KEY = os.environ.get('AWS_CLOUDFRONT_KEY', None)
    AWS_CLOUDFRONT_KEY_ID = os.environ.get('AWS_CLOUDFRONT_KEY_ID', None)

4. Generated a CloudFront Key Pair as specified in the docs
5. Updated my ENV vars with the corresponding values:

AWS_CLOUDFRONT_KEY=-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
AWS_CLOUDFRONT_KEY_ID=APK....

The Key file is inline as specified by @terencehonles but I still seem to get the buffer error @DataGreed mentioned

I'll keep playing around with it, and update this comment as I go.

EDIT 1

I have added

AWS_CLOUDFRONT_KEY = os.environ.get('AWS_CLOUDFRONT_KEY', None).encode('ascii')

Which fixes the buffer error but raises a new one:

  File "/app/.scalingo/python/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1324, in _load_key
    self._handle_key_loading_error()
  File "/app/.scalingo/python/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1382, in _handle_key_loading_error
    raise ValueError("Could not deserialize key data.")

Exception Type: ValueError at /
Exception Value: Could not deserialize key data.

EDIT 2:

The value of the key in my ENV VAR was wrong because the key contained an equal sign and my ENV parser considered it as another ENV VAR. I have found a workaround to define the appropriate value un my ENV vars and everything seems to be working fine now.

I have made a PR to update the docs #900

Hi @SkiFamily Could you please elaborate on the process of resolving ValueError and Could not deserialize key data? I have followed AWS and Django-storages documentation and configured CloudFront but couldn't succeed
AWS_CLOUDFRONT_KEY = os.environ.get('AWS_CLOUDFRONT_KEY', None).encode('ascii')
AWS_CLOUDFRONT_KEY_ID = os.environ.get('AWS_CLOUDFRONT_KEY_ID', None)
or
AWS_CLOUDFRONT_KEY = os.environ.get('AWS_CLOUDFRONT_KEY', None)
AWS_CLOUDFRONT_KEY_ID = os.environ.get('AWS_CLOUDFRONT_KEY_ID', None)