Adding handling of media files to the documentation

[kasperj93]

Would it be possible to add documentation to the handling of media files?

It would have been really helpful in my case:

From @elnygren 's post on stackoverflow: https://stackoverflow.com/questions/34247702/configure-django-and-google-cloud-storage
and answer to #491
I struggled a lot with handling media files between a django website on an appengine and static files in google cloud storage (both static markup files and media files). After a long try, I finally managed to succeed with the following settings.

Credit to @elnygren!
In my settings.py I include the following:

DEFAULT_FILE_STORAGE = 'config.storage_backends.GoogleCloudMediaStorage'
STATICFILES_STORAGE = 'config.storage_backends.GoogleCloudStaticStorage'
GS_PROJECT_ID = '<project-id>'
GS_MEDIA_BUCKET_NAME = '<media-bucket-name>'
GS_STATIC_BUCKET_NAME = '<static-bucket-name>'
STATIC_URL = 'https://storage.googleapis.com/{}/'.format(GS_STATIC_BUCKET_NAME)
MEDIA_URL = 'https://storage.googleapis.com/{}/'.format(GS_MEDIA_BUCKET_NAME)
GS_DEFAULT_ACL = 'private'  # makes the files to private

Credit to @elnygren!
Creating a folder in the root directoy named 'config' I inserted the following 'storage_backends.py' file:

"""
  GoogleCloudStorage extensions suitable for handing Django's
  Static and Media files.

  Requires following settings:
  MEDIA_URL, GS_MEDIA_BUCKET_NAME
  STATIC_URL, GS_STATIC_BUCKET_NAME

  In addition to
  https://django-storages.readthedocs.io/en/latest/backends/gcloud.html
  """
  from django.conf import settings
  from storages.backends.gcloud import GoogleCloudStorage
  from storages.utils import setting
  from urllib.parse import urljoin


  class GoogleCloudMediaStorage(GoogleCloudStorage):
      """GoogleCloudStorage suitable for Django's Media files."""

      def __init__(self, *args, **kwargs):
          if not settings.MEDIA_URL:
              raise Exception('MEDIA_URL has not been configured')
          kwargs['bucket_name'] = setting('GS_MEDIA_BUCKET_NAME', strict=True)
          super(GoogleCloudMediaStorage, self).__init__(*args, **kwargs)

      def url(self, name):
          """.url that doesn't call Google."""
          return urljoin(settings.MEDIA_URL, name)


  class GoogleCloudStaticStorage(GoogleCloudStorage):
      """GoogleCloudStorage suitable for Django's Static files"""

      def __init__(self, *args, **kwargs):
          if not settings.STATIC_URL:
              raise Exception('STATIC_URL has not been configured')

App engine already instantiate GOOGLE_APPLICATION_CREDENTIALS so for development, I added:
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path/to/service_account.json"
in my "dev settings"

[ajs-88]

@kasperj93 I'm getting an error,

google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{\n "error": "invalid_grant",\n "error_description": "Invalid JWT Signature."\n}')

[kasperj93]

@aruljane . Can you provide some more information?

[sww314]

@aruljane - There is probably something wrong with your service account file.
Try something like this:

from google.oauth2 import service_account

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    "path/to/credentials.json"
)

[ajs-88]

@kasperj93 @sww314

I have followed your steps in settings.py & config/storage_backends.py
Note: GS_CREDENTIALS also specified.
in config/storage_backends.py
kwargs['bucket_name'] = setting('GS_MEDIA_BUCKET_NAME',)# strict=True)

and i tried,

./manage.py collectstatic

to test,

and i got the following,

./manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings.

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
Traceback (most recent call last):
File "./manage.py", line 15, in
execute_from_command_line(sys.argv)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/django/core/management/base.py", line 316, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/django/core/management/base.py", line 353, in execute
output = self.handle(*args, **options)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 188, in handle
collected = self.collect()
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
handler(path, prefixed_path, storage)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 343, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 249, in delete_file
if self.storage.exists(prefixed_path):
File "/Users/Arul/PycharmProjects/Gazelle/venv/src/django-storages/storages/backends/gcloud.py", line 196, in exists
return bool(self.bucket.get_blob(self._encode_name(name)))
File "/Users/Arul/PycharmProjects/Gazelle/venv/src/django-storages/storages/backends/gcloud.py", line 127, in bucket
self._bucket = self._get_or_create_bucket(self.bucket_name)
File "/Users/Arul/PycharmProjects/Gazelle/venv/src/django-storages/storages/backends/gcloud.py", line 135, in _get_or_create_bucket
return self.client.get_bucket(name)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/cloud/storage/client.py", line 227, in get_bucket
bucket.reload(client=self)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/cloud/storage/_helpers.py", line 106, in reload
method="GET", path=self.path, query_params=query_params, _target_object=self
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 315, in api_request
target_object=_target_object,
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 192, in _make_request
return self._do_request(method, url, headers, data, target_object)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 221, in _do_request
return self.http.request(url=url, method=method, headers=headers, data=data)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/auth/transport/requests.py", line 198, in request
self._auth_request, method, url, request_headers)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/auth/credentials.py", line 122, in before_request
self.refresh(request)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/oauth2/service_account.py", line 322, in refresh
request, self._token_uri, assertion)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/oauth2/_client.py", line 145, in jwt_grant
response_data = _token_endpoint_request(request, token_uri, body)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/oauth2/_client.py", line 111, in _token_endpoint_request
_handle_error_response(response_body)
File "/Users/Arul/PycharmProjects/Gazelle/venv/lib/python3.7/site-packages/google/oauth2/_client.py", line 61, in _handle_error_response
error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', '{\n "error": "invalid_grant",\n "error_description": "Invalid JWT Signature."\n}')

[sww314]

Do not do both. Since you are running locally, download the service account JSON and specify the GS_CREDENTIALS in settings.py.

Delete any other mention of GOOGLE_APPLICATION_CREDENTIALS.

I run this way every day.

As a side note, you can not make the files private and the hack the URL like you have in the example code. It might work for you, but your users will not see the files.

Since this is a support issue vs a bug. It might be better if you ask the question on Stackoverflow and post it here.

[ajs-88]

@sww314 I disabled the the GOOGLE_APPLICATION_CREDENTIALS, and GS_DEFAULT_ACL as well. but still the same error :(

[sww314]

Please post your code somewhere and move this to StackOverflow. No one can help you without seeing the details.

[ajs-88]

@sww314 @kasperj93 Sorry guys, it was an issue with my GOOGLE_APPLICATION_CREDENTIALS json file. Once i created a new one with appropriate permission, things works like charm. :)