Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azure backend strips some non-alphabetic characters from the file name #609

Closed
moriyoshi opened this issue Oct 1, 2018 · 4 comments · Fixed by #752
Closed

Azure backend strips some non-alphabetic characters from the file name #609

moriyoshi opened this issue Oct 1, 2018 · 4 comments · Fixed by #752
Labels
azure

Comments

@moriyoshi
Copy link

moriyoshi commented Oct 1, 2018 •
edited
Loading

https://github.com/jschneier/django-storages/blame/master/storages/backends/azure_storage.py#L109

Is there any reason for stripping characters other than:

  • a-z A-Z 0-9
  • _
  • -
  • .
  • /

?

As much as I learn from the comment, url-escaping will do for special characters that occur inbetween .
@davidfischer
Copy link
Contributor

This bug causes a problem if AzureStorage is used with the Django's ManifestFilesMixin. For example, the contents of my CSS files are being post processed from url('../fonts/fontawesome-webfont.eot?v=4.4.0') to url("../fonts/fontawesome-webfont.674f50d287a8.eotv4.4.0") (note the missing ? and =).

davidfischer added a commit to readthedocs/readthedocs.org that referenced this issue Dec 5, 2018
@davidfischer
Workaround for a django-storages bug
3d3a700 
- jschneier/django-storages#609
@davidfischer davidfischer mentioned this issue Dec 5, 2018
Merged
@nitely
Copy link
Contributor

nitely commented Jan 24, 2019 •
edited
Loading

It's not a bug, Django's default storage does not allow this either. What happens when someone uploads a file with a special character that is not allowed by all OS filesystems? can those files be downloaded later? (i.e interrogation mark is not allowed in Windows)

Also, allowing all characters means the developer may have to quote the name to avoid XSS. I believe Django templates will do that, but that's not the case for REST APIs, other template systems (jinja?), etc.

For static files, Django's collect command should strip querystrings, since those are not needed.

nitely added a commit to satellogic/django-storages that referenced this issue Feb 12, 2019
@nitely
remove file name special char cleaner fixes jschneier#609
6600749
@nitely
Copy link
Contributor

nitely commented Feb 13, 2019 •
edited
Loading

^ After some testing I've found Django will sanitize the file name when saving into the DB, doing basically the same as the Azure storage. This can be bypassed by calling default_storage.save(...) or passing a callable to models.FileField(upload_to=callable) (that's fixed in Django 1.11) or overriding the get_valid_name to return the name avoiding the cleaning. So I believe we should make the Azure storage behave like the rest of django-storages's storages. I've a PR fixing this underway.

nitely added a commit to satellogic/django-storages that referenced this issue Feb 20, 2019
@nitely
remove file name special char cleaner fixes jschneier#609
e765569
@nitely nitely mentioned this issue Feb 20, 2019
Closed
nitely added a commit to satellogic/django-storages that referenced this issue Feb 23, 2019
@nitely
remove file name special char cleaner fixes jschneier#609
6a12ed1
@sww314 sww314 added the azure label May 13, 2019
@akshaybabloo
Copy link

Any update on merging the pull request?

@nitely nitely mentioned this issue Sep 9, 2019
Merged
jschneier pushed a commit that referenced this issue Sep 9, 2019
@nitely @jschneier
Azure: improve handling of special characters in filenames (#752)
2fbaa70 
*  remove file name special char cleaner fixes #609

* tests
bors bot added a commit to mozilla/normandy that referenced this issue Sep 17, 2019
@bors @pyup-bot
Merge #1984
f91d3a5 
1984: Scheduled weekly dependency update for week 37 r=rehandalal a=pyup-bot






### Update [botocore](https://pypi.org/project/botocore) from **1.12.224** to **1.12.228**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.12.228
   ```
   ========

* api-change:``elbv2``: Update elbv2 client to latest version
* api-change:``ec2``: Update ec2 client to latest version
* api-change:``workmailmessageflow``: Update workmailmessageflow client to latest version
* api-change:``medialive``: Update medialive client to latest version
   ```
   
  
  
   ### 1.12.227
   ```
   ========

* api-change:``stepfunctions``: Update stepfunctions client to latest version
* api-change:``rds``: Update rds client to latest version
* api-change:``ec2``: Update ec2 client to latest version
* api-change:``mediaconnect``: Update mediaconnect client to latest version
* api-change:``ses``: Update ses client to latest version
* api-change:``config``: Update config client to latest version
   ```
   
  
  
   ### 1.12.226
   ```
   ========

* api-change:``storagegateway``: Update storagegateway client to latest version
   ```
   
  
  
   ### 1.12.225
   ```
   ========

* api-change:``qldb``: Update qldb client to latest version
* api-change:``marketplacecommerceanalytics``: Update marketplacecommerceanalytics client to latest version
* api-change:``appstream``: Update appstream client to latest version
* api-change:``ec2``: Update ec2 client to latest version
* api-change:``robomaker``: Update robomaker client to latest version
* api-change:``appmesh``: Update appmesh client to latest version
* api-change:``qldb-session``: Update qldb-session client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/botocore
  - Changelog: https://pyup.io/changelogs/botocore/
  - Repo: https://github.com/boto/botocore
</details>





### Update [certifi](https://pypi.org/project/certifi) from **2019.6.16** to **2019.9.11**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/certifi
  - Homepage: https://certifi.io/
</details>





### Update [datadog](https://pypi.org/project/datadog) from **0.29.3** to **0.30.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 0.30.0
   ```
   * [BUGFIX] Treat `API_HOST` as URL, not as string [411][]
* [FEATURE] Add `return_raw_response` option to `initialize` to enable adding raw responses to return values [414][]
* [IMPROVEMENT] Add project URLs to package metadata [413][] (thanks [Tenzer][])
* [IMPROVEMENT] Add support for handling a 401 status as an API error [418][]
* [IMPROVEMENT] Allow configuring proxy in `~/.dogrc` for usage with dogshell [415][]
* [IMPROVEMENT] Update `user` resource name to `users` to match new plural endpoints [421][]
* [OTHER] Add deprecation warning to old aws lambda threadstats integration [417][]
* [OTHER] Removed functionality to delete events and comments, as it&#39;s no longer supported by API [420][]
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/datadog
  - Changelog: https://pyup.io/changelogs/datadog/
  - Homepage: https://www.datadoghq.com
</details>





### Update [importlib-metadata](https://pypi.org/project/importlib-metadata) from **0.20** to **0.23**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/importlib-metadata
  - Docs: http://importlib-metadata.readthedocs.io/
</details>





### Update [pluggy](https://pypi.org/project/pluggy) from **0.12.0** to **0.13.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 0.13.0
   ```
   ==========================

Trivial/Internal Changes
------------------------

- `222 &lt;https://github.com/pytest-dev/pluggy/issues/222&gt;`_: Replace ``importlib_metadata`` backport with ``importlib.metadata`` from the
  standard library on Python 3.8+.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pluggy
  - Changelog: https://pyup.io/changelogs/pluggy/
  - Repo: https://github.com/pytest-dev/pluggy
</details>





### Update [boto3](https://pypi.org/project/boto3) from **1.9.224** to **1.9.228**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.9.228
   ```
   =======

* api-change:``elbv2``: [``botocore``] Update elbv2 client to latest version
* api-change:``ec2``: [``botocore``] Update ec2 client to latest version
* api-change:``workmailmessageflow``: [``botocore``] Update workmailmessageflow client to latest version
* api-change:``medialive``: [``botocore``] Update medialive client to latest version
   ```
   
  
  
   ### 1.9.227
   ```
   =======

* api-change:``stepfunctions``: [``botocore``] Update stepfunctions client to latest version
* api-change:``rds``: [``botocore``] Update rds client to latest version
* api-change:``ec2``: [``botocore``] Update ec2 client to latest version
* api-change:``mediaconnect``: [``botocore``] Update mediaconnect client to latest version
* api-change:``ses``: [``botocore``] Update ses client to latest version
* api-change:``config``: [``botocore``] Update config client to latest version
   ```
   
  
  
   ### 1.9.226
   ```
   =======

* api-change:``storagegateway``: [``botocore``] Update storagegateway client to latest version
   ```
   
  
  
   ### 1.9.225
   ```
   =======

* api-change:``qldb``: [``botocore``] Update qldb client to latest version
* api-change:``marketplacecommerceanalytics``: [``botocore``] Update marketplacecommerceanalytics client to latest version
* api-change:``appstream``: [``botocore``] Update appstream client to latest version
* api-change:``ec2``: [``botocore``] Update ec2 client to latest version
* api-change:``robomaker``: [``botocore``] Update robomaker client to latest version
* api-change:``appmesh``: [``botocore``] Update appmesh client to latest version
* api-change:``qldb-session``: [``botocore``] Update qldb-session client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/boto3
  - Changelog: https://pyup.io/changelogs/boto3/
  - Repo: https://github.com/boto/boto3
</details>





### Update [django-countries](https://pypi.org/project/django-countries) from **5.4** to **5.5**.


<details>
  <summary>Changelog</summary>
  
  
   ### 5.5
   ```
   =======================

- Django 3.0 compatibility.

- Plugin system for extending the ``Country`` object.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/django-countries
  - Changelog: https://pyup.io/changelogs/django-countries/
  - Repo: https://github.com/SmileyChris/django-countries/
</details>





### Update [django-storages](https://pypi.org/project/django-storages) from **1.7.1** to **1.7.2**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.7.2
   ```
   ******************

S3
--

- Avoid misleading ``AWS_DEFAULT_ACL`` warning for insecure ``default_acl`` when
  overridden as a class variable (`591_`)
- Propagate file deletion to cache when ``preload_metadata`` is ``True``,
  (not the default) (`743`_, `749`_)
- Fix exception raised on closed file (common if using ``ManifestFilesMixin`` or
  ``collectstatic``. (`382`_, `754`_)

Azure
-----

- Pare down the required packages in ``extra_requires`` when installing the ``azure`` extra to only
  ``azure-storage-blob`` (`680`_, `684`_)
- Fix compatability with ``generate_blob_shared_access_signature`` updated signature (`705`_, `723`_)
- Fetching a file now uses the configured timeout rather than hardcoding one (`727`_)
- Add support for configuring all blobservice options: ``AZURE_EMULATED_MODE``, ``AZURE_ENDPOINT_SUFFIX``,
  ``AZURE_CUSTOM_DOMAIN``, ``AZURE_CONNECTION_STRING``, ``AZURE_CUSTOM_CONNECTION_STRING``,
  ``AZURE_TOKEN_CREDENTIAL``. See the docs for more info. Huge thanks once again to nitely. (`750`_)
- Fix filename handling to not strip special characters (`609`_, `752`_)


Google Cloud
------------

- Set the file acl in the same call that uploads it (`698`_)
- Reduce the number of queries and required permissions when ``GS_AUTO_CREATE_BUCKET`` is
  ``False`` (the default) (`412`_, `718`_)
- Set the ``predefined_acl`` when creating a ``GoogleCloudFile`` using ``.write``
  (`640`_, `756`_)
- Add ``GS_BLOB_CHUNK_SIZE`` setting to enable efficient uploading of large files (`757`_)

Dropbox
-------

- Complete migration to v2 api with file fetching and metadata fixes (`724`_)
- Add ``DROPBOX_TIMEOUT`` to configure client timeout defaulting to 100 seconds
  to match the underlying sdk. (`419`_, `747`_)

SFTP
----

- Fix reopening a file (`746`_)

.. _591: jschneier/django-storages#591
.. _680: jschneier/django-storages#680
.. _684: jschneier/django-storages#684
.. _698: jschneier/django-storages#698
.. _705: jschneier/django-storages#705
.. _723: jschneier/django-storages#723
.. _727: jschneier/django-storages#727
.. _746: jschneier/django-storages#746
.. _724: jschneier/django-storages#724
.. _412: jschneier/django-storages#412
.. _718: jschneier/django-storages#718
.. _743: jschneier/django-storages#743
.. _749: jschneier/django-storages#749
.. _750: jschneier/django-storages#750
.. _609: jschneier/django-storages#609
.. _752: jschneier/django-storages#752
.. _382: jschneier/django-storages#382
.. _754: jschneier/django-storages#754
.. _419: jschneier/django-storages#419
.. _747: jschneier/django-storages#747
.. _640: jschneier/django-storages#640
.. _756: jschneier/django-storages#756
.. _757: jschneier/django-storages#757
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/django-storages
  - Changelog: https://pyup.io/changelogs/django-storages/
  - Repo: https://github.com/jschneier/django-storages
</details>





### Update [kinto-http](https://pypi.org/project/kinto-http) from **10.4.1** to **10.5.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 10.5.0
   ```
   ===================

**New features**

- Add history support (fixes 112), Thanks FlorianKuckelkorn!
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/kinto-http
  - Changelog: https://pyup.io/changelogs/kinto-http/
  - Repo: https://github.com/Kinto/kinto-http.py/
</details>







Co-authored-by: pyup-bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
azure
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Azure name cleanup
Fix Azure name cleanup satellogic/django-storages
5 participants
@moriyoshi @davidfischer @sww314 @nitely @akshaybabloo