Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

File system operations do not adhere to Django file storage API #6414

Closed
mtnorthcott opened this issue Sep 7, 2020 · 0 comments · Fixed by #6425
Closed

File system operations do not adhere to Django file storage API #6414

mtnorthcott opened this issue Sep 7, 2020 · 0 comments · Fixed by #6425
Assignees
@mtnorthcott
Labels
enhancement in progress Work on issue has already started
Milestone
3.1

Comments

@mtnorthcott
Copy link
Contributor

mtnorthcott commented Sep 7, 2020 •
edited
Loading

Expected Behavior

All file system operations correctly utilise Django's file storage API allowing third-party Django plugins such as django-storage-swift to function correctly if they implement Django's documented functions.

Actual Behavior

Not all file system operations adhere to the Django file storage API causing issues when third-party extensions are used. For example, line 1319 of base/models.py attempts a glob at the specified thumbnail path in the filesystem. If this filesystem is remotely hosted using django-storage-swift or similar, an exception is thrown and the thumbnail fails to generate.

Steps to Reproduce the Problem

  1. Activate your GeoNode virtual environment and run pip install django-storage-swift
  2. Edit your settings.py and add parameters listed in the django-storage-swift documentation. You do not need to specify static file storage parameters.
  3. Load GeoNode and add a layer
  4. Observe failure to generate the thumbnail resulting in the default being used (subsequent 'Set Thumbnail' operations will also fail)

Specifications

  • GeoNode version: 3.0
  • Installation method (manual, GeoNode Docker, SPCGeoNode Docker): manual
  • Platform: Ubuntu 18.04 LTS
  • Additional details:
@mtnorthcott mtnorthcott mentioned this issue Sep 8, 2020
Merged
11 tasks
@mtnorthcott mtnorthcott changed the title Thumbnails fail to generate when using remote Django storage File system operations do not adhere to Django file storage API Sep 8, 2020
mtnorthcott pushed a commit to mtnorthcott/geonode that referenced this issue Sep 8, 2020
[Issue GeoNode#6414] Delete thumbnails using Django file storage API
3131bc4
@t-book t-book added enhancement in progress Work on issue has already started labels Sep 8, 2020
@t-book t-book added this to the 3.1 milestone Sep 8, 2020
afabiani pushed a commit that referenced this issue Sep 8, 2020
[Issue #6414] Delete thumbnails using Django file storage API (#6415)
d4c9311 
Co-authored-by: Alessio Fabiani <[email protected]>
mtnorthcott pushed a commit to mtnorthcott/geonode that referenced this issue Sep 9, 2020
[Issue GeoNode#6414] Use Django storage API when generating document …
6308fb0 
…thumbnails
afabiani pushed a commit that referenced this issue Sep 10, 2020
[Issue #6414] Delete thumbnails using Django file storage API (#6415)
90c436d 
Co-authored-by: Alessio Fabiani <[email protected]>
(cherry picked from commit d4c9311)
@mtnorthcott mtnorthcott mentioned this issue Sep 11, 2020
Merged
11 tasks
mtnorthcott pushed a commit to mtnorthcott/geonode that referenced this issue Sep 22, 2020
[Issue GeoNode#6414] Use Django storage API in delete_orphaned_* func…
fde4d1d 
…tions
mtnorthcott pushed a commit to mtnorthcott/geonode that referenced this issue Sep 22, 2020
[Issue GeoNode#6414] Use Django storage API in geonode.qgis_server.te…
0e9d9fe 
…sts.test_views
mtnorthcott pushed a commit to mtnorthcott/geonode that referenced this issue Sep 22, 2020
[Issue GeoNode#6414] Use Django storage API when generating document …
e3c0a96 
…thumbnails
mtnorthcott pushed a commit to mtnorthcott/geonode that referenced this issue Sep 22, 2020
[Issue GeoNode#6414] Thumbnail generation fix for local storage
3c4fca5
afabiani pushed a commit that referenced this issue Sep 23, 2020
[Issue #6414] File system operations do not adhere to Django file sto…
771e154 
…rage API (#6425)

* [Issue #6414] Use Django storage API in delete_orphaned_* functions

* [Issue #6414] Use Django storage API in geonode.qgis_server.tests.test_views

* [Issue #6414] Use Django storage API when generating document thumbnails

* [Issue #6414] Thumbnail generation fix for local storage

* Add thumbnail convenience functions

* Cleanup Django storage API changes
afabiani pushed a commit that referenced this issue Sep 23, 2020
[Issue #6414] File system operations do not adhere to Django file sto…
2b364dc 
…rage API (#6425)

* [Issue #6414] Use Django storage API in delete_orphaned_* functions

* [Issue #6414] Use Django storage API in geonode.qgis_server.tests.test_views

* [Issue #6414] Use Django storage API when generating document thumbnails

* [Issue #6414] Thumbnail generation fix for local storage

* Add thumbnail convenience functions

* Cleanup Django storage API changes

(cherry picked from commit 771e154)
afabiani pushed a commit that referenced this issue Dec 4, 2020
@t-book
[Fixes #6685] GNIP-79: GeoNode REST APIs (v2) (#6686)
07e9c1b 
* [WIP] GeoNode API v2 - Prototype

* [ref. #6388] Error when installing core GeoNode into virtual environment

* - GeoNode REST APIs v2 - Swagger schema

* [ref. #6388] Error when installing core GeoNode into virtual environment

* - GeoNode REST API v2 : Map and MapLayers endpoints

* - GeoNode REST API v2 : Adding workflow ResourceBase metadata fields

* [Dependencies] Removing conflicts

* - Minor fixes and improvements

* - Improving ResourceBase REST api permissions

* - Improve Dynamic REST Sorting/Filtering

* - Expose more metadata fields

* - Expose more polymorphic_ctype_id field for filtering

* - Improve Metadata Fields serialization in order to make it searchable/filterable

* - Updating style sheets

* - Introducing "DynamicSearchFilter" exposing search_fields

* Merge branch 'master' of https://github.com/GeoNode/geonode into rest_api_v2_proof_of_concept

# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.

* - Allow gs proxy to parse workspace prefix dynamically

* - Downgrade pyproj to 2.6.1

* Revert " - Downgrade pyproj to 2.6.1"

This reverts commit a702740.

* - Improve Dockerfile

* [Issue #6414] Use Django storage API in delete_orphaned_* functions

* [Issue #6414] Use Django storage API in geonode.qgis_server.tests.test_views

* [Issue #6414] Use Django storage API when generating document thumbnails

* [Issue #6414] Thumbnail generation fix for local storage

* Add thumbnail convenience functions

* Cleanup Django storage API changes

* [Hardening] Minor improvements and error checks

* [Minor] Headers and formatting files

* - Adding "OAuth2Authentication" to the default REST APIs auth_classes

* - Fix "get_perms" api issue with non existant keys

* - Fix "get_perms" api issue with non existant keys

* [APIs] superusers have always perms to change resources

* [Dependencies] bump djangorestframework to >=3.1.0,<3.12.0

* - Improve IsOwnerOrReadOnly REST permission class

* - Improve IsOwnerOrReadOnly REST permission class

* - Improve IsOwnerOrReadOnly REST permission class

* - Fix Recenet Activity List for Documents

* [Hardening] - Recenet Activity List for Documents error when actor is None

* [Frontend] Monitoring: Bump "node-sass" to version 4.14.1

* [Frontend] Bump jquery to version 3.5.1

* [Fixes: #6519] Bump jquery to 3.5.1 (#6526)

(cherry picked from commit e532813)

# Conflicts:
#	geonode/static/lib/css/assets.min.css
#	geonode/static/lib/css/bootstrap-select.css
#	geonode/static/lib/css/bootstrap-table.css
#	geonode/static/lib/js/assets.min.js
#	geonode/static/lib/js/bootstrap-select.js
#	geonode/static/lib/js/bootstrap-table.js
#	geonode/static/lib/js/leaflet-plugins.min.js
#	geonode/static/lib/js/leaflet.js
#	geonode/static/lib/js/moment-timezone-with-data.js
#	geonode/static/lib/js/underscore.js

* Merge branch 'master' of https://github.com/GeoNode/geonode into rest_api_v2_proof_of_concept

# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.

* [Hardening] Re-create the map thumbnail only if it is missing

* Fixes error with GDAL 3.0.4 due to a breaking change on GDAL (https://code.djangoproject.com/ticket/30645)

* Fixes error with GDAL 3.0.4 due to a breaking change on GDAL (https://code.djangoproject.com/ticket/30645)

* - Introducing REST APIs for "Docuemnts" resources too

* - Bump django_mapstore_adapter to 2.0.6 / Bump django-geonode-mapstore-client to 2.0.9

* - Travis and LGTM fixes

* - Swagger OAS3

* - Adding REST API v2 Test Suite

* - Implementing API v2 "extent" filter

* - Travis and LGTM fixes

Co-authored-by: Matthew Northcott <[email protected]>
Co-authored-by: Toni <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mtnorthcott mtnorthcott

Labels
enhancement in progress Work on issue has already started
Projects
None yet
Milestone
3.1
Development

Successfully merging a pull request may close this issue.

[Issue #6414] File system operations do not adhere to Django file storage API mtnorthcott/geonode
2 participants
@mtnorthcott @t-book