Chunk Upload Big File for Upload Layer #294
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add qgis_server backend support for geonode: * Add QGIS Server app. * Push layers to local storage for qgis backend. * Handling for updating old layer object. * Implementation for deleting a layer. * Put delete layer in pre_delete QGISServerLayer * Add link to download zip file. * Better link in layer detail page. * create a QGIS project when a layer is uploaded * update qgis project * add gis tools * serve tiles with a WMS wrapper * forward legend request to QGIS WMS if the legend is not in cache * Add tile url to link. Able to show the layer now. * Add link for legend. * check if the PNG is valid, remove it if it is not valid * get a thumbnail from qgis server * Move import signals to end of model to avoid circular import. * Add thumbnail for layers. * add indicator for analysis task process * Hide filename for generated qml legend * First working wms map in the map page creation. * better code structure and add local_setting sample. * enable WFS in the qgis template * Replace SRID 900913 to 3857. * Add DescribeLayer in qgisserver app. * Add DescribeFeatureType in qgisserver app. * Add Describe FeatureType * Renaming wms to better method name. * Styling legend. * Make print map dialog works. * Temporary hack for map printing. * fix transparency when calling the WMS * fix qgis project with a raster * fix the projection in the QGIS template if different than 4326 * remove cache when we remove the qgis layer * set the attributes for a vector layer using WFS * remove qgis project template * fix attribute table when calling the WFS with the new server plugin * Set local layer. * Call the server plugin while saving the layer in the DB to create the qgis project * update mimetype to content_type due to django 1.7 * create qgis map * Refactor django.conf.settings access * Refactor thumbnail creation to use celery * Use relative url for layer thumbnail. * Update migrations for geonode:2.6.x * fix kartoza#19: thumbnail and legend url resolver * issue kartoza#21: Provide GeoTIFF download url * delete and copy layers properly: * fix kartoza#24: File saved as different name sometimes * fix kartoza#25: Cache files not deleted when layer deleted
- Add thumbnail for map - Fix thumbnail tests for layer and map
* include layer links in layers API * include map layer list in maps API * include geonode version in output for resource based API
This is to refrain from bothering them with unnecessary emails
- Refactor thumbnail url generator for QGIS Server. Now giving direct url to QGIS Server, instead of proxying to Django. - Remove unnecessary thumbnail view in qgis_server, because geonode already have internal cache for thumbnail. Thumbnail should just be accessed directly to QGIS Server without proxy. - Fix map creation test. - Refactor wrong/inconsistent bbox convention for maps.
Enable ASCII file upload and create link for ASCII file.
* Add download zipped files for non-shapefile. * Use image/tiff for qgis server GeoTIFF mime links. * Add unit test for zipped all files link for raster.
Update geonode master
Regular update from Geonode/master
**Included to bower.json** * Leaflet opacity control plugin * Leaflet measure plugin * Leaflet navbar plugin **Modify gruntfile tasks** * Modify css url images regex pattern for replace task * Create another minified file for leaflet plugins * Refactor some js filename to match in what used in the template when DEBUG_STATIC is true * Add task replace for task production **Related changes** * Included images from library from grunt copy:development task * New minified js for assets.min.js and leaflet-plugins.min.js from production task * New minified css for assets.min.css and leaflet-plugins.min.css * Moved some pngs from lib/css folder to lib/img from the new replace task * Include Geonode Logo
# Refactor additional fields in Layers API # Implement Layer Style Change. - User now can upload new QML file to change layer style in layer detail UI - Refactor QGIS Project creations # Implement set thumbnail functionality - Add unittest to make sure thumbnail url can be used # Improve support for management command - Add management command to delete orphaned layer files - Add management command to delete orphaned QGIS Server related files - Allow overwrite for QGIS Server backend when running importlayers management command - Add unittest for importlayers management command for qgis_server backend
BBOX will be validated by qgis_server helpers.
- Convert bbox to EPSG:4326 for displaying layers - Try to retrieve layer projection and save Layer CRS - Standardize thumbnail generation to use EPSG:4326
lucernae
reviewed
Jul 14, 2017
There was a problem hiding this comment.
Thanks @euclidian @meomancer
Some several notes:
- How can I test this? Can you explain how do you test that it works?
- Can you explain just a little bit, how this works? 😄 I still don't understand from reading this. Just a summary is ok.
- I think this functionality should go directly to core geonode (if this is an improvement from current layer upload and not just an alternative). We can do it after it review, then you probably want to squash these commits into one, then rebase to geonode master on other branch, then do PR directly to geonode master.
| @@ -0,0 +1,59 @@ | |||
| from django.contrib.auth.decorators import login_required | |||
|
|
|||
| __author__ = 'Irwan Fathurrahman <[email protected]>' | |||
There was a problem hiding this comment.
Change file header to standard geonode header, like in other files, and follow the convention.
|
|
||
| LayerInfo.prototype._chunkUpload = function(file, url, uuid, onSuccess) { | ||
| var loaded = 0; | ||
| var step = 1024; // Size of a chunk |
There was a problem hiding this comment.
Size of chunk should be configurable from Geonode Settings.
| //get all files that listed with this layer | ||
| var files = this.files; | ||
| var self = this; | ||
| var url = "/layers/api/chunk-file-uploader"; |
There was a problem hiding this comment.
Can this url becomes parameter, so you can reverse it in the template? Just to make sure if someone changes urls, it doesn't break.
| @@ -21,18 +21,25 @@ | |||
| from django.conf.urls import patterns, url | |||
| from django.conf import settings | |||
| from django.views.generic import TemplateView | |||
| from layer_uploader import upload_chunk | |||
There was a problem hiding this comment.
use absolute import --> geonode.layers.layer_uploader
| f.write(_file.read()) | ||
|
|
||
| # send response with appropriate mime type header | ||
| return HttpResponse(status=200, content=json.dumps({ |
There was a problem hiding this comment.
content-type to application/json?
| @@ -77,6 +77,7 @@ <h2 class="page-title">{% trans "Upload Layers" %}</h2> | |||
|
|
|||
| <form id="file-uploader" method="post" enctype="multipart/form-data"> | |||
| <!-- UI change to hide the list of previously uploaded files from the user --> | |||
| <input id="file_uuid" name="file_uuid" type="text" hidden/> | |||
| cache: false, | ||
| contentType: false, | ||
| processData: false | ||
| }).done(function(data){ |
There was a problem hiding this comment.
why not just put:
}).done(done);
😄
| // this function will be invoked each time a file reader read a file | ||
| // Thus will be recursively called for each chunk | ||
| reader.onload = function(e){ | ||
| console.log("Step Count: " + stepCount++ + " for file: " + file.name); |
There was a problem hiding this comment.
delete this after you finished testing.
| @@ -309,7 +309,8 @@ define(['underscore', | |||
| alert(gettext('You are trying to upload an incomplete set of files or not all mandatory options have been validated.\n\nPlease check for errors in the form!')); | |||
| } else { | |||
| $.each(layers, function (name, layerinfo) { | |||
| layerinfo.uploadFiles(); | |||
| //before uploading file, chunk upload first | |||
| layerinfo.chunkUpload(); | |||
There was a problem hiding this comment.
So, this will replace the previous functionality?
|
@euclidian please update |
lucernae
force-pushed
the
master-qgis_server
branch
from
aa5509a
to
b48e347
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes #52
I've added chunking functionality for big file. But this needs to be tested further. For now I set it for 1 KB for each slice data (need to changed).
Test that I've tried
Can anyone please provide me other list of tests that I need to do to make sure that this PR works ?