Download filtered layer #3321

karakostis · 2017-09-25T17:42:11Z

In GeoNode it is possible to download parts of a layer using a Query Builder. The Query Builder provides a step by step graphical environment, where the user is able to select attributes existing in the layer (e.g. population of an area) and set numerical or text filters.

The filters can be chained together either by an AND or an OR operator. Moreover when building the Query statements, the user can retrieve all the available values for each attribute. This is particularly useful, when the user wants to filter data based on numbers, locations etc. which he/she is not familiarized.

As this is my first commit to GeoNode community, kindly let me know in case of corrections, best practices and recommendations. Thanks!

…load_layer

…load_filtered_layer

simod · 2017-09-26T07:12:20Z

Thanks @karakostis , some comments in line. The tests are failing, could you take a look? I could be simple pep8 violations.

…load_filtered_layer

karakostis · 2017-09-26T10:54:41Z

Hey @simod thanks! I think it was a pep8 issue with setup.py. Lets see if is resolved. Silly question but where did u put the inline comments? :)

simod · 2017-09-26T11:05:15Z

:) here in the review https://github.com/GeoNode/geonode/pull/3321/files#diff-36bc05580e37aef61b66c480db47cc38R435

simod · 2017-09-26T07:08:07Z

geonode/layers/views.py

@@ -430,8 +432,117 @@ def layer_detail(request, layername, template='layers/layer_detail.html'):
    if settings.SOCIAL_ORIGINS:
        context_dict["social_links"] = build_social_links(request, layer)

+
+    #d


@karakostis Is this part of code needed by your feature?

simod · 2017-09-26T07:09:00Z

geonode/layers/views.py

+
+        # loop the dictionary based on the values on the list and add the properties
+        # in the dictionary (if doesn't exist) together with the value
+        from six import string_types


@karakostis can you import this at the top?

simod · 2017-09-26T07:10:18Z

setup.py

@@ -141,7 +141,7 @@
        "factory_boy",
        # "WeasyPrint",

-        ],


this is causing conflicts, can you merge with master?

simod · 2017-09-26T11:06:10Z

oh probably was my fault, I didn't hit the submit comments... sorry

karakostis · 2017-09-26T22:06:37Z

I am not really sure what is the issue. Is there a way to see where is the conflict?

simod · 2017-09-27T07:21:37Z

woah this log became way too long to read. Did you run the tests locally? @afabiani most of the noise seems to be related to geoserver messaging, any way we can limit it?

simod · 2017-09-27T07:23:52Z

geonode/layers/views.py

@@ -429,12 +432,119 @@ def layer_detail(request, layername, template='layers/layer_detail.html'):
    if settings.SOCIAL_ORIGINS:
        context_dict["social_links"] = build_social_links(request, layer)

+


@karakostis I was talking about this whole chunk of code, is it related to your feature?

Yes. This part of code is related to the feature.

So the issue must be related with the that chunk of code. As I see in the log file something breaks there and returns the Exception message:

LL: <CDLL 'libc.so.6', handle 7fb2157ee9b0 at 7fb1f573ce90>
.on_geoserver_messages: RECEIVED MSG - body: {'rating': '0', 'constraints_other': 'None', 'bbox_x0': 'None', 'csw_mdsource': 'local', 'csw_schema': 'http://www.isotc211.org/2005/gmd', 'csw_typename': 'gmd:MD_Metadata', 'date_type': 'publication', 'srid': 'EPSG:4326', 'bbox_x1': 'None', 'edition': 'None', 'featured': 'False', 'owner': 'None', 'detail_url': 'None', 'id': 1, 'category': 'None', 'license': 'None', 'spatial_representation_type': 'None', 'uuid': u'', 'title': u'', 'alternate': 'None', 'regions': '[]', 'thumbnail_url': 'None', 'share_count': '0', 'abstract': u'', 'bbox_y0': 'None', 'bbox_y1': 'None', 'popular_count': '0', 'contacts': '[]', 'purpose': 'None', 'date': '2017-09-26 15:58:14.589434', 'group': 'None', 'temporal_extent_end': 'None', 'language': 'eng', 'data_quality_statement': 'None', 'restriction_code_type': 'None', 'maintenance_frequency': 'None', 'csw_anytext': 'None', 'metadata_xml': '<gmd:MD_Metadata xmlns:gmd="http://www.isotc211.org/2005/gmd"/>', 'tkeywords': '[]', 'metadata_uploaded_preserve': 'False', 'csw_type': 'dataset', 'keywords': '[]', 'metadata_uploaded': 'False', 'csw_wkt_geometry': 'POLYGON((-180 -90,-180 90,180 90,180 -90,-180 -90))', 'temporal_extent_start': 'None', 'is_published': True}
Layer matching query does not exist.
Traceback (most recent call last):
File "/home/travis/build/GeoNode/geonode/geonode/geoserver/signals.py", line 94, in geoserver_post_save_local
instance = Layer.objects.get(id=layer_id)
File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/django/db/models/query.py", line 334, in get
self.model._meta.object_name
DoesNotExist: Layer matching query does not exist.
on_geoserver_messages: finished
finished.

it is not clear, a geoserver post_save signal cannot find the resource for which it was called. Strange.

karakostis · 2017-09-28T19:10:46Z

The errors take place in the following two lines. In the first one I use gsconfig:
resource = cat.get_resource(name, workspace=workspace)
For some reason this returns nothing back ('resource', None)
The second line of code uses OWSLib:
schema = get_schema(location, name, username=username, password=password)
I wonder if this is actually related with the tests them selves and not with the code.
I have imported these two libs in views.py (for layers) specifically for this functionality).

simod · 2017-09-29T07:43:13Z

geonode/layers/views.py

+
+    try:
+        # get type of layer (raster or vector)
+        cat = Catalog(settings.OGC_SERVER['default']['LOCATION'] + "rest", settings.OGC_SERVER['default']['USER'], settings.OGC_SERVER['default']['PASSWORD'])


@karakostis we actually should avoid to call the catalog here. Catalog operations should only be delegated to the geoserver app, layer detail should only be reading from the database.

karakostis · 2017-10-01T09:20:31Z

Thanks @simod . Well noted. I will change this. But I think there must be another more general issue. I tried to run the tests locally in my master branch and I still keep getting the same errors. Can you replicate? Tests run smoothly from your side?

simod · 2017-10-02T05:53:41Z

@karakostis consider that right now the tests are failing on master, probably due to some recent updates in the java machinery packages. But I'll run them locally too and help out to see what we can change.

…download_filtered_layer

simod · 2017-10-03T07:28:58Z

@karakostis can you merge from master now? Let's see if the tests give better result.

…download_filtered_layer

karakostis · 2017-10-04T12:29:03Z

@simod I have refactored part of the code in the layers/views.py so that I don't call the Catalog. Running the tests causes same issues. When I run them locally I get errors in:

ERROR: testLayerNotifications (geonode.layers.tests.LayerNotificationsTestCase)
ERROR: testMapsNotifications (geonode.maps.tests.MapsNotificationsTestCase)
ERROR: testDocumentNotifications (geonode.documents.tests.DocumentNotificationsTestCase)

afabiani · 2017-10-04T14:01:56Z

@karakostis @simod I pushed another build of this PR. Currently the unique issue should be related to formatting stuff

flake8 geonode
geonode/layers/views.py:436:5: E303 too many blank lines (2)
geonode/layers/views.py:441:13: F841 local variable 'workspace' is assigned to but never used
geonode/layers/views.py:487:5: E303 too many blank lines (2)
geonode/layers/views.py:490:1: E302 expected 2 blank lines, found 1
geonode/layers/views.py:513:9: E303 too many blank lines (2)
geonode/layers/views.py:534:1: E302 expected 2 blank lines, found 1

coveralls · 2017-10-04T14:55:49Z

Coverage decreased (-0.08%) to 44.569% when pulling 5c719b9 on karakostis:download_filtered_layer into d4140ce on GeoNode:master.

afabiani · 2017-10-04T15:00:49Z

@karakostis just tested it, It works but I noticed a javascript error when clicking on "+" button. Can you maybe double check?

karakostis · 2017-10-04T15:30:14Z

@afabiani sure!

karakostis and others added 17 commits September 11, 2017 16:02

changes in the libraries in setup.py and requirements.txt

280ca9c

Merge branch 'master' of https://github.com/GeoNode/geonode into down…

d1acfdd

…load_layer

download_layer

9d404b8

download_layer

8eeb044

resolve merge conflict

288b129

Merge branch 'master' of https://github.com/GeoNode/geonode into down…

1d49c60

…load_layer

remove unnecessary code

e0e2d6f

Merge branch 'master' of https://github.com/GeoNode/geonode into down…

6bdd9c7

…load_layer

adding max features and clear button

b5f8e3e

Add a link to create-layer in profile page (#3301)

cc02341

Download a layer filtered by attributes

240e157

Merge branch 'master' of https://github.com/GeoNode/geonode into down…

0127d25

…load_layer

Merge branch 'master' of https://github.com/GeoNode/geonode into down…

2c77f09

…load_layer

- Avoid Multiple Tags Exception

801511b

updated gitignore

4ebba31

Merge branch 'master' of https://github.com/GeoNode/geonode into down…

5143af3

…load_filtered_layer

small fixes

00aeffc

karakostis added 2 commits September 26, 2017 12:37

pep8 syntax issue fix

9f7d174

Merge branch 'master' of https://github.com/GeoNode/geonode into down…

33c2d41

…load_filtered_layer

simod reviewed Sep 26, 2017

View reviewed changes

karakostis added 2 commits September 26, 2017 18:08

proposed minor fixes

ab25093

merge conflict

42269da

simod reviewed Sep 27, 2017

View reviewed changes

simod reviewed Sep 29, 2017

View reviewed changes

karakostis added 2 commits October 2, 2017 08:41

refactoring download filtered layer

633482d

Merge branch 'master' of https://github.com/GeoNode/geonode into new_…

15469ee

…download_filtered_layer

karakostis added 3 commits October 3, 2017 09:39

Merge branch 'master' of https://github.com/GeoNode/geonode into new_…

6ac18d4

…download_filtered_layer

Merge branch 'master' of https://github.com/GeoNode/geonode into new_…

65576b6

…download_filtered_layer

fixes

db95c2b

blank line fixes

5c719b9

afabiani approved these changes Oct 4, 2017

View reviewed changes

afabiani merged commit eb65887 into GeoNode:master Oct 4, 2017

francbartoli added bdd-tests labels Oct 28, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download filtered layer #3321

Download filtered layer #3321

karakostis commented Sep 25, 2017 •

edited

Loading

simod commented Sep 26, 2017

karakostis commented Sep 26, 2017 •

edited

Loading

simod commented Sep 26, 2017

simod Sep 26, 2017

simod Sep 26, 2017

simod Sep 26, 2017

simod commented Sep 26, 2017

karakostis commented Sep 26, 2017

simod commented Sep 27, 2017

simod Sep 27, 2017

karakostis Sep 27, 2017

karakostis Sep 27, 2017

simod Sep 27, 2017

karakostis commented Sep 28, 2017 •

edited

Loading

simod Sep 29, 2017

karakostis commented Oct 1, 2017

simod commented Oct 2, 2017

simod commented Oct 3, 2017

karakostis commented Oct 4, 2017

afabiani commented Oct 4, 2017

coveralls commented Oct 4, 2017

afabiani commented Oct 4, 2017

karakostis commented Oct 4, 2017

		@@ -429,12 +432,119 @@ def layer_detail(request, layername, template='layers/layer_detail.html'):
		if settings.SOCIAL_ORIGINS:
		context_dict["social_links"] = build_social_links(request, layer)

Download filtered layer #3321

Download filtered layer #3321

Conversation

karakostis commented Sep 25, 2017 • edited Loading

simod commented Sep 26, 2017

karakostis commented Sep 26, 2017 • edited Loading

simod commented Sep 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simod commented Sep 26, 2017

karakostis commented Sep 26, 2017

simod commented Sep 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karakostis commented Sep 28, 2017 • edited Loading

Choose a reason for hiding this comment

karakostis commented Oct 1, 2017

simod commented Oct 2, 2017

simod commented Oct 3, 2017

karakostis commented Oct 4, 2017

afabiani commented Oct 4, 2017

coveralls commented Oct 4, 2017

afabiani commented Oct 4, 2017

karakostis commented Oct 4, 2017

karakostis commented Sep 25, 2017 •

edited

Loading

karakostis commented Sep 26, 2017 •

edited

Loading

karakostis commented Sep 28, 2017 •

edited

Loading