Layer symbology differs from the one in thumbnails generated on map creation

[NyakudyaA]

Problem

When creating a map you expect the thumbnail to be generated to be identical to the icons/symbology of the layer used to create the map with.

This is the thumbnail that is generated.

Solution

Investigate what is happening there. Why is the request being made returning icons that are different

[NyakudyaA]

Also related to this is the update map thumbnail which seems not to be working because of the above scenario. You cannot be sure if the thumbnail has changed because it has wrong symbology

[gubuntu]

duplicate of #325?

[boney-bun]

i'm confused with this issue, particularly for health facilities layer.
i've tried creating a map using other layers, all seems to be picked up the style:

trying passing STYLE parameter to a WMS GetMap request into qgis server. the returned image doesn't pick up the style.
the qgis server seems to ignore the parameter.

this led me to think that maybe something is not right with the health facilities layer.
how do we validate a style on a layer?

[boney-bun]

tested using postman on jalan sleman layer by specifyng either default or testing style, the qgis server can pick up the style.

so, my guess is the style of health facilities may have some bug.
can you help me with this @NyakudyaA ?

[gubuntu]

@boney-bun which server are these examples from? (Should be testing)

Note this issue is specifically about GeoNode Maps. The styles of the layers in maps are the same as the original layers. So the maps should use default layer styles.

This is still very much an issue (with Madagascar 1975 Map on testing)

There's no issue with Healthsites - the default style of any layer should be consistently used in layers, maps and thumbnails.

[boney-bun]

@gubuntu i've uploaded the health layer to the testing (http://testing.geonode.kartoza.com/layers/geonode:health_facilities).
when creating a map using this layer, the thumbnail is in a round shape (the original style is stars).
i think @NyakudyaA refers to this layer on the OP.

for the madagascar layer (and other layers), i think the qgis server tends to change the origin colors of the layer. an exception is for tandale layer.

[NyakudyaA]

@boney-bun You need to be checking this:

• When a user saves a map - How is the thumbnail generated ?. This method is currently wrong.
• When a user changes a style - How is the thumbnail generated. This is the correct method.

You need to make sure when a user clicks on save map the correct signal is passed to create the thumbnail.

A simple logic would be:

• If a user is creating a map using a single layer that already has a thumbnail in the layer detail page the thumbnail colors for the layer should be the same. Generate the thumbnail because it will also consist of a base map.
• If a user is creating a map using multiple layers then the thumbnail should consist of two or more layers plus the base map.

See the example below from Geoserver.

• Thumbnail for map
  

• Thumbnail in layer detail page
  

NB: If I am guessing right what could be happening is that for each time a user is creating a map QGIS server is creating a separate QGIS project for that layer and hence generate random colors for the layer as specified in the OTF and hence generating a thumbnail that is different. S

[boney-bun]

thanks for your explanation and hints @NyakudyaA !
Yes, you are right, the problem seems to be on the qgs file used by the map.

I've looked and compared at the qgs files of both map and layer from health layer.
i found something interesting on both qgs files:

the original qgs layer:
<prop k="name" v="star" />

the map qgs:
<prop k="name" v="circle" />

i think the bug is because the map doesn't store a style.
hence, the map qgs doesn't generate a proper style as well.

[gubuntu]

I think we should maybe refactor how a map is stored in QGIS server backend. Am I correct in understanding that it creates a new QGIS project to define a map? (what does it do when you edit the map?)

GeoNode-geoserver stores a map as a list of layers in the database, a bit like a simplified WMC.

Why can we not do the same with QGIS backend?

Or, if we keep the current method of creating a new project, then the layer styles used in the map project must be transferred (copied) from the original layers.

@timlinux, @Gustry your input?

[NyakudyaA]

@boney-bun the scenario you mentioned above is mainly because of what I said. The map creates a new QGIS project which it uses to create a thumbnail.

• When a user saves a map the geonode is creating a new project using OTF. This will have random symbology.
• When a user then previews the created map it uses the old project for the layer hence the icons come out ok.

Solution

• When a user selects a single layer to create a map with. Use the already existing project to generate the thumbnail.
• Establish why the 'create map' needs a new project. If it needs a new project rather make a copy of the existing project for maps consisting of single layers.
• Is the new qgis project for the map only being used for creating the thumbnail or used to reference the map itself?
  NB: If a user is creating a map using multiple layers - how do we extract the symbology for each layer from the existing project since the symbology is saved within the qgis projects

[boney-bun]

@gubuntu yes, you are right. everything in layers and maps should be on project format (qgs) in qgis-server backend. i think this is one of the reasons why we have otf-project.
when we edit a map, the qgs will be updated by otf-project.

cmiiw, here is what happen when a user saves a map:

• geonode calls otf-project for creating a new project (qgs file)
• geonode then calls qgis-server for generating thumbnail by using WMS-GetMap request. the call will use the qgs file that is generated by the otf-project. The bug occurs because the map's qgs doesn't specify the layer the same way as in layer's qgs.

@NyakudyaA yes, it was your hint that helped me going deeper into qgs file. but, your solution seems to be infeasible.

When a user saves a map geonode creates a new project using OTF. This will have random symbology.

the random symbology from the OTF is because we didn't specify the style when generate a new project for the map. this is not the case for generating a new project for a new uploaded layer.

When a user then previews the created map it uses the old project for the layer hence the icons come out ok.

i have tested this with postman. the thumbnail for the map will be correct if we use the layer's qgs. but, we can't simply copy paste the layer's qgs into map's qgs, mostly for the case when a map has more than one layer.

I'm currently interested in modifying QgisServerMap model. It doesn't specify QgisServerStyle as in QgisServerLayer. so, the solution may go into this way: adding the style into QgisServerMap.

what do you think?

[gubuntu]

• if a map view is fetching layers from their original projects, then why do we need new projects for maps? Either
  • we keep map projects for maps, in which case map views and thumbnails must come from that project AND styles in that project must be transferred from the original layer projects.
  • OR we do away with map projects and store the map definition in Django in which case map thumbnails and map views are always composed from the original layer projects.

The only benefit of the former that I can see is that the map layers will be rendered together by QGIS, so will look better than if they are fetched separately by Leaflet and overlaid.

Behaviour should be as close as possible to Geoserver backend, so GeoNode users have a consistent experience.

@boney-bun before diving in, I want to see a clear plan of your approach to this, with at least you, me and @NyakudyaA on the same page.

[boney-bun]

if a map view is fetching layers from their original projects, then why do we need new projects for maps? Either
we keep map projects for maps, in which case map views and thumbnails must come from that project AND styles in that project must be transferred from the original layer projects.

yes, this is what happens now with the current implementation.
going to the second option will have big refactoring consequence.
So, i believe we need to stick with the current implementation.

the bug is because the map projects don't store the style (or symbol) from its original layers.
qgis-server create a new symbol when it doesn't find one.

i suspect that the qgis-server doesn't pick up the style from a layer because the map doesn't have style and default-style definitions in the model.
so, my plan is:

• add style and default-style into map model.
• add QGISServerStyle class into QGISServerMap model (as in QGISServerLayer).
• add a relation between QGISServerMap and Map models.

So far, the QGIS server still doesn't pickup the style.
probably because i still doing it wrong in my local.
I need to understand the way QGIS server works in generating style for layer.

what do you think with this approach @gubuntu @NyakudyaA ?

[gubuntu]

• if the data file has a properly named style file with it, which it should have, then when the map project is created and it loads that layer, it should be styled with that default style. So, first, check why that is not working.
• if there is no default style file, then it needs to be saved as a default style from the layer project and used in the map project.

[gubuntu]

this is not fixed on testing so should not be in the testing column. Moved back to review

[boney-bun]

cmiiw, the original problem can be found on this map:
http://testing.geonode.kartoza.com/maps/30
the map's thumbnail differs from the map.

this PR intends to fix the issue above.
just created a new map in the testing:
http://testing.geonode.kartoza.com/maps/40
the screencast is:

[gubuntu]

try a map with more than one layer, the thumbnail is still wrong

[NyakudyaA]

@boney-bun this is not working at all. Here is a sample layer I have uploaded.

In the map page.

[NyakudyaA]

@boney-bun the following are my observations:

• If you create a map with two or more layers the thumbnail is correct.
• If you create a map with 1 layer the thumbnail is correct.

The following is not working:

• If you update an existing map by adding a new layer you would expect the thumbnail to be regenerated again. Or simply regenerate the thumbnail on update signal regardless of changes in the layers or not. Ie if a user changes the extent of the map do we expect to regenerate thumbnails?

• If you go to a layer with one or more two layers then try to manually update the thumbnail then it is not working as well. Or maybe the thumbnail is cached. Tried to hard refresh the browser but to no avail.

• If you remove the layer from a map with two or more layers then the thumbnail does not update. I think the issue is with the UPDATE button. It does not send any thumbnail generation signal.

• Layers that are not in 4326 are rendering in the wrong place or their extents are wrong and they result in the wrong layout. I think this is covered in a separate issue.

• Why do we have no map backgrounds in thumbnails generated for maps. Are we using different projects or techniques to thumbnail generation in layer detail's page? Or is this the same behaviour from vanilla geonode.

• When a user uploads a new style and associates it with a layer and makes a new style the default. What happens to the thumbnail for maps using that layer. I think they should be updated because the layer itself is updated?