GNIP - Add OSM Import and Update Functionality using GeoGig

[jj0hns0n]

Overview/Background

Many GeoNode admins have expressed the desire to include OpenStreetMap data as layers in their GeoNodes so that their users can download country or region sized chunks of OSM in 'layers' that are familiar to them as GIS users. Sites like geofabrik provide this sort of functionality, but not for every country and only with a fixed and inflexible set of layers. Other OSM download mechanisms (some described here ) are cumbersome and tedious and don't let the admin easily update the data as it changes over time.

The basic idea behind this GNIP is to leverage functionality found in the GeoGig project to provide this set of functionality. It is currently possible to use the GeoGig command line utilities to download, import, map and export OSM data to aPostGIS database and then to configure the tables in this database in GeoServer and then in GeoNode using updatelayers or similar. This has been documented This proposal is specifically about using GeoGig's web api another link here which takes the form of a GeoServer extension.

The current geonode.contrib.geogig module really contains only static media, templates and some simple template tags that leverage this web api to read and display information, but provide no functionality to actually control the repositories etc.

NOTE: The GeoSHAPE project already leverages this functionality, but the bulk of the GeoGig operations that interact with OSM are primarily done on using the command line interface. Recent work funded by @state-hiu has added the OSM specific commands to the web api and the functionality proposed in this GNIP would leverage this new functionality.

Proposed By:

@jj0hns0n

Assigned to release:

2.5+ (Should be merged to master after 2.4 final)

State:

For Initial Discussion

Motivation:

Many administrators running GeoNode sites have expressed an interest in adding OpenStreetMap data to their GeoNode site. There are currently several ways to accomplish this, but none of them are straightforward or easy or and they rely on various tools that may or may not be familiar to the administrator.

Having the ability to use only the GeoNode interface (including the admin) to define the OSM data to be downloaded, how it is to be mapped to GeoNode layers and the ability to keep it up to date over time would be significantly easier for many GeoNode admins and would allow them to bootstrap their instance with data very quickly and easily.

Proposal:

It is proposed to expand on the current geonode.contrib.geogig module to provide a way for admin users to create and maintain geogig repos, to configure bounding boxes for data download, manage the mappings between osm nodes/ways and normal GeoNode layers and to manage the update schedule.

Specifically:

• A set of models
  • geonode.contrib.geogig.models.Repository - GeoGig Repositories
  • geonode.contrib.geogig.models.Footprint - Download Bounding Boxes
  • geonode.contrib.geogig.models.Filter - Overpass QL Filters
  • geonode.contrib.geogig.models.Mapping - Node/Way -> Layer mappings
• A set of views providing API endpoints
• A set of tasks providing jobs for asynchronous execution
• A set of integration tests
• A set of templates and client code

In order for GeoNode administrators to use this functionality, they will need to use the GeoGig GeoServer plugin. As such, it will need to be compiled against a version of GeoServer that also supports the GeoNode authentication plugin and instructions provided on how to install and use it.

Use cases:

• As a GeoNode administrator, I want to be able to create an empty repository to store OpenStreetMap data using the GeoNode UI (vs the command line)
• As a GeoNode Administrator, I want to be able to configure the footprint or bounding box for data to be store in the repository using the GeoNode UI (vs the command line)
• As a GeoNode Administrator, I want to be able to configure the mappings used to map OSM nodes and ways to layers using the GeoNode UI (vs the command line)
• As a GeoNode Administrator, I want to be able to configure the schedule of tasks used to update the data from OSM Planet.

Issues:

• GeoGig is a relatively new project and the primary risk associated with using it is that repositories can become corrupted and are then completely useless.
• It is not known how well repos with long history perform from a performance standpoint.
• Others (@garnertb and @state-hiu looking at you)

Testing:

Alternatives

Feedback

External Links

• http://dev.boundlessgeo.com/~groldan/geogig-user-manual/interaction/geoserver_ui.html
• http://dev.boundlessgeo.com/~groldan/geogig-user-manual/interaction/osm-web-api.html

[JJediny]

Think this and other data import methods moving forward should be approached on the client side to distribute processing away from geonode server. OL3 now supports dynamic rendering of OSM-XML http://openlayers.org/en/v3.1.1/examples/vector-osm.html?q=OSM . OSM/GeoJSON/KML are some of the most common formats clients are likely to be comfortable/fimilar with. Hits on the bigger issue of how and if we handle client side rendering / data conversion. Nodejs and Angularjs makes it more feasible to support a more outsourced processing approach - http://naturalatlas.github.io/node-gdal/classes/gdal.html

All of these are multigeometry features (point/line/polygon in one file/stream) need to figure out how this will conform or make us rethink the pure one geometry per layer native to geonode's uploading.

[gamesbook]

The problem with moving all your importing to the client-side is that it
becomes much more difficult to built third-party front-end apps which
leverage GeoNode capabilities on the "back end" (which is the direction we
see things moving). My personal take is that data ingestion/conversion is
a server-side responsibility. If the core team does not want to take on
enhancing GeoNode to handle a gajillion data formats, then perhaps a robust
API with a pluggable/extendable back-end would be the way to approach this?

On 24 January 2015 at 09:15, John Jediny [email protected] wrote:

Think this and other data import methods moving forward should be
approached on the client side to distribute processing away from geonode
server. OL3 now supports dynamic rendering of OSM-XML
http://openlayers.org/en/v3.1.1/examples/vector-osm.html?q=OSM .
OSM/GeoJSON/KML are some of the most common formats clients are likely to
be comfortable/fimilar with. Hits on the bigger issue of how and if we
handle client side rendering / data conversion. Nodejs and Angularjs makes
it more feasible to support a more outsourced processing approach -
http://naturalatlas.github.io/node-gdal/classes/gdal.html

All of these are multigeometry features (point/line/polygon in one
file/stream) need to figure out how this will conform or make us rethink
the pure one geometry per layer native to geonode's uploading.

—
Reply to this email directly or view it on GitHub
#1931 (comment).

[JJediny]

By no means speaking for the team and no doubt its a complex issue. But I think moving forward we need to separate the discussion of big bulk uploads to "Layers" in Geonode from smaller/dynamic types of data. Know I'm broadening this thread way beyond this ticket and OSM could be either type depending on the use; but my comment before was more geared to the later (small/dynamic), with other use-cases/needs that have come up around things like - adhoc user created layers/features for quick map notes OR dynamically rendering a feature service directly to the canvas like with some xml based (kml/osm/georss/etc.) or json based data streams / small single files and being able to "save" all or a selection from that data in geonode... and that importing=/=saving may best be handled separately. As all of the creating/selecting for that kind of data is going to be all client side anyway. Know I'm off-base and should redirect the comment moving forward, but just wanted to clarify where I was coming from

[gamesbook]

@JJediny - that all makes sense. We have also been looking at exactly the same issue of "dynamically rendering a feature service directly to the canvas like with some xml based (kml/osm/georss/etc.) or json based data streams / small single files" but using OL3, not Leaflet. If GeoNode, in some future version, moved to using OL3, I think it could be extended to handle this kind of use-case natively.

My point was more that the community gets more and more fragmented as you move away from what the core provides; so you & I, for example, may end up developing exactly the same kind of functionality for slightly different front-ends ... with no way to incorporate that learning into GeoNode. If we can avoid this,we should. If not, then so be it.

[JJediny]

@gamesbook - there are alot of ongoing discussions on the future map clients for Geonode 2.5+ ; think everyone sees the horizon for Geoexplorer which has served geonode well but is built on older since deprecated libraries. Seems like there are two obvious mapping libraries being played with in leaflet and OL3; but as far as a main mapping framework for geonode that could support modularization so projects could customize functionality but still contribute back so others could plugin or emulate... there is a glaring missing gap (which geoext and opengeo sdk had filled). Currently MapLoom which uses Angularjs/OL3 is the frontrunner, namely because it was developed to work off of geonode as it's backend (using same var/hooks). But alot of work needed to document and turn it into a pluggable framework; personally interested because of it's core geogig support and optimistic we can pull in some useful features from the project it was originally forked from in geoadmin3. But on the other hand it would be great to emulate mapbox's simple geojson editor for adhoc layer creation and notation from leaflet to OL3. Without corporate sponsor or a well funded project coming to the rescue; it's not going be easy to herd the cats as a community alone; but when it comes to map clients think we need to establish some platform for the future so we can all benefit, think the clear first step is to nail down the foundation of the javascript framework and mapping library for my money that's in Angularjs/OL3. Also see related #1331

[gamesbook]

Yes; the Angularjs/OL3 combination is a solid one.

[jj0hns0n]

Thanks for the comments @JJediny and @gamesbook ... but I think the use case hasn't even been defined yet, so the discussion is entirely premature.

The functionality this proposed feature is intended to provide is something like what you see in this GeoNode http://www.masdap.mw/layers/ ... basically country or region sized chunks of OSM where the data has been mapped to layers and is then available for download from the GeoNode and updated regularly from OSM planet. Something along the lines of what Geofabrik provides here http://download.geofabrik.de/ but in a GeoNode and more flexible ways to map OSM nodes/ways into GeoNode layers.

The discussion on mapping clients and backends properly belongs in this GNIP #1331 and was discussed at the sprint.

Note that I dont necessarily think GeoGig is the end all be all here, but it can provide the functionality we are looking for.

So, let me finish the GNIP and then lets discuss :)