Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental feature to merge multiple OSM layers together #380

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

craigtaverner
Copy link
Contributor

@craigtaverner craigtaverner commented Apr 19, 2021

The idea here is that if we have imported multiple overlapping chunks of OSM data, it should be possible to stitch them together. The process does as follows:

  • Identify the two layers (two OSM models), with one designated as the layer to merge into. The other layer will be removed from the index.
  • Call spatial.merge.into(main,other)
  • This will perform the following steps:
    • iterate over all OSMNode nodes in other - for each check if there is a matching node in main and update to other properties (if the node moves, record that fact for re-indexing). If there is no matching node, re-label the node to the new layer (moving it in internal indexes). All nodes with geometry nodes attached are recorded for later re-indexing
    • iterate over all OSMWay nodes in other - for each check if there is a matching node in main and if not move the layer by re-labeling. If there is a matching way, then do a node level merge by finding common nodes and re-sorting.
    • repeat this process for OSMRelation too
    • If re-indexing is required, remove all main nodes from the RTree index and run re-indexing

craigtaverner and others added 12 commits April 10, 2021 23:01
This includes tests that pass for various combinations of simple point layers, but should also work for any layers that have property only geometry encoders.

There are failing tests for merging into OSM, which should not be supported, and for merging one OSM into another, which we hope to support in the next update.

In fact, this commit is preparatory work for OSM-OSM merging capabilities, which are planned, but not implemented yet.
This required some refactoring of the OSM model with changes to OSMImporter, OSMLayer and OSMDataset.
In addition we added OSMModel for collecting general contants and functions appropriate to OSM, and we added OSMMerger with the intelligence to merge two OSM datasets together.

This is currently only tested in a trivial case, which is two identical models with one trivial change (added a free floating OSM node).

One change to the OSM model is that we moved User nodes to a linked list like we do for Ways and Relations.
This involved a big change to the OSM model. The OSM model uses less linked lists and more label-property indexes, in particular User, Changset, Way and Node are all now accessible via hashed labels and their OSM ids, using schema indexes. And all have been removed from their layer linked lists, except the OSMNode which remains in the OSMWay->OSMWayNode->OSMNode tree/chain.

The OSMImporters LabelHasher was moved to the OSMDataset to be shared with the OSMGeometryEncoder, bringing the imported model and the CRUD model closer.

The OSMGeometryEncoder now can generate new fake OSM ids for all supported types (relation, way, node) saving the max value in the layer node so restarts will not re-use ids. The fake ids are all negative. This allows, in particular, an imported OSM dataset to have new data added through the normal layer interface without clashing OSM ids (real are positive, and fake are negative).
We made the code more verbose for now, but we could simplify to simply not care about missing geometries.
# Conflicts:
#	src/main/java/org/neo4j/gis/spatial/AbstractGeometryEncoder.java
#	src/main/java/org/neo4j/gis/spatial/DefaultLayer.java
#	src/main/java/org/neo4j/gis/spatial/GeometryEncoder.java
#	src/main/java/org/neo4j/gis/spatial/SpatialTopologyUtils.java
#	src/main/java/org/neo4j/gis/spatial/osm/OSMDataset.java
#	src/main/java/org/neo4j/gis/spatial/osm/OSMGeometryEncoder.java
#	src/main/java/org/neo4j/gis/spatial/osm/OSMImporter.java
#	src/main/java/org/neo4j/gis/spatial/osm/OSMLayer.java
#	src/main/java/org/neo4j/gis/spatial/osm/OSMRelation.java
#	src/main/java/org/neo4j/gis/spatial/osm/RoadDirection.java
#	src/main/java/org/neo4j/gis/spatial/procedures/SpatialProcedures.java
#	src/main/java/org/neo4j/gis/spatial/rtree/RTreeIndex.java
#	src/test/java/org/neo4j/gis/spatial/LayersTest.java
#	src/test/java/org/neo4j/gis/spatial/OsmAnalysisTest.java
#	src/test/java/org/neo4j/gis/spatial/TestOSMImport.java
#	src/test/java/org/neo4j/gis/spatial/TestSpatial.java
#	src/test/java/org/neo4j/gis/spatial/pipes/GeoPipesDocTest.java
#	src/test/java/org/neo4j/gis/spatial/procedures/SpatialProceduresTest.java
Copy link

github-actions bot commented May 29, 2024

Test Results

306 tests  +16   306 ✅ +16   9m 51s ⏱️ +16s
 33 suites + 1     0 💤 ± 0 
 33 files   + 1     0 ❌ ± 0 

Results for commit 595c92f. ± Comparison against base commit 12f6d46.

♻️ This comment has been updated with latest results.

# Conflicts:
#	src/test/java/org/neo4j/gis/spatial/procedures/SpatialProceduresTest.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants