Unable to replicate output of tags-filter command

[fbarbe00]

I have been trying to replicate the output of the following bash command with pyosmium:

osmium tags-filter osm_data/aachen.osm.pbf \
w/highway w/public_transport=platform w/railway=platform w/park_ride r/type=restriction \
-o aachen-filtered2.osm.pbf

Here's my python code:

import osmium

# Define the filters for the desired tags
filters = [
    osmium.filter.KeyFilter('highway').enable_for(osmium.osm.WAY),
    osmium.filter.TagFilter(('public_transport', 'platform')).enable_for(osmium.osm.WAY),
    osmium.filter.TagFilter(('railway', 'platform')).enable_for(osmium.osm.WAY),
    osmium.filter.KeyFilter('park_ride').enable_for(osmium.osm.WAY),
    osmium.filter.TagFilter(('type', 'restriction')).enable_for(osmium.osm.RELATION)
]

# Apply filters using a ForwardReferenceWriter to maintain referential completeness
writer = osmium.ForwardReferenceWriter('aachen-filtered.osm.pbf', ref_src='osm_data/aachen.osm.pbf', overwrite=True)

fp = osmium.FileProcessor('osm_data/aachen.osm.pbf')

# Add filters to the file processor
for f in filters:
  fp = fp.with_filter(f)

# Process the file and write filtered data
with writer:
  for obj in fp:
      writer.add(obj)

I would have expected these two to result in the same output file, but that does not seem to be the case.
Here's a comparison of the osmium check-refs command for both files:

# bash command
There are 49207 nodes, 14943 ways, and 272 relations in this file.
Nodes in ways missing: 0

# python code
There are 293587 nodes, 59950 ways, and 1035 relations in this file.
Nodes in ways missing: 0

Could anyone provide an explaination for the difference, to make a working python version of the bash command?

[fbarbe00]

For reference, the osm file I tried this with comes from here: https://github.com/motis-project/test-data/raw/aachen/aachen.osm.pbf

[lonvia]

Thanks for trying out the new filters.

Your filters are not quite right. When you chain filters, then they applied one after the other. That effectively is a conjunction of conditions. You are thus filtering for ways with highway and public_transport=platform and railway=platform and park_ride tags. You can do or-filtering of tags or keys by adding multiple parameters to KeyFilter() or TagFilter(). However, it is currently not possible to or-chain filters. So if you have a mix of keys and tags you want to filter for, then you need to pre-filter by tag keys only and then further filter for the appropriate key-value combinations in the reader loop.

The other problem with the filters is that all of them are restricted to either ways or relations. That means that all nodes will pass your filter chain. They will all end up in your output file. If you are only interested in ways and relations, then use the EntityFilter to restrict to these types first.

Finally, there is the tricky part regarding referential completeness. To get the members and nodes of ways and relations, you need the backward references, not the forward references. See the user manual for an attempt to define these terms.

Putting it all together, this is what the script should look like:

# Define the filters for the desired tags
filters = [
    osmium.filter.EntityFilter(osmium.osm.WAY | osmium.osm.RELATION),
    osmium.filter.KeyFilter('highway', 'public_transport', 'railway', 'park_ride').enable_for(osmium.osm.WAY),
    osmium.filter.TagFilter(('type', 'restriction')).enable_for(osmium.osm.RELATION)
]

# Apply filters using a BackReferenceWriter to maintain referential completeness
writer = osmium.BackReferenceWriter(outfile, ref_src=infile, overwrite=True)

fp = osmium.FileProcessor(infile)

# Add filters to the file processor
for f in filters:
  fp = fp.with_filter(f)

# Process the file and write filtered data
with writer:
  for obj in fp:
    if obj.is_relation() or\
       'highway' in obj.tags\
       or 'park_ride' in obj.tags\
       or obj.tags.get('public_transport') == 'platform' \
       or obj.tags.get('railway') == 'platform':
      writer.add(obj)

For additional information on tag filtering with pyosmium, have a look at the tag filter cookbook. It gives some background on what is happening here.

[fbarbe00]

Thank you so incredibly much, this has been giving me a headache for the past few days. I was suspecting it was an AND filter instead of OR, but was confused as to why there were more resulting nodes in the end :D. This now all makes sense!