Support preclustering filtering

[lucaswoj]

Allow users to filter features before they are clustered. Some users may want this filtering to happen without transferring GeoJSON between the main thread and worker threads.

We can filter these within the GL JS codebase before passing them to supercluster.

continued from mapbox/supercluster#11

cc @mourner @lyzidiamond

[jruddell]

@mourner any progress on this? i'm working with very large data sets and calling .setData every time i want to filter is fairly taxing

[benderlidze]

any updates on this? Thanks.

[robbeman]

This would be more than helpful. Concerning the syntax, would this be something in the likes of?

[
  'all',
  ['match', ['get', 'category'], selectedCategories, true, false],
  // On a side note, I don't know what the true and false at the end do exactly,
  // I just took this from the examples
  ['has', 'point_count']
]

Another side note: the documentation for setFilter links to deprecated syntax https://www.mapbox.com/mapbox-gl-js/api#map#setfilter, but looking at the expression syntax it seems quite similar to me, no?

Would order be important? As in: any filters coming before the "has point count" filter would apply to anything symbol specific and anything after would apply to the cluster?

[csimpi]

I'm trying to revive this:
#7887 (comment)

[crow7m]

Any update on this one ? When working with huge data sets, might be very useful

[csimpi]

@crow7m : Take a look at their answer to my ticket above, it's clear they are not really willing to work on this :(

[ashuga]

I'm also looking forward to updates on this. I've just started using Mapbox and it would be really useful for my specific project.

In the meantime, does anyone have a link to an example of the work around? I'm trying to create filters based on the feature's properties for my clustered map. I see that it involves filtering manually and using setData, but is there an example out there of that?

[mathiaslaramee]

No progress on this for soon 4 years? This would solve a lot of problems performance wise when filtering huge data-sets that are clustered. @mourner

[strech345]

I think with the clusterProperties its now possible.

for the source
"clusterProperties": { "cluster_count": ["+", ["case", <yourFilterExpression>, 1, 0]] }

and for the layer
"filter": ["all", ["==", ["get", "cluster"], true], [">", ["get", "cluster_count"], 0]]

Update:
maybe not :-(
It works so far, but it shows cluster of one item (if there is only one unfiltered item)

Update2:
I now use this filter api. It works to also use the mapbox filter expression
https://github.com/mapbox/mapbox-gl-js/tree/master/src/style-spec/feature_filter
But yes, not nice to store the row geojson

[crow7m]

Hi all, is there any news about this feature state?

[crow7m]

@mourner @lucaswoj hi all, thanks for cool library , question, is there a plan to implement this in near feature ?? This is quite old issue. thank you for your time and great work

[mourner]

Yes, we plan to work on this soon since this is such a highly requested feature — stay tuned for news in the coming weeks!

[jhadenfeldt]

@mourner : Since it's been a few weeks, are there any updates on this feature?

[mcollins-GPI]

@mourner I am also interested in being able to apply filters on sources dynamically.

[csimpi]

still nothing guys? this shouldn't be that hard... we have this bug on our website and apps for a really long time and had to explain all the time we can't fix it cause Mapbox is not willing to fix :/

[diegosys89]

Hi guys! Any aupdate? like I see this can not be implemented, (4 years of requesting).

[crow7m]

@mourner @lucaswoj , hi hope all are doing well and healthy, small question, is there any approximate date for this so cool feature to be available to public? Thank you and stay healthy.

[mourner]

Hey friends, I took a stab at this at #9864 — please let me know if this works for your use case!

It's a minimal implementation, only allowing setting the initial filter when adding the source, but there seems to be workaround for that — e.g. removing and re-adding a source with different options, which should be fine performance-wise since we have to re-process the whole data anyway, and the file by the URL is already cached by the browser.

[crow7m]

Hey friends, I took a stab at this at #9864 — please let me know if this works for your use case!

It's a minimal implementation, only allowing setting the initial filter when adding the source, but there seems to be workaround for that — e.g. removing and re-adding a source with different options, which should be fine performance-wise since we have to re-process the whole data anyway, and the file by the URL is already cached by the browser.

Hi @mourner , thanks for the effort, so just to make sure i got it correct, instead of heavy operations with data, we will be able to pass same data to source and just update the filter in order to get filtered results, and in order for clusters to redraw on the map, correct ? if so i guess it will be better than filtering the whole data set every time we want to filter and redraw clusters. Thanks for your time and effort.

[enersis-pst]

like i understand there is no possibility to update the source filter, right? Only by removing and adding a new source is not a solution for me, because therefore i also need to remove an recreate all depening layers.

[Codain]

I also have to admit that I don't understand what to do in order to implement both clusters and filters... I have the feeling it is still not possible to implement both. Can you please explain how to implement both in order to make sure this issue has been addressed ? The example in the pull request does not help at all. If needed I can create an issue requesting to add an example in Mapbox documention. Thanks.

[enersis-pst]

@Codain
filters could be set once https://docs.mapbox.com/mapbox-gl-js/style-spec/sources/#geojson-filter, but coudnt be updated after source was set. That was the function i was missing.
It might be possible by storing unfiltered features and process them if filter on source is set.

[angelabelle]

For some reason the current solution did not work when using a feature property for filtering. I used this as a workaround. Giving credits here: https://gis.stackexchange.com/questions/276268/filter-clustering-mapbox-gl-js

This is my code right before the cluster layers are added, where my dataset, analogFeatures, is filtered by a feature property:

    analogFeatures.features = analogFeatures.features.filter(feature => feature.properties.flagged === true);

    map.addSource('ana-data', {
      'type': 'geojson',
      'data': analogFeatures,
      'cluster': true,
      'clusterMaxZoom': 14, // Max zoom to cluster points on
      'clusterRadius': 20 // Radius of each cluster when clustering points (defaults to 50)
    });

I used this example for clustering https://docs.mapbox.com/mapbox-gl-js/example/cluster/, and it works how I want it to.

Hope this helps someone!

[mourner]

@angelabelle you could just do "filter": ["==", ["get", "flagged"], true] in source options. This feature was added in #9864.

[mathiaslaramee]

Still doesn't let us filter the data after the source has been set initially which essentially forces us to setData, which sucks, everytime we want to filter cluster-data on the fly :(

[enersis-pst]

for me it works by using setStyle with the new filter on source.

[tmlmt]

for me it works by using setStyle with the new filter on source.

@enersis-pst could you elaborate a bit with a code snippet? Did you manage to filter cluster-data without using setData with filtered features?

[enersis-pst]

@tmlmt
here is a example

on hover you see first only one feature inside the circle.
after click on the map there are two inside the circle.
https://jsfiddle.net/swu5a7kj/5/

[tmlmt]

Thanks a lot, @enersis-pst! I took the liberty to make a fork which shows a bit more visually what happens on-click, without the verbose console output. Just keep clicking and it will switch back and forth between points A or B+C being shown. https://jsfiddle.net/4fao60cu/3/

In the meantime I managed to make cluster filtering work as suggested above by filtering the source and using getSource().setData() and it's surprisingly very smooth (I have few points, ~300). I wonder if a benchmark could be done between yours and this method should be done; but it might be irrelevant, as both are about redoing some kind of re-rendering anyway, they may be similarly performant. If anyone can chip in?

Thanks all for the above comments.