A project for scalable hierachical clustering, thanks to a Flexible, Incremental, Scalable, Hierarchical Density-Based Clustering algorithms (FISHDBC, for the friends).
This package lets you use an arbitrary dissimilarity function you write (or reuse from somebody else's work!) to cluster your data.
Please see the paper at https://arxiv.org/abs/1910.07283
- Python 3
- Cython
- hdbscan: https://github.com/scikit-learn-contrib/hdbscan
- scipy: https://www.scipy.org/
python3 setup.py install
There are plenty of configuration options, inherited by HNSWs and HDBSCAN, but the only compulsory argument is a dissimilarity function between arbitrary data elements:
import flexible_clustering
clusterer = flexible_clustering.FISHDBC(my_dissimilarity)
for elem in my_data:
clusterer.add(elem)
labels, probs, stabilities, condensed_tree, slt, mst = clusterer.cluster()
for elem in some_new_data: # support cheap incremental clustering
clusterer.add(elem)
# new clustering according to the newly available data
labels, probs, stabilities, condensed_tree, slt, mst = clusterer.cluster()
Make sure to run everything from outside the source directory, to avoid confusing Python path.
As documented in the HDBSCAN source code:
- labels : ndarray, shape (n_samples, )
- Cluster labels for each point. Noisy samples are given the label -1.
- probabilities : ndarray, shape (n_samples, )
- Cluster membership strengths for each point. Noisy samples are assigned 0.
- cluster_persistence : array, shape (n_clusters, )
- A score of how persistent each cluster is. A score of 1.0 represents a perfectly stable cluster that persists over all distance scales, while a score of 0.0 represents a perfectly ephemeral cluster. These scores can be guage the relative coherence of the clusters output by the algorithm.
- condensed_tree : record array
- The condensed cluster hierarchy used to generate clusters.
- single_linkage_tree : ndarray, shape (n_samples - 1, 4)
- The single linkage tree produced during clustering in scipy hierarchical clustering format (see http://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html).
- min_spanning_tree : ndarray, shape (n_samples - 1, 3)
- The minimum spanning as an edgelist.
Look at the fishdbc_example.py file for something more (it requires matplotlib to be run).
Send me an email at [email protected]. I'll improve the docs as and if people use this.
Matteo Dell'Amico
BSD 3-clause; see the LICENSE file.