Minhash token filter needs better documentation

[rpedela]

The Minhash Token Filter documentation only describes the interface for the token filter. That is fine for most token filters, but this one is more complicated.

1. It should list possible use cases such as an alternative to the "more like this" query.
2. It should talk about the recommended number of shingles: 5.
3. It should give small but complete examples for 1 and 2.

In the Lucene issue, they discuss Jaccard and cosine similarities. Did that make it into the final patch? If so, should that be exposed as a setting?

[clintongormley]

@rpedela I know nothing about it. Fancy sending a PR with the details?

[pkmital]

Also struggling to use this! Any help would be appreciated.

[romseygeek]

cc @elastic/es-search-aggs

[wayliew]

mark it

[Kukunin]

Just for the next people that might be confused as me, I want to leave the following hint.

I wondered to see the bucket_count parameter for the min_hash filter, despite the official wiki says nothing about it: https://en.wikipedia.org/wiki/MinHash

I found the clue here https://issues.apache.org/jira/browse/LUCENE-6968?focusedCommentId=15263867&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15263867

After a bit more digging, the single hash and keeping the minimum set can be improved.

See:
[1] http://jmlr.org/proceedings/papers/v32/shrivastava14.pdf
[2] http://www.auai.org/uai2014/proceedings/individuals/225.pdf

In summary: rather than keep the minimum set, split the hash space up into 500 buckets (for a 500 hash fingerprint) and keep the minimum for each bucket. To fill an empty bucket, take the minimum from the next non-empty bucket on the right with rotation.

[Vince-Smith]

I'm curious if anyone can provide guidance on how to query these min_hash fields once they've been analyzed. I'm not finding it to be intuitive.

[yaroslav-tykhonchuk]

I'm curious if anyone can provide guidance on how to query these min_hash fields once they've been analyzed. I'm not finding it to be intuitive.

When the field was analyzed using minHash analyzer, you can query the tokens using _termvectors API.

You need to query the document id, noting on which field you want to see terms. The amount of terms your field has, depends on the amount of hash functions, length of your text, to how many shingles you split your text and other parameters.

Then if you will query this field using more_like_this query, elasticsearch will use these terms to find similar documents, which will speed up the process a lot.

Additionally, when using like with documents, either _source must be enabled or the fields must be stored or store term_vector. In order to speed up analysis, it could help to store term vectors at index time.