Lucene & limit

[silvanheller]

Lucene currently does not propagate a limit to the underlying IndexSearcher, see:

cottontaildb/cottontaildb-dbms/src/main/kotlin/org/vitrivr/cottontail/dbms/index/lucene/LuceneIndex.kt

Line 481 in ae750e7

private val results = this.searcher.search(this.query, Integer.MAX_VALUE)

This was never noticed because Cineast did not tell cottontail, see: vitrivr/cineast#312

[ppanopticon]

This is intended because a LIMIT expressed in a query to Cottontail DB cannot always be pushed-down to Lucene, e.g., if we use Lucene to satisfy a predicate and then sort the result based on another column that the index does not cover.

The question is: Is the LIMIT respected or is it ignored completely?

[silvanheller]

Limit is respected but not pushed down to the Lucene index, even for simple queries where it would make sense

[lucaro]

So this means the result of the query as a whole is correct, but lucene returns unnecessarily many results at an intermediate stage which have to be discarded afterward?

[silvanheller]

Yes, this is visible in the logs:

2022-05-17 15:45:44 [DefaultDispatcher-worker-2] DEBUG TransactionalGrpcService:114 - [3414, q-cin-ft-rows-4f8] Preparation of Select completed successfully in 32.095876ms.
2022-05-17 15:45:52 [cottontaildb-query-worker-4] DEBUG IndexScanOperator:70 - Read 428892 entries from warren.cineast.features_audiotranscription.idx_feature_lucene.
2022-05-17 15:45:52 [DefaultDispatcher-worker-2] INFO  TransactionalGrpcService:152 - [3414, q-cin-ft-rows-4f8] Execution of Select completed successfully in 8.392559588s.

[ppanopticon]

So this is a performance issue I take it? I'll have a look!

[ppanopticon]

I have made some changes on master. Those do not address the raised issue directly but still iron-out a few glitches, which I suspect to be the culprit.

Could you quickly test, if there is a performance improvement using that version?

[silvanheller]

Much faster, thanks!