Lazy materialization of index scans #1003
Conversation
Also several refactorings for the IndexScan class.
The `ScanType` is now completely unused inside the `IndexScan` class.
coroutine for scanning.
permutation-templating out of the code. TODO: Many functions can now be moved into the IndexImpl.cpp
_this to that_ in the IndexImpl class.
Next step: actually implement the logic in the JOIN class.
removed some unused functions to make Codecov happ(y | ier).
# Conflicts: # src/index/IndexImpl.Text.cpp
# Conflicts: # src/engine/GroupBy.cpp # src/engine/IndexScan.cpp # src/engine/IndexScan.h # src/engine/Join.cpp # src/index/Index.cpp # src/index/IndexImpl.Text.cpp # src/index/IndexImpl.cpp # src/index/IndexImpl.h # src/index/Permutations.h # test/IndexTest.cpp
block joining function, We yet need to write unit tests for it.
|
Under certain conditions, index scans are now lazily materialized (and there is a special status for this in the runtime information, see ad-freiburg/qlever-ui#54). The conditions are: (1) the index scan is the operand of a single-column join operation, (2) the other operand of the join operation contains no UNDEF values in the join column and the query planner knows this, and (3) the number of rows of the index scan, if it were fully materialized, is above the runtime parameter Lazily materialized then means two things: (1) a subset of the blocks is identified that is sufficient to compute the correct result, and (2) these blocks are produced one after the other and buffered only to a limited extent. There are two reasons for the buffering: (1) it is sometimes necessary to have several blocks in memory at the same time to compute cross-products efficiently, and (2) there is an For all cases, where the conditions for lazy materialization are not met, one of the three existing join implementations is used: the standard zipper join, the galloping join, or the general-purpose join (which can handle multiple columns and UNDEF values). |
# Conflicts: # src/util/ThreadSafeQueue.h # test/CMakeLists.txt # test/ThreadSafeQueueTest.cpp
… have the test coverage for the thread safe queue as well).
# Conflicts: # src/util/ThreadSafeQueue.h # test/ThreadSafeQueueTest.cpp
fixed very minor stuff.
# Conflicts: # test/CMakeLists.txt
# Conflicts: # src/util/ThreadSafeQueue.h # test/ThreadSafeQueueTest.cpp
hannahbast
changed the title
|
Kudos, SonarCloud Quality Gate passed!
|
This is not yet quite done and will be split up into several
preparation PRs.