Show information about "lazy scans" in "Analysis" view #54
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The runtime information shown in the "Analysis" tree now contains additional information, notably such related to "lazy index scans", see PR ad-freiburg/qlever#1003 . The status of the "normal" operations has been renamed to "fully materialized" and is not explicitly shown in the tree. The status of the lazy index scans is called "lazily materialized". When hovering over a node in the tree, more detailed infomation is shown at the bottom if available.
hannahbast
pushed a commit
to ad-freiburg/qlever
that referenced
this pull request
Jul 12, 2023
Under certain conditions, index scans are now lazily materialized (and there is a special status for this in the runtime information, see ad-freiburg/qlever-ui#54). The conditions are: (1) the index scan is the operand of a single-column join operation, (2) the other operand of the join operation contains no UNDEF values in the join column and the query planner knows this, and (3) the number of rows of the index scan, if it were fully materialized, is above the runtime parameter `lazy-index-scan-max-size-materialization`. Lazily materialized then means two things: (1) a subset of the blocks is identified (using the metadata of the blocks) that is sufficient to compute the correct result, and (2) these blocks are produced one after the other and buffered only to a limited extent. There are two reasons for the buffering: (1) it is sometimes necessary to have several blocks in memory at the same time to compute cross-products efficiently, and (2) there is an `OrderedThreadSafeQueue` for the blocks (see #1011 and #1023) with runtime parameters `lazy-index-scan-queue-size` and `lazy-index-scan-num-threads` in order to allow parallel decompression of the blocks and minimize waiting times for the join operation consuming the blocks. For all cases, where the conditions for lazy materialization are not met, the operands of the join are fully materialized, and one of the three existing join implementations is used: the standard zipper join, the galloping join (when the size of one operand is much larger than the other), or the general-purpose join (which can handle multiple columns and UNDEF values).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The runtime information shown in the "Analysis" tree now contains additional information, notably such related to "lazy index scans", see PR ad-freiburg/qlever#1003 .
The status of the "normal" operations has been renamed to "fully materialized" and is not explicitly shown in the tree. The status of the lazy index scans is called "lazily materialized".
When hovering over a node in the tree, more detailed infomation is shown at the bottom if available.