Make (Ordered)ThreadSafeQueue safer und easier to use
#1023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #1023 +/- ##
==========================================
+ Coverage 75.17% 75.81% +0.64%
==========================================
Files 255 262 +7
Lines 23829 24470 +641
Branches 2943 3007 +64
==========================================
+ Hits 17913 18552 +639
+ Misses 4735 4730 -5
- Partials 1181 1188 +7
☔ View full report in Codecov by Sentry. |
hannahbast
approved these changes
Jul 7, 2023
hannahbast
requested changes
Jul 11, 2023
hannahbast
approved these changes
Jul 11, 2023
|
Make |
hannahbast
changed the title
finish method of ThreadSafeQueue noexcept(Ordered)ThreadSafeQueue safer und easier to use
|
Kudos, SonarCloud Quality Gate passed!
|
hannahbast
pushed a commit
that referenced
this pull request
Jul 12, 2023
Under certain conditions, index scans are now lazily materialized (and there is a special status for this in the runtime information, see ad-freiburg/qlever-ui#54). The conditions are: (1) the index scan is the operand of a single-column join operation, (2) the other operand of the join operation contains no UNDEF values in the join column and the query planner knows this, and (3) the number of rows of the index scan, if it were fully materialized, is above the runtime parameter `lazy-index-scan-max-size-materialization`. Lazily materialized then means two things: (1) a subset of the blocks is identified (using the metadata of the blocks) that is sufficient to compute the correct result, and (2) these blocks are produced one after the other and buffered only to a limited extent. There are two reasons for the buffering: (1) it is sometimes necessary to have several blocks in memory at the same time to compute cross-products efficiently, and (2) there is an `OrderedThreadSafeQueue` for the blocks (see #1011 and #1023) with runtime parameters `lazy-index-scan-queue-size` and `lazy-index-scan-num-threads` in order to allow parallel decompression of the blocks and minimize waiting times for the join operation consuming the blocks. For all cases, where the conditions for lazy materialization are not met, the operands of the join are fully materialized, and one of the three existing join implementations is used: the standard zipper join, the galloping join (when the size of one operand is much larger than the other), or the general-purpose join (which can handle multiple columns and UNDEF values).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows for easier lifetime management of the queue.