Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide framework for generic lazily evaluated operation results #1350

Merged
merged 152 commits into from
Aug 22, 2024

Conversation

RobinTF
Copy link
Collaborator

@RobinTF RobinTF commented May 18, 2024

QLever now supports lazy operations, that can produce their result in batches instead of fully materializing them. This PR implements the general infrastructure needed for lazy evaluation, and provides a lazy implementation of the IndexScan and Filter operation. This means that queries that consist of a single triple and one or more subsequent FILTER clauses will not materialize the full index scan.

src/engine/Operation.cpp Outdated Show resolved Hide resolved
Copy link

codecov bot commented May 18, 2024

Codecov Report

Attention: Patch coverage is 95.61905% with 23 lines in your changes missing coverage. Please review.

Project coverage is 89.45%. Comparing base (5007401) to head (0437773).
Report is 3 commits behind head on master.

Files Patch % Lines
src/engine/ExportQueryExecutionTrees.cpp 93.33% 0 Missing and 9 partials ⚠️
src/util/ConcurrentCache.h 76.92% 0 Missing and 9 partials ⚠️
src/util/CacheableGenerator.h 63.63% 0 Missing and 4 partials ⚠️
src/engine/Operation.cpp 99.09% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1350      +/-   ##
==========================================
+ Coverage   89.37%   89.45%   +0.07%     
==========================================
  Files         345      346       +1     
  Lines       24886    25262     +376     
  Branches     3309     3371      +62     
==========================================
+ Hits        22242    22597     +355     
- Misses       1503     1505       +2     
- Partials     1141     1160      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

joka921 pushed a commit that referenced this pull request May 23, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
This PR contains all the changes from the infrastructure for lazy operation evaluation (#1350)  that are simple and repetitive, but touch many files. In particular:

* Rename the `ResultTable` class to `Result` (a TODO suggested by @hannahbast some time ago).
* Add a new parameter `bool requestLaziness` to `Operation::computeResult`. This parameter is currently unused.
RobinTF added 25 commits May 23, 2024 16:26

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.
RobinTF added 17 commits August 15, 2024 22:31

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.
…t-table

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.
Copy link
Member

@joka921 joka921 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A round of reviews on everything but the tests (those I will review later).

const IdTable& idTable_;
std::ranges::iota_view<uint64_t, uint64_t> view_;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need the original absolute indices into the IdTable?
Otherwise you could return a single
std::ranges::subrange<const IdTable&>
and all your loops become singular again, and all the sonarcloud stuff goes away?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need the values as well as a reference to the original IdTable to fill in the ConstructQueryExportContext

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be possible however, to refactor ConstructQueryExportContext to get some row reference instead, but this might be better suited for a follow-up PR.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about it it doesn't solve the original issue. yielding subranges still requires 2 loops

src/engine/Filter.cpp Outdated Show resolved Hide resolved
src/engine/Operation.h Show resolved Hide resolved
src/engine/Operation.h Outdated Show resolved Hide resolved
src/engine/Operation.h Outdated Show resolved Hide resolved
src/engine/Operation.cpp Outdated Show resolved Hide resolved
src/engine/Result.cpp Outdated Show resolved Hide resolved
src/engine/Result.cpp Show resolved Hide resolved
src/engine/Result.cpp Show resolved Hide resolved
src/engine/Result.h Outdated Show resolved Hide resolved

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.
Copy link
Member

@joka921 joka921 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please please please fix the large amount of copy and paste code in your tests.
I have made several suggestions for this endeavour.

test/ConcurrentCacheTest.cpp Show resolved Hide resolved
test/ConcurrentCacheTest.cpp Show resolved Hide resolved
test/ConcurrentCacheTest.cpp Outdated Show resolved Hide resolved
test/ExportQueryExecutionTreesTest.cpp Outdated Show resolved Hide resolved
test/ExportQueryExecutionTreesTest.cpp Outdated Show resolved Hide resolved
test/ResultTest.cpp Show resolved Hide resolved
test/ResultTest.cpp Outdated Show resolved Hide resolved
test/ResultTest.cpp Outdated Show resolved Hide resolved
test/ResultTest.cpp Outdated Show resolved Hide resolved
test/ResultTest.cpp Outdated Show resolved Hide resolved

Verified

This commit was signed with the committer’s verified signature.
Copy link
Member

@joka921 joka921 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already much simpler,
but I have identified some additional simplifications.

test/ConcurrentCacheTest.cpp Outdated Show resolved Hide resolved
test/ExportQueryExecutionTreesTest.cpp Outdated Show resolved Hide resolved
test/ExportQueryExecutionTreesTest.cpp Show resolved Hide resolved
EXPECT_EQ(iterator, generator.end());
auto referenceTable = makeIdTableFromVector({{2}});
EXPECT_THAT(convertToVector(std::move(generator)),
ElementsAre(Eq(std::cref(referenceTable))));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Eq is redundant/implicit`.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From https://google.github.io/googletest/reference/matchers.html

WARNING: Equality matching via EXPECT_THAT(actual_value, expected_value) is supported, however note that implicit conversions can cause surprising results. For example, EXPECT_THAT(some_bool, "some string") will compile and may pass unintentionally.

Also the Eq seems to be required for the cref stuff to work otherwise it tries to copy the IdTable

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you say the cref(...) without Eq() doesn't also work?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly. No idea why but I just tried it to confirm

test/ExportQueryExecutionTreesTest.cpp Show resolved Hide resolved
test/OperationTest.cpp Outdated Show resolved Hide resolved
test/ResultTest.cpp Show resolved Hide resolved
test/ResultTest.cpp Outdated Show resolved Hide resolved
test/util/OperationTestHelpers.h Outdated Show resolved Hide resolved
test/util/OperationTestHelpers.h Outdated Show resolved Hide resolved

Verified

This commit was signed with the committer’s verified signature.

Verified

This commit was signed with the committer’s verified signature.
…t-table

Verified

This commit was signed with the committer’s verified signature.
…t-table
Copy link
Member

@joka921 joka921 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really nice and is now running on our public endpoint.
I will run some tests tomorrow (but it already looks very nice to look at the incremental runtime information).

Copy link

@joka921 joka921 merged commit f986808 into ad-freiburg:master Aug 22, 2024
20 checks passed
@RobinTF RobinTF deleted the refactor-result-table branch August 22, 2024 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants