Code

Lazy materialization of index scans#1003

Code

Merged

hannahbast merged 179 commits intoad-freiburg:masterad-freiburg/qlever:masterfrom joka921:join_with_scanjoka921/QLever:join_with_scanCopy head branch name to clipboardJul 12, 2023

+2,557-487

Commits

Commits on Apr 25, 2023

• A function to merge a join column with a block.

  Show description for 84070c9
  joka921

  committed
  84070c9

  Copy full SHA for 84070c9

• Continued the refactoring.

  Show description for 06225fb
  joka921

  committed
  06225fb

  Copy full SHA for 06225fb

• Implemented a (serial)

  Show description for 5e5352b
  joka921

  committed
  5e5352b

  Copy full SHA for 5e5352b

Commits on Apr 28, 2023

• In the middle of something not yet working.

  joka921

  committed
  620231a

  Copy full SHA for 620231a

Commits on May 4, 2023

• In the middle of everything, first clean up the permutation business.

  joka921

  committed
  f2bc0ce

  Copy full SHA for f2bc0ce

• Changing everything to a single class of MetaData.

  joka921

  committed
  efe72a7

  Copy full SHA for efe72a7

• Also delete some additional unused Code.

  joka921

  committed
  6ffc83e

  Copy full SHA for 6ffc83e

• Completely removed the

  Show description for 975c59a
  joka921

  committed
  975c59a

  Copy full SHA for 975c59a

• Moved many functions from IndexImpl.h to IndexImpl.cpp

  joka921

  committed
  31883bc

  Copy full SHA for 31883bc

• Added some unit tests.

  joka921

  committed
  e8388e6

  Copy full SHA for e8388e6

• Changed the members from

  Show description for c86d7d0
  joka921

  committed
  c86d7d0

  Copy full SHA for c86d7d0

• Fixed some code smells.

  joka921

  committed
  6079437

  Copy full SHA for 6079437

• Merge branch 'no_templated_permutations' into join_with_scan

  joka921

  committed
  cacf92d

  Copy full SHA for cacf92d

• Continuing after merge.

  joka921

  committed
  75897aa

  Copy full SHA for 75897aa

• Implemented the IndexScan::lazyScansForJoin method.

  Show description for 16b45a5
  joka921

  committed
  16b45a5

  Copy full SHA for 16b45a5

Commits on May 5, 2023

• Removed some code smells.

  joka921

  committed
  b7ee48d

  Copy full SHA for b7ee48d

• Started to write some tests.

  joka921

  committed
  a25914b

  Copy full SHA for a25914b

• Merge branch 'master' into no_templated_permutations

  joka921

  committed
  1c6db64

  Copy full SHA for 1c6db64

• Added some unit tests and

  Show description for e87b15c
  joka921

  committed
  e87b15c

  Copy full SHA for e87b15c

• Merge branch 'master' into no_templated_permutations

  Show description for a5aae12
  joka921

  committed
  a5aae12

  Copy full SHA for a5aae12

• Merge in the updated master.

  joka921

  committed
  f0cbb21

  Copy full SHA for f0cbb21

• Merge branch 'no_templated_permutations' into join_with_scan

  joka921

  committed
  6509d53

  Copy full SHA for 6509d53

• In the middle of figuring stuff out.

  joka921

  committed
  a5bd352

  Copy full SHA for a5bd352

Commits on May 25, 2023

• I am beginning to understand

  joka921

  committed
  2a42cfa

  Copy full SHA for 2a42cfa

Commits on May 26, 2023

• Merge branch 'master' into join_with_scan

  Show description for 3ae1055
  joka921

  committed
  3ae1055

  Copy full SHA for 3ae1055

• Fixed the merge and compilation and tests

  joka921

  committed
  96c3829

  Copy full SHA for 96c3829

Commits on May 28, 2023

• In the middle of everything

  joka921

  committed
  b138441

  Copy full SHA for b138441

Commits on Jun 9, 2023

• Continuing some work on this.

  joka921

  committed
  d1de8ab

  Copy full SHA for d1de8ab

Commits on Jun 12, 2023

• A first draft of the

  Show description for d70fe89
  joka921

  committed
  d70fe89

  Copy full SHA for d70fe89

• The code now passes some unit tests.

  joka921

  committed
  ecde3a6

  Copy full SHA for ecde3a6

• Fixed some more unit tests.

  joka921

  committed
  79f291b

  Copy full SHA for 79f291b

• More unit tests run through, but there's still some stuff that I don't understand.

  joka921

  committed
  1ba3c79

  Copy full SHA for 1ba3c79

• Delete a comment.

  joka921

  committed
  712eb52

  Copy full SHA for 712eb52

Commits on Jun 13, 2023

• The getBlocksForJoin method is semantically wrong.

  joka921

  committed
  8df3277

  Copy full SHA for 8df3277

• Currently not yet working, but we are getting there.

  joka921

  committed
  76f7320

  Copy full SHA for 76f7320

Commits on Jun 14, 2023

• The first version seems to work (joining two index scans)

  joka921

  committed
  3b9bbee

  Copy full SHA for 3b9bbee

• The tests compile again, but they fail. TODO<joka921> handle that stuff.

  joka921

  committed
  c171d6d

  Copy full SHA for c171d6d

• Only unimportant unit tests fail now.

  Show description for f826526
  joka921

  committed
  f826526

  Copy full SHA for f826526

• Started to cleanup the central routine.

  joka921

  committed
  8928684

  Copy full SHA for 8928684

• Several additional cleanups.

  joka921

  committed
  551d567

  Copy full SHA for 551d567

Commits on Jun 15, 2023

• Completely cleaned up the JoinAlgorithms.h and the corresponding tests.

  joka921

  committed
  8006784

  Copy full SHA for 8006784

• Merge branch 'master' into join_with_scan

  Show description for f53b49f
  joka921

  committed
  f53b49f

  Copy full SHA for f53b49f

• It compiles and the tests run.

  Show description for c98c5b3
  joka921

  committed
  c98c5b3

  Copy full SHA for c98c5b3

• Several simplifications and moved a lot of stuff into the Permutation.cpp

  joka921

  committed
  520c98f

  Copy full SHA for 520c98f

• A lot of additional cleanups.

  joka921

  committed
  83aec3b

  Copy full SHA for 83aec3b

• Addressed several sonar and other issues.

  joka921

  committed
  4afd602

  Copy full SHA for 4afd602

• Start the threadsafe ordered queue and don't materialize unneeded results

  joka921

  committed
  1ca7406

  Copy full SHA for 1ca7406

Commits on Jun 16, 2023

• Fix the merge

  joka921

  committed
  5abe837

  Copy full SHA for 5abe837

Commits on Jun 19, 2023

• Added declaration of needed function

  joka921

  committed
  f816eab

  Copy full SHA for f816eab

• Merge branch 'master' into only_small_scans_when_query_planning

  Show description for a615359
  joka921

  committed
  a615359

  Copy full SHA for a615359

• Merge branch 'refactor_index_scan' into only_small_scans_when_query_planning

  joka921

  committed
  c3ab80a

  Copy full SHA for c3ab80a

• Do not read the index scans during query planning.

  joka921

  committed
  7284272

  Copy full SHA for 7284272

• Remove some code smells and eliminate some code duplication.

  joka921

  committed
  cd9f98c

  Copy full SHA for cd9f98c

• Fixed the format.

  joka921

  committed
  7b59e3f

  Copy full SHA for 7b59e3f

• Merge branch 'master' into refactor_index_scan

  joka921

  committed
  32601bc

  Copy full SHA for 32601bc

• Changes from a review with Hannah.

  joka921

  committed
  feb0117

  Copy full SHA for feb0117

• Merge branch 'refactor_index_scan' into only_small_scans_when_query_planning

  Show description for 2cedbc9
  joka921

  committed
  2cedbc9

  Copy full SHA for 2cedbc9

• Merged in the approved version of the other PR.

  joka921

  committed
  b76d825

  Copy full SHA for b76d825

Commits on Jun 20, 2023

• Merge branch 'master' into only_small_scans_when_query_planning

  Show description for 25538af
  joka921

  committed
  25538af

  Copy full SHA for 25538af

• Merge branch 'only_small_scans_when_query_planning' into join_with_scan

  Show description for c621c7d
  joka921

  committed
  c621c7d

  Copy full SHA for c621c7d

• Merged in the lattest other PR.

  joka921

  committed
  9a30097

  Copy full SHA for 9a30097

• Clang format.

  joka921

  committed
  a40bab6

  Copy full SHA for a40bab6

• Very minimal changes.

  joka921

  committed
  dffe587

  Copy full SHA for dffe587

Commits on Jun 21, 2023

• Fix two subtle bugs.

  joka921

  committed
  c43f66a

  Copy full SHA for c43f66a

• Parallel reading of the blocks.

  Show description for b72666e
  joka921

  committed
  b72666e

  Copy full SHA for b72666e

• Merge branch 'master' into join_with_scan

  Show description for 245a87a
  joka921

  committed
  245a87a

  Copy full SHA for 245a87a

• Merged in the master and formatted everything.

  joka921

  committed
  e67d818

  Copy full SHA for e67d818

• Added tests for the threadsafe queue.

  Show description for 557e719
  joka921

  committed
  557e719

  Copy full SHA for 557e719

• Added tests for the threadsafe queue and fixed the tests for now because it's annoying to always have them fail.

  joka921

  committed
  a10c84d

  Copy full SHA for a10c84d

• Fixed warnings

  joka921

  committed
  c07e8d4

  Copy full SHA for c07e8d4

Commits on Jun 22, 2023

• Fixed several bugs, this should now work.

  joka921

  committed
  92ac4d9

  Copy full SHA for 92ac4d9

• Commented the tests for the threadsafe queue.

  joka921

  committed
  35bbf4f

  Copy full SHA for 35bbf4f

Commits on Jun 26, 2023

• Fixed some tests

  joka921

  committed
  79ad8ca

  Copy full SHA for 79ad8ca

• Implement an ordered threadsafe queue and test it.

  joka921

  committed
  82adab5

  Copy full SHA for 82adab5

• Got rid of some code smells in the threadsafe queue.

  joka921

  committed
  00ea579

  Copy full SHA for 00ea579

• Got rid of a lot of code duplication

  joka921

  committed
  723c043

  Copy full SHA for 723c043

• Use 10 blocks.

  joka921

  committed
  0846369

  Copy full SHA for 0846369

• Make the tests more robust for the threadsafe queue.

  joka921

  committed
  31e2c50

  Copy full SHA for 31e2c50

• Make the tests pass consistently.

  joka921

  committed
  ce58398

  Copy full SHA for ce58398

• Small changes

  joka921

  committed
  519d797

  Copy full SHA for 519d797

Commits on Jun 27, 2023

• Fixed the error from yesterday.

  joka921

  committed
  b6e94ee

  Copy full SHA for b6e94ee

• Merge branch 'master' into join_with_scan

  Show description for 736b490
  joka921

  committed
  736b490

  Copy full SHA for 736b490

• Simplified the interface.

  joka921

  committed
  b164415

  Copy full SHA for b164415

• Make it possible to only get the result from an operation if it can be read from the cache.

  joka921

  committed
  3e2a00f

  Copy full SHA for 3e2a00f

• Add comments.

  joka921

  committed
  64b6652

  Copy full SHA for 64b6652

• Fix the build.

  joka921

  committed
  54e4444

  Copy full SHA for 54e4444

• Hopefully fix a bug on mac, and addressed some code smells.

  joka921

  committed
  50741d0

  Copy full SHA for 50741d0

• Make the tests more robust.

  joka921

  committed
  cf94ba4

  Copy full SHA for cf94ba4

• Merge branch 'get_only_cached_result' into join_with_scan

  joka921

  committed
  0453f07

  Copy full SHA for 0453f07

• Do not use lazy scans when the full result of the scans is already cached.

  joka921

  committed
  66ff69e

  Copy full SHA for 66ff69e

• Add some initial tests for the so far untested stuff.

  joka921

  committed
  ccff07b

  Copy full SHA for ccff07b

• Some initial unit tests.

  joka921

  committed
  a494db2

  Copy full SHA for a494db2

• Last changes from a review.

  joka921

  committed
  536de70

  Copy full SHA for 536de70

• Last rename to finish.

  joka921

  committed
  97ba0f9

  Copy full SHA for 97ba0f9

Commits on Jun 28, 2023

• Merge branch 'master' into join_with_scan

  Show description for 7148fab
  joka921

  committed
  7148fab

  Copy full SHA for 7148fab

• Fixed some code smells and removed some code duplication.

  joka921

  committed
  05a1ee1

  Copy full SHA for 05a1ee1

• Added comments and several refactorings.

  Show description for f983e79
  joka921

  committed
  f983e79

  Copy full SHA for f983e79

• Some refactorings in the CompresseddRelation.cpp file that reduce the amount of code duplication.

  joka921

  committed
  07891b8

  Copy full SHA for 07891b8

• Several further refactorings.

  joka921

  committed
  c367566

  Copy full SHA for c367566

• Added comments etc.

  joka921

  committed
  a6d8d54

  Copy full SHA for a6d8d54

• Not much done yet.

  joka921

  committed
  213cf8b

  Copy full SHA for 213cf8b

• Merge branch 'master' into join_with_scan

  Show description for ce774c7
  joka921

  committed
  ce774c7

  Copy full SHA for ce774c7

Commits on Jun 29, 2023

• This doesn't work, find the bug.

  joka921

  committed
  e5f0e4c

  Copy full SHA for e5f0e4c

• Several refactorings that fortunately work again.

  joka921

  committed
  322423c

  Copy full SHA for 322423c

• Almost all scans return IdTable now.

  joka921

  committed
  207fff0

  Copy full SHA for 207fff0

• Changes from Hannah with review.

  joka921

  committed
  6eb0b68

  Copy full SHA for 6eb0b68

Commits on Jun 30, 2023

• Several additional improvementts.

  Show description for 0c906a6
  joka921

  committed
  0c906a6

  Copy full SHA for 0c906a6

• Changes from a round of reviews with Hannah.

  joka921

  committed
  5fa348c

  Copy full SHA for 5fa348c

• Extend the generator class by the functionality to add details.

  joka921

  committed
  c6104a5

  Copy full SHA for c6104a5

• Merge branch 'generators_with_additional_info' into join_with_scan

  Show description for b64da37
  joka921

  committed
  b64da37

  Copy full SHA for b64da37

• Add additional information.

  joka921

  committed
  dcbae54

  Copy full SHA for dcbae54

• Add additional details and size information.

  joka921

  committed
  0a97511

  Copy full SHA for 0a97511

Commits on Jul 3, 2023

• Changed the interface and the bugs.

  joka921

  committed
  0e8d2bb

  Copy full SHA for 0e8d2bb

• Started to do stuff for perfomannce, currently not yet done.

  joka921

  committed
  8a86653

  Copy full SHA for 8a86653

• Fix the unit tests.

  joka921

  committed
  6aaa445

  Copy full SHA for 6aaa445

• Still in the middle of anything.

  joka921

  committed
  e45d064

  Copy full SHA for e45d064

Commits on Jul 4, 2023

• Small changes from a review with Hannahh.

  joka921

  committed
  34d4a21

  Copy full SHA for 34d4a21

• Merge branch 'master' into join_with_scan

  joka921

  committed
  d0d1110

  Copy full SHA for d0d1110

• Several performance improvements.

  Show description for 461c92c
  joka921

  committed
  461c92c

  Copy full SHA for 461c92c

Commits on Jul 5, 2023

• Improved some infrastructure and added missing runtime information.

  joka921

  committed
  5d51c24

  Copy full SHA for 5d51c24

• Added an empty details class as the default.

  joka921

  committed
  6168e39

  Copy full SHA for 6168e39

• Merge remote-tracking branch 'origin/master' into join_with_scan

  

  Hannah Bast

  committed
  2ac317f

  Copy full SHA for 2ac317f

• Merged in hannah's merge.

  joka921

  committed
  f6fff69

  Copy full SHA for f6fff69

• Fix the unit tests.

  joka921

  committed
  99f219f

  Copy full SHA for 99f219f

• Small changes from a review with Hannahh.

  joka921

  committed
  fc08333

  Copy full SHA for fc08333

• Added an empty details class as the default.

  joka921

  committed
  63ff2f1

  Copy full SHA for 63ff2f1

• Several additional cleanups.

  Show description for a4e5db2
  joka921

  committed
  a4e5db2

  Copy full SHA for a4e5db2

• Better comments.

  Show description for 180456a
  joka921

  committed
  180456a

  Copy full SHA for 180456a

• Add the possibility to register an external details object.

  joka921

  committed
  16fdc9a

  Copy full SHA for 16fdc9a

• Merge branch 'generators_with_additional_info' into join_with_scan

  Show description for 784ce8a
  joka921

  committed
  784ce8a

  Copy full SHA for 784ce8a

• Several smaller cleanups and fixed as possible source of the ubsan unhappiness.

  joka921

  committed
  907992a

  Copy full SHA for 907992a

• Add unit tests for the getBlocksForJoin function.

  joka921

  committed
  ceee114

  Copy full SHA for ceee114

• A stub for unit tests.

  joka921

  committed
  9f71a65

  Copy full SHA for 9f71a65

• Tiny changes from a review.

  joka921

  committed
  25b3249

  Copy full SHA for 25b3249

• Some changes from a review.

  joka921

  committed
  79a5b97

  Copy full SHA for 79a5b97

• Changes from review.

  joka921

  committed
  33d9b14

  Copy full SHA for 33d9b14

• Try of fix

  joka921

  committed
  6a7e901

  Copy full SHA for 6a7e901

Commits on Jul 6, 2023

• Make the blocksize of the Permutations configurable via the Index class.

  joka921

  committed
  461f594

  Copy full SHA for 461f594

• Merge branch 'configurable_block_size' into join_with_scan

  joka921

  committed
  0205654

  Copy full SHA for 0205654

• Started to write tests for the index scan class.

  joka921

  committed
  f72c972

  Copy full SHA for f72c972

• Add unit tests for the changes in the IndexScan

  Show description for f4b2d1c
  joka921

  committed
  f4b2d1c

  Copy full SHA for f4b2d1c

• Only the changes for the AddCombinedRowToTable etc, without the generator.

  joka921

  committed
  2bb795c

  Copy full SHA for 2bb795c

• Fixed a bug in the generator.

  joka921

  committed
  88c6f3a

  Copy full SHA for 88c6f3a

• Some initial tests for the new methods of the Join class.

  joka921

  committed
  5480f6b

  Copy full SHA for 5480f6b

Commits on Jul 7, 2023

• Hopefully make the tests pass on MacOS without introducing deadlocks.

  joka921

  committed
  0b47bc0

  Copy full SHA for 0b47bc0

• Several additional changes, excluding the ones for updating the status.

  joka921

  committed
  033c73f

  Copy full SHA for 033c73f

• Include functionality for explicitly setting the status of an operation that was "optimized out" or otherwise executed by other means than its getResult() method.

  joka921

  committed
  081b076

  Copy full SHA for 081b076

• Actually use the new status.

  joka921

  committed
  2406aec

  Copy full SHA for 2406aec

• Safer finishing of the Threadsafe queue.

  joka921

  committed
  ad91a03

  Copy full SHA for ad91a03

• Some improvements, we mostly are lacking comments and a few tests.

  joka921

  committed
  f070a7c

  Copy full SHA for f070a7c

• Commented the test for theIndexScan class

  joka921

  committed
  3e32390

  Copy full SHA for 3e32390

• Better stuff.

  joka921

  committed
  0706a9f

  Copy full SHA for 0706a9f

• Merge branch 'master' into join_with_scan

  Show description for 9aa5a5c
  joka921

  committed
  9aa5a5c

  Copy full SHA for 9aa5a5c

• Merged in the master and several changes from a review.

  joka921

  committed
  93882ff

  Copy full SHA for 93882ff

• Small change from a review.

  joka921

  committed
  3837029

  Copy full SHA for 3837029

• Clang format.

  joka921

  committed
  6adc20a

  Copy full SHA for 6adc20a

• Merge branch 'master' into join_with_scan

  Show description for ed20f3a
  joka921

  committed
  ed20f3a

  Copy full SHA for ed20f3a

• Fixed the merge.

  joka921

  committed
  8b0ac05

  Copy full SHA for 8b0ac05

• Fixed the merge.

  joka921

  committed
  5f7c172

  Copy full SHA for 5f7c172

Commits on Jul 10, 2023

• Make the lazy joins with very small

  Show description for 7b256fd
  joka921

  committed
  7b256fd

  Copy full SHA for 7b256fd

• Materialize small-ish scans

  joka921

  committed
  30835cc

  Copy full SHA for 30835cc

• Several improvements and comments.

  joka921

  committed
  0030114

  Copy full SHA for 0030114

• Use supported comparison.

  joka921

  committed
  aed7ea0

  Copy full SHA for aed7ea0

• formatted this

  joka921

  committed
  944ed2a

  Copy full SHA for 944ed2a

• Small changes from a review with Hannah.

  joka921

  committed
  b620bff

  Copy full SHA for b620bff

• Better stuff.

  joka921

  committed
  f82ad65

  Copy full SHA for f82ad65

Commits on Jul 11, 2023

• Improve test coverage.

  joka921

  committed
  94cb174

  Copy full SHA for 94cb174

• Implement a useful helper.

  joka921

  committed
  54ace64

  Copy full SHA for 54ace64

• Merge branch 'threadsafe_ordered_queue' into join_with_scan

  Show description for ef58a42
  joka921

  committed
  ef58a42

  Copy full SHA for ef58a42

• Several improvements of the test coverage. This should be it (once we have the test coverage for the thread safe queue as well).

  joka921

  committed
  d57319a

  Copy full SHA for d57319a

• Fixed the test.

  joka921

  committed
  c4fa334

  Copy full SHA for c4fa334

• Add a simple coroutine that makes working with the quees much simpler.

  joka921

  committed
  9c8ebcd

  Copy full SHA for 9c8ebcd

• Merge branch 'noexcept_finish_threadsafe_queue' into join_with_scan

  Show description for bb6dec0
  joka921

  committed
  bb6dec0

  Copy full SHA for bb6dec0

• Merged in the current threadsafe queue, and

  Show description for 65790fc
  joka921

  committed
  65790fc

  Copy full SHA for 65790fc

• Merge branch 'master' into join_with_scan

  Show description for e9d427b
  joka921

  committed
  e9d427b

  Copy full SHA for e9d427b

• Merge branch 'master' into join_with_scan

  Show description for dc478dd
  joka921

  committed
  dc478dd

  Copy full SHA for dc478dd

• Small changes from a review with Hannah.

  joka921

  committed
  4be2f92

  Copy full SHA for 4be2f92

Commits on Jul 12, 2023

• Fix a possible race condition in the case that multiple threads are pushing exceptions.

  joka921

  committed
  bb2dbde

  Copy full SHA for bb2dbde

• Fix clang format.

  joka921

  committed
  2519a74

  Copy full SHA for 2519a74