Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partition-scoped storage reader in invoker #1474

Merged
merged 2 commits into from
Apr 29, 2024
Merged

Partition-scoped storage reader in invoker #1474

merged 2 commits into from
Apr 29, 2024

Conversation

AhmedSoliman
Copy link
Contributor

@AhmedSoliman AhmedSoliman commented Apr 26, 2024

Partition-scoped storage reader in invoker

This changes the invoker to allow partitions to pass down scoped storage readers along registration messages in invoker. This means that the invoker uses the partition-supplied storage reader for invocations owned by this particular partition.
Additionally, this attempts to consolidate (in most places) the journal reader and state reader under a single generic type.


Stack created with Sapling. Best reviewed with ReviewStack.

Copy link

github-actions bot commented Apr 26, 2024

Test Results

 95 files  ±0   95 suites  ±0   7m 23s ⏱️ -2s
 82 tests ±0   80 ✅ ±0  2 💤 ±0  0 ❌ ±0 
212 runs  ±0  206 ✅ ±0  6 💤 ±0  0 ❌ ±0 

Results for commit 9b9dff9. ± Comparison against base commit 5a826ce.

♻️ This comment has been updated with latest results.

Copy link
Contributor

@tillrohrmann tillrohrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work @AhmedSoliman. LGTM. +1 for merging.

This is one step in a series of changes to move to a one CF per partition-id. The change consolidates all partitions into a single CF `data-unpartitioned`.

Notes:
- This introduces a fixed key prefix to partition CF to improve bloom filter lookups. This change require some subtle handling of scans that spans multiple prefixes, this is handled and is covered by existing test cases.
- This adds the ability for a table to have multiple KeyKinds attached. Each TableKey has an associated kind, and the table (TableKind) has a set of KeyKinds associated.
- This is designed to allow future separation of tables, or even individual key kinds to separate CFs.
- Some values are intentionally passed but not used for (a) clarity at call site and (b) to help with follow-up refactoring.


Side note: cluster marker tests fail sporadically, this adds a flush after each temp file write in an attempt to fix them.
This changes the invoker to allow partitions to pass down scoped storage readers along registration messages in invoker. This means that the invoker uses the partition-supplied storage reader for invocations owned by this particular partition.
Additionally, this attempts to consolidate (in most places) the journal reader and state reader under a single generic type.
@AhmedSoliman AhmedSoliman merged commit 05eb4ba into main Apr 29, 2024
13 checks passed
@AhmedSoliman AhmedSoliman deleted the pr1474 branch April 29, 2024 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants