Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Naive interleaved filtering and data reading #918

Merged
merged 7 commits into from
Sep 24, 2024

Conversation

AdamGS
Copy link
Contributor

@AdamGS AdamGS commented Sep 24, 2024

Read filter-related data using a different LayoutReader, but with a shared message cache. Once we wire-in metadata, we can use this information to skip chunks/parts of chunks.

There are some assumptions on overall alignments/batch size, that are held by the underlying BatchReader.

@AdamGS AdamGS marked this pull request as ready for review September 24, 2024 12:57
@AdamGS AdamGS marked this pull request as draft September 24, 2024 12:58
@AdamGS AdamGS force-pushed the adamg/inline-filtering-thing-v2 branch from b4328c8 to 465231a Compare September 24, 2024 13:02
@AdamGS AdamGS marked this pull request as ready for review September 24, 2024 14:56
Copy link
Member

@robert3005 robert3005 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This removes having to unify schemas of filters and projections but having two readers is awkward. The scan to layout is not used beyond being used for schema projection and we keep leaning on stream having a copy of top level scan

@AdamGS AdamGS merged commit e9f0d4d into develop Sep 24, 2024
5 checks passed
@AdamGS AdamGS deleted the adamg/inline-filtering-thing-v2 branch September 24, 2024 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants