-
Notifications
You must be signed in to change notification settings - Fork 317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(mito): parquet memtable reader #4967
base: main
Are you sure you want to change the base?
feat(mito): parquet memtable reader #4967
Commits on Nov 15, 2024
-
Configuration menu - View commit details
-
Copy full SHA for d98d10b - Browse repository at this point
Copy the full SHA d98d10bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5c5e36c - Browse repository at this point
Copy the full SHA 5c5e36cView commit details -
Refactor MemtableRowGroupReader to streamline data fetching
- Added early return when fetch_ranges is empty to optimize performance. - Replaced inline chunk data assignment with a call to `assign_dense_chunk` for cleaner code.
Configuration menu - View commit details
-
Copy full SHA for 068dc06 - Browse repository at this point
Copy the full SHA 068dc06View commit details -
Configuration menu - View commit details
-
Copy full SHA for b5a5ff2 - Browse repository at this point
Copy the full SHA b5a5ff2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 34478d0 - Browse repository at this point
Copy the full SHA 34478d0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4470d7c - Browse repository at this point
Copy the full SHA 4470d7cView commit details -
Enhance BulkPart Iteration with Filtering
- Introduced `RangeBase` to `BulkIterContext` for improved filter handling. - Implemented filter application in `BulkPartIter` to prune batches based on predicates. - Updated `SimpleFilterContext::new_opt` to be public for broader access.
Configuration menu - View commit details
-
Copy full SHA for 69008a5 - Browse repository at this point
Copy the full SHA 69008a5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6d7c8d6 - Browse repository at this point
Copy the full SHA 6d7c8d6View commit details -
Configuration menu - View commit details
-
Copy full SHA for cc422ab - Browse repository at this point
Copy the full SHA cc422abView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1719213 - Browse repository at this point
Copy the full SHA 1719213View commit details -
Enhance BulkPart read method to return Option<BoxedBatchIterator>
- Modified `BulkPart::read` to return `Option<BoxedBatchIterator>` to handle cases where no row groups are selected. - Added logic to return `None` when all row groups are filtered out. - Updated tests to handle the new return type and added a test case to verify behavior when no row groups match the pr
Configuration menu - View commit details
-
Copy full SHA for e60b832 - Browse repository at this point
Copy the full SHA e60b832View commit details -
refactor/separate-paraquet-reader: Add helper function to parse parqu…
…et metadata and integrate it into BulkPartEncoder
Configuration menu - View commit details
-
Copy full SHA for c9ceb72 - Browse repository at this point
Copy the full SHA c9ceb72View commit details -
refactor/separate-paraquet-reader:
Change BulkPartEncoder row_group_size from Option to usize and update tests
Configuration menu - View commit details
-
Copy full SHA for dbb4e3e - Browse repository at this point
Copy the full SHA dbb4e3eView commit details -
refactor/separate-paraquet-reader: Add context module for bulk memtab…
…le iteration and refactor part reading • Introduce context module to encapsulate context for bulk memtable iteration. • Refactor BulkPart to use BulkIterContextRef for reading operations. • Remove redundant code in BulkPart by centralizing context creation and row group pruning logic in the new context module. • Create new file context.rs with structures and logic for handling iteration context. • Adjust part_reader.rs and row_group_reader.rs to reference the new BulkIterContextRef.
Configuration menu - View commit details
-
Copy full SHA for 74c0474 - Browse repository at this point
Copy the full SHA 74c0474View commit details -
refactor/separate-paraquet-reader: Refactor RowGroupReader traits and…
… implementations in memtable and parquet reader modules • Rename RowGroupReaderVirtual to RowGroupReaderContext for clarity. • Replace BulkPartVirt with direct usage of BulkIterContextRef in MemtableRowGroupReader. • Simplify MemtableRowGroupReaderBuilder by directly passing context instead of creating a BulkPartVirt instance. • Update RowGroupReaderBase to use context field instead of virt, reflecting the trait renaming and usage. • Modify FileRangeVirt to FileRangeContextRef and adjust implementations accordingly.
Configuration menu - View commit details
-
Copy full SHA for 0eebb5f - Browse repository at this point
Copy the full SHA 0eebb5fView commit details -
refactor/separate-paraquet-reader: Refactor column page reader creati…
…on and remove unused code • Centralize creation of SerializedPageReader in RowGroupBase::column_reader method. • Remove unused RowGroupCachedReader and related code from MemtableRowGroupPageFetcher. • Eliminate redundant error handling for invalid column index in multiple places.
Configuration menu - View commit details
-
Copy full SHA for 0af7d70 - Browse repository at this point
Copy the full SHA 0af7d70View commit details -
Configuration menu - View commit details
-
Copy full SHA for 356bb47 - Browse repository at this point
Copy the full SHA 356bb47View commit details
Commits on Dec 1, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 1ec83c2 - Browse repository at this point
Copy the full SHA 1ec83c2View commit details