Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Improved flexibility of reading parquet #820

Merged
merged 2 commits into from
Feb 6, 2022
Merged

Improved flexibility of reading parquet #820

merged 2 commits into from
Feb 6, 2022

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Feb 6, 2022

This exposes an API to read parquet in different ways:

  • read (IO-bounded) columns in parallel (!= concurrently)
  • read and deserialize columns one by one (reduce memory pressure for column-based execution engines)

This allows the both of the two worlds: allow reading both multiple fields and distributed/concurrent work over individual fields.

@jorgecarleitao jorgecarleitao added the enhancement An improvement to an existing feature label Feb 6, 2022
@codecov
Copy link

codecov bot commented Feb 6, 2022

Codecov Report

Merging #820 (8539846) into main (4fbbd90) will decrease coverage by 0.00%.
The diff coverage is 82.35%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #820      +/-   ##
==========================================
- Coverage   71.27%   71.27%   -0.01%     
==========================================
  Files         327      327              
  Lines       17544    17543       -1     
==========================================
- Hits        12504    12503       -1     
  Misses       5040     5040              
Impacted Files Coverage Δ
src/io/parquet/read/file.rs 69.38% <ø> (ø)
src/io/parquet/read/row_group.rs 74.24% <82.35%> (-0.39%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4fbbd90...8539846. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 3d528c9 into main Feb 6, 2022
@jorgecarleitao jorgecarleitao deleted the more_flex branch February 6, 2022 11:56
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement An improvement to an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant