Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support for pushdown projection in reading Avro #827

Merged
merged 2 commits into from
Feb 9, 2022

Conversation

jorgecarleitao
Copy link
Owner

This allows skipping deserializing columns when they are not required.

The speedup is proportional to the number of columns that are not needed (times the average cost of deserializing a column type over all skipped columns).

@codecov
Copy link

codecov bot commented Feb 9, 2022

Codecov Report

Merging #827 (7caea8e) into main (a026b66) will decrease coverage by 0.00%.
The diff coverage is 75.60%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #827      +/-   ##
==========================================
- Coverage   71.55%   71.54%   -0.01%     
==========================================
  Files         326      326              
  Lines       17581    17643      +62     
==========================================
+ Hits        12580    12623      +43     
- Misses       5001     5020      +19     
Impacted Files Coverage Δ
src/io/avro/read/schema.rs 69.62% <50.00%> (ø)
src/io/avro/read/deserialize.rs 73.03% <74.32%> (-1.27%) ⬇️
src/io/avro/read/mod.rs 82.35% <100.00%> (+2.35%) ⬆️
src/io/avro/read_async/metadata.rs 95.00% <100.00%> (ø)
src/bitmap/utils/slice_iterator.rs 86.20% <0.00%> (-1.73%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a026b66...7caea8e. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 2c4dbb2 into main Feb 9, 2022
@jorgecarleitao jorgecarleitao deleted the avro_rename branch February 9, 2022 15:24
@jorgecarleitao jorgecarleitao added enhancement An improvement to an existing feature feature A new feature and removed feature A new feature enhancement An improvement to an existing feature labels Mar 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant