-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support writing nested frames to parquet files #83
Conversation
Click here to view all benchmarks. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #83 +/- ##
==========================================
+ Coverage 98.65% 98.68% +0.02%
==========================================
Files 15 15
Lines 818 836 +18
==========================================
+ Hits 807 825 +18
Misses 11 11 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, I just have one question on what the output for by_layer with multiple partitions looks like/should look like
Change Description
Here we add a
to_parquet
method which serializes aNestedFrame
as a parquet file, either on a "per-layer" basis where each layer is written as its own parquet file in a specified directory or as a single file where nested layers are already embedded within the columns.read_parquet
is adjusted to handle either case by no longer requiring theto_pack
argument.We also add a helper method in
generation.py
to take write randomly generated data to parquet file(s).Addresses #43
Solution Description
Code Quality
New Feature Checklist