Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

parquet redux #1476

Merged
merged 16 commits into from
Jul 19, 2022
Merged

parquet redux #1476

merged 16 commits into from
Jul 19, 2022

Conversation

martindurant
Copy link
Contributor

@codecov
Copy link

codecov bot commented May 18, 2022 •
edited
Loading

Codecov Report

Merging #1476 (721f523) into main (78a6535) will increase coverage by 0.23%.
The diff coverage is 73.86%.

Impacted Files Coverage Δ
src/awkward/_v2/contents/indexedoptionarray.py 89.14% <54.54%> (-0.70%) ⬇️
src/awkward/_v2/highlevel.py 70.77% <57.60%> (-2.76%) ⬇️
src/awkward/_v2/operations/ak_from_parquet.py 87.02% <88.09%> (+37.87%) ⬆️
src/awkward/_v2/operations/ak_to_parquet.py 51.02% <95.23%> (+3.02%) ⬆️
src/awkward/_v2/_connect/numba/arrayview.py 96.76% <100.00%> (ø)
src/awkward/_v2/contents/content.py 76.61% <100.00%> (ø)
src/awkward/_v2/operations/ak_copy.py 100.00% <100.00%> (ø)
src/awkward/_v2/operations/ak_is_none.py 96.55% <100.00%> (+0.39%) ⬆️
...awkward/_v2/operations/ak_metadata_from_parquet.py 100.00% <100.00%> (+68.42%) ⬆️
src/awkward/_v2/record.py 76.86% <100.00%> (-0.18%) ⬇️
... and 19 more

@jpivarski
Copy link
Member

AK101: raise Exception → raise ak._v2._util.error(Exception)

@martindurant
Copy link
Contributor Author

AK101: raise Exception → raise ak._v2._util.error(Exception)

Not too worried about the lint for the moment :)

More worried about this kind of fudge.

@martindurant
Copy link
Contributor Author

@jpivarski , ready for you to have a look at this, so we can discuss.

Two outstanding issues I haven't addressed:

  • what happens when you have one Record from each row-group. I suppose this is an array, but exactly how to concat is different
  • I am not checking the column names reported in the metadata, as I'm not convinced that the writing arm is producing the right thing. We should come up with concrete tests to make certain.

@martindurant martindurant marked this pull request as ready for review June 27, 2022 18:33
@jpivarski jpivarski mentioned this pull request Jun 30, 2022
Closed
jpivarski
jpivarski approved these changes Jul 19, 2022
View reviewed changes
Copy link
Member

@jpivarski jpivarski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to the CI tests, I also tested it with some big files locally, and varied the options. All is good.

Reading the code itself, everything looks good to me!

I'll set this to auto-merge if all tests pass.

@jpivarski jpivarski enabled auto-merge (squash) July 19, 2022 16:38
@jpivarski jpivarski merged commit a3fd9f9 into scikit-hep:main Jul 19, 2022
ManasviGoyal pushed a commit that referenced this pull request Aug 6, 2022
@martindurant @pre-commit-ci
@jpivarski @ManasviGoyal
parquet redux (#1476)
96e15eb 
* stop

* get some tests going

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Selection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* a fix

* questionable fix

* tidy

* update exception calls

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* correct to_parq (thanks flake)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jim Pivarski <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jpivarski jpivarski jpivarski approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@martindurant @jpivarski