Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix default parameter values of write_csv and write_parquet #6967

Merged
merged 2 commits into from
Dec 11, 2020

Conversation

vuule
Copy link
Contributor

@vuule vuule commented Dec 10, 2020

Fixes #6671, #6851

  • Set the rows_per_chunk in csv_writer_options to the size of the input table.
  • Change rows_per_chunk type to size_type (used for number of rows).
  • Set the default compression in to_parquet/write_parquet to "snappy".

@vuule vuule added bug Something isn't working Python Affects Python cuDF API. cuIO cuIO issue breaking Breaking change labels Dec 10, 2020
@vuule vuule self-assigned this Dec 10, 2020
@vuule vuule added the libcudf Affects libcudf (C++/CUDA) code. label Dec 10, 2020
@vuule vuule marked this pull request as ready for review December 10, 2020 19:13
@vuule vuule requested review from a team as code owners December 10, 2020 19:13
@vuule
Copy link
Contributor Author

vuule commented Dec 10, 2020

rerun tests

Copy link
Collaborator

@kkraus14 kkraus14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python lgtm

@vuule vuule added 3 - Ready for Review Ready for review by team 4 - Needs cuIO Reviewer 4 - Needs Review Waiting for reviewer to review or respond labels Dec 10, 2020
Copy link
Contributor

@codereport codereport left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@kkraus14 kkraus14 added 5 - Ready to Merge Testing and reviews complete, ready to merge 6 - Okay to Auto-Merge and removed 3 - Ready for Review Ready for review by team 4 - Needs cuIO Reviewer labels Dec 10, 2020
@codecov
Copy link

codecov bot commented Dec 10, 2020

Codecov Report

Merging #6967 (f54f10d) into branch-0.18 (8a1a6d7) will increase coverage by 1.33%.
The diff coverage is n/a.

Impacted file tree graph

@@               Coverage Diff               @@
##           branch-0.18    #6967      +/-   ##
===============================================
+ Coverage        81.59%   82.93%   +1.33%     
===============================================
  Files               96       96              
  Lines            15907    17388    +1481     
===============================================
+ Hits             12980    14420    +1440     
- Misses            2927     2968      +41     
Impacted Files Coverage Δ
python/cudf/cudf/utils/ioutils.py 78.71% <ø> (ø)
python/dask_cudf/dask_cudf/io/parquet.py 91.07% <ø> (+0.24%) ⬆️
python/cudf/cudf/io/csv.py 93.75% <0.00%> (-0.25%) ⬇️
python/cudf/cudf/io/feather.py 100.00% <0.00%> (ø)
python/cudf/cudf/comm/serialize.py 0.00% <0.00%> (ø)
python/cudf/cudf/_fuzz_testing/io.py 0.00% <0.00%> (ø)
python/dask_cudf/dask_cudf/_version.py 0.00% <0.00%> (ø)
python/dask_cudf/dask_cudf/io/tests/test_csv.py 100.00% <0.00%> (ø)
python/dask_cudf/dask_cudf/io/tests/test_orc.py 100.00% <0.00%> (ø)
python/dask_cudf/dask_cudf/io/tests/test_json.py 100.00% <0.00%> (ø)
... and 38 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a1a6d7...f54f10d. Read the comment docs.

@harrism harrism removed the 4 - Needs Review Waiting for reviewer to review or respond label Dec 10, 2020
@vuule
Copy link
Contributor Author

vuule commented Dec 11, 2020

Rerun tests

@rapids-bot rapids-bot bot merged commit df5d452 into rapidsai:branch-0.18 Dec 11, 2020
@vuule vuule deleted the bug-cuio-defaults branch April 20, 2022 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 - Ready to Merge Testing and reviews complete, ready to merge breaking Breaking change bug Something isn't working cuIO cuIO issue libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] CSV writer parameter rows_per_chunk has a bad default value
5 participants