Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor Python factories and remove usage of Table for libcudf output handling #8687

Merged
merged 56 commits into from
Aug 3, 2021
Merged

Refactor Python factories and remove usage of Table for libcudf output handling #8687

merged 56 commits into from
Aug 3, 2021

Conversation

vyasr
Copy link
Contributor

@vyasr vyasr commented Jul 8, 2021 •
edited
Loading

This PR serves two purposes:

  1. Provide a simpler mode of interaction with libcudf APIs that does not involve constructing and then deconstructing a Table object.
  2. Provide a single fast path (the _from_data method) for constructing Frame-like objects from other Frame-like objects that uses the lowest common denominator (a dict of columns and an optional index) for all other current methods (e.g. _from_table and _copy_construct).

vyasr added 30 commits July 7, 2021 16:37
@vyasr
Redefine DataFrame._from_table in terms of DataFrame._from_data.
922bf03
@vyasr
Redefine Series._from_table in terms of Series._from_data.
f85be21
@vyasr
Add new API for getting columns from a unique ptr.
025ffa1
@vyasr
Use new API to return list of columns from libcudf, write new code to…
d68fa95 
… parse in Python.
@vyasr
Move conversion function to utils.
6b47727
@vyasr
Combine utility functions into one.
df49647
@vyasr
Convert explode to new API.
0439935
@vyasr
Convert gather to new API.
a9a52b6
@vyasr
Convert hash to new API.
c684d17
@vyasr
Convert scatter to new API.
c3add5b
@vyasr
Convert empty_like and partition.
529385b
@vyasr
Convert more APIs.
27b99ad
@vyasr
Remove remaining from_unique_ptr calls that pass up to frame.py.
abe12fd
@vyasr
Change read_parquet.
012d6fe
@vyasr
Change string partitioning (but still wrapped in a Frame in Python fo…
98a4623 
…r the ColumnMethods).
@vyasr
Change string splitting (but still wrapped in a Frame in Python for t…
9f36fa4 
…he ColumnMethods).
@vyasr
Change string extract and findall.
71e5dae
@vyasr
Change merge_sorted.
da00ee1
@vyasr
Update interop.
48dbdf4
@vyasr
Update avro and json.
0d8c719
@vyasr
Switch Groupby.groups.
b47edb2
@vyasr
Update orc.
986516a
@vyasr
Update groupby shift.
ae2e731
@vyasr
Update groupby replace_nulls.
ffe520c
@vyasr
Update groupby aggregate.
5c6f254
@vyasr
Update csv.
8af8ec9
@vyasr
Remove from_unique_ptr.
c4f7cb5
@vyasr
Reduce usage of _from_table in frame.py.
9e60845
@vyasr
Remove unnecessary test.
321071d
@vyasr
Add new function for data from table view.
9e1ab29
@vyasr vyasr added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Jul 26, 2021
@vyasr vyasr mentioned this pull request Jul 26, 2021
Merged
@vyasr vyasr requested review from shwina and isVoid and removed request for marlenezw and skirui-source July 26, 2021 22:10
isVoid
isVoid reviewed Jul 27, 2021
View reviewed changes
Copy link
Contributor

@isVoid isVoid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any red flags. Small comments on doc below

python/cudf/cudf/_lib/utils.pyx Outdated Show resolved Hide resolved
@vyasr vyasr requested a review from isVoid July 27, 2021 23:41
@vyasr vyasr mentioned this pull request Jul 28, 2021
Merged
@shwina
Copy link
Contributor

shwina commented Jul 29, 2021

Before merging, I'd check with cuML (cc: @dantegd) as well as cuGraph (cc: @BradReesWork) if/how this will break them.

@shwina
Copy link
Contributor

shwina commented Jul 29, 2021

cc: @thomcom for cuSpatial as well

@vyasr
Copy link
Contributor Author

vyasr commented Jul 29, 2021 •
edited
Loading

I've already opened rapidsai/cuspatial#437 for when this PR gets merged, happy to help out if this affects cuML or cuGraph.

rlratzel
rlratzel approved these changes Aug 3, 2021
View reviewed changes
Copy link
Contributor

@rlratzel rlratzel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly checked the API changes and did not dive too much into details. These changes do not look like they'll break cuGraph.

@shwina
Copy link
Contributor

shwina commented Aug 3, 2021

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 7f704d6 into rapidsai:branch-21.10 Aug 3, 2021
rapids-bot bot pushed a commit to rapidsai/cuspatial that referenced this pull request Aug 4, 2021
@vyasr
Update cudf table apis (#437)
960894f 
rapidsai/cudf#8687 changes the internals of cuDF and removes the means by which libcudf table objects are converted to cuDF Frames. This PR should be merged after that one to match those APIs. I have verified locally that this PR's Cython code compiles and runs against that branch of cuDF.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Paul Taylor (https://github.com/trxcllnt)

URL: #437
@vyasr vyasr deleted the refactor/factories branch January 14, 2022 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@galipremsagar galipremsagar galipremsagar left review comments

@isVoid isVoid isVoid approved these changes

@shwina shwina shwina approved these changes

@rlratzel rlratzel rlratzel approved these changes

Assignees

@vyasr vyasr

Labels
3 - Ready for Review Ready for review by team improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
None yet
Milestone
cuDF Python Refactoring
Development

Successfully merging this pull request may close these issues.
5 participants
@vyasr @shwina @rlratzel @galipremsagar @isVoid