-
Notifications
You must be signed in to change notification settings - Fork 224
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pyarrow: Support date32[day] and date64[ms] dtypes in pandas objects #2845
Conversation
Handle date columns in pandas.DataFrame with pyarrow dtypes like date32[day][pyarrow] or date64[ms][pyarrow] by modifying the vectors_to_arrays conversion function. Added some parametrized unit tests to test_info.py to ensure this works.
Need to handle Python lists that don't have the dtype attribute, unlike pandas.Series objects. Also ensure that we return a C-contiguous array.
Ensure that pyarrow date32 and date64 dtypes are converted to numpy.datetime64 dtype. Added pyarrow dependency to ci_doctests.yaml. Also changed from using `"date" in vec_dtype` to `vec_dtype.startswith("date")`.
Co-Authored-By: Dongdong Tian <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, but need to mark the PR as ready for review and merge the main branch to make sure all tests pass.
I'm still debating on whether to add pyarrow into pygmt/pygmt/tests/test_plot.py Lines 409 to 452 in 20054a1
That test is already a bit long though ... |
On second thought, I might do this in a follow-up PR. PyArrow also has time32/time64 types, so I might add a |
Description of proposed changes
Handle date columns in pandas.DataFrame with pyarrow dtypes like
date32[day][pyarrow]
ordate64[ms][pyarrow]
.This is implemented by modifying the
vectors_to_arrays
conversion function to convert date32/date64 dtypes to numpy.datetime64.An altenative implementation would be to handle pyarrow date32/date64 dtypes natively without converting, but this might require more work and discussion at #2800.
Note:
References:
Part of #2800, related to #242, extends #464
Reminders
make format
andmake check
to make sure the code follows the style guide.doc/api/index.rst
.Slash Commands
You can write slash commands (
/command
) in the first line of a comment to performspecific operations. Supported slash commands are:
/format
: automatically format and lint the code/test-gmt-dev
: run full tests on the latest GMT development version