You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
We want to add `date_dtype, datetime_dtype, time_dtype, and timestamp_dtype to the to_dataframe API, similar to bool_dtype, int_dtype, float_dtype, and string_dtype (#1529).
Pyarrow's conversion of data values to nanosecond precision can result in an out-of-bounds error for datetime values like 9999-12-31T23:59:59.999999. Therefore, the default datetime_dtype mapping will be ignored, and all datetime columns will be of object type. Once pandas supports pyarrow dtype backend (link), we can use more flexible custom time dtypes, such as pandas.ArrowDtype(pyarrow.timestamp("us", tz="UTC")).
Describe the solution you'd like
Add custom dtype mapping for these time dtypes conversion.
Describe alternatives you've considered
We can always cast the object dtypes to these custom dtype but would cause performance loss.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
We want to add `date_dtype, datetime_dtype, time_dtype, and timestamp_dtype to the to_dataframe API, similar to bool_dtype, int_dtype, float_dtype, and string_dtype (#1529).
Pyarrow's conversion of data values to nanosecond precision can result in an out-of-bounds error for datetime values like
9999-12-31T23:59:59.999999
. Therefore, the defaultdatetime_dtype
mapping will be ignored, and alldatetime
columns will be ofobject
type. Once pandas supportspyarrow dtype
backend (link), we can use more flexible custom time dtypes, such aspandas.ArrowDtype(pyarrow.timestamp("us", tz="UTC"))
.Describe the solution you'd like
Add custom dtype mapping for these time dtypes conversion.
Describe alternatives you've considered
We can always cast the object dtypes to these custom dtype but would cause performance loss.
The text was updated successfully, but these errors were encountered: