Update type conversion from pandas to timestamp to support various the timestamp types #1603
Conversation
achals
force-pushed
the
achal/timestamp-infer
branch
from
3c5a9ef to
96d76b0
Compare
Codecov Report
@@ Coverage Diff @@
## master #1603 +/- ##
==========================================
- Coverage 83.61% 83.57% -0.05%
==========================================
Files 65 65
Lines 5761 5771 +10
==========================================
+ Hits 4817 4823 +6
- Misses 944 948 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
| @@ -438,8 +439,14 @@ def pa_to_value_type(pa_type: object): | |||
|
|
|||
|
|
|||
| def pa_to_feast_value_type(value: Union[pa.lib.ChunkedArray, str]) -> ValueType: | |||
| value_type = ( | |||
| value.type.__str__() if isinstance(value, pa.lib.ChunkedArray) else value | |||
There was a problem hiding this comment.
Would you mind explaining this line?
There was a problem hiding this comment.
This is moved from line 459 - basically this converts the type of a pandas column into a string representation, if the stringified version has not already been passed in.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: achals, woop The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
Signed-off-by: Achal Shah <[email protected]>
…e specifications Signed-off-by: Achal Shah <[email protected]>
Signed-off-by: Achal Shah <[email protected]>
Signed-off-by: Achal Shah <[email protected]>
achals
force-pushed
the
achal/timestamp-infer
branch
from
fdf1708 to
1acd139
Compare
|
New changes are detected. LGTM label has been removed. |
|
/lgtm |
|
@achals: you cannot LGTM your own PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
…e timestamp types (#1603) * Fix precommit to run black on commit Signed-off-by: Achal Shah <[email protected]> * Update type conversion from pandas to timestamp to support various the specifications Signed-off-by: Achal Shah <[email protected]> * Undo precommit changs Signed-off-by: Achal Shah <[email protected]> * make format Signed-off-by: Achal Shah <[email protected]>
What this PR does / why we need it:
Currently, Inferring features fails on columns of type timestamp[ms, tz=UTC] or other tz. This diff attempts to fix using a regex instead of enumerating all the different possibilities.
An option is to use the pandas type instead of the string representation, but this is probably good enough.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: