BUG: setitem with mixed-resolution dt64s

[jbrockmendel]

• closes BUG/API: setitem/where with mixed dt64 units #56410 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[MarcoGorelli]

what happens after this PR for the cases in #56410 ? would they emit a warning now, and raise in the future?

[jbrockmendel]

what happens after this PR for the cases in #56410 ? would they emit a warning now, and raise in the future?

The __setitem__ case issues a PDEP6 warning and will raise when that is enforced. The .where and .fillna cases case to "M8[ms]".

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[jbrockmendel]

@MarcoGorelli gentle ping

[jbrockmendel]

if there are no further comments, i plan to merge this next week

[MarcoGorelli]

sorry for the delay, will try to take a look tomorrow

[MarcoGorelli]

Sorry this has taken a while, just not sure about something

Isn't this going to risk causing overflows? E.g.

ser = pd.Series(np.array(['NaT', '3000-01-01', '3000-01-02'], dtype='datetime64[s]'))

item = pd.Timestamp('2020-01-01T00:00:00.123456789')
print(ser.fillna(item))

This raises

pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 3000-01-01 00:00:00

Checking the behaviour for nullable integers:

pd.Series([pd.NA, 2, 3], dtype='Int64').fillna(1.5)  # raises
pd.Series([pd.NA, 2, 3], dtype='Int64[pyarrow]').fillna(1.5)  # silently truncates 1.5 to 1

[jbrockmendel]

Isn't this going to risk causing overflows? E.g.

Yes, and that is a good thing. In the status quo we silently round that non-nano fill value and give an incorrect result.

The pyarrow example looks pretty clearly buggy to me.

[MarcoGorelli]

to be honest I'm not totally sure about this - I'm not going to block it, but don't really feel comfortable approving, could you ask someone else please?

[jbrockmendel]

@mroeschke @phofl thoughts?

[mroeschke]

I do find the overflow behavior a better behavior than silently truncating.

But I also find the exception message in @MarcoGorelli 's #56419 (comment) kinda opaque compared to setting a float into a nullable int (TypeError: Invalid value '1.5' for dtype Int64)

[jbrockmendel]

It is a bit tough to tailor the exception message due to where the exception is raised. How about f"Incompatible (high-resolution) value for {self.dtype}. Explicitly cast before operating."

[mroeschke]

How about f"Incompatible (high-resolution) value for {self.dtype}. Explicitly cast before operating."

Yeah I would be good with that

[jbrockmendel]

npdev failure looks unrelated

[mroeschke]

Thanks @jbrockmendel