PDEP-6: Ban upcasting in setitem-like operations

[MarcoGorelli]

Original discussion: #39584

Turning into a PDEP as I think the int -> float warrants some care

[asishm]

I like it!

2 questions

1. You have mentioned int vs float upcast/downcast as being an open item. What about upcasts within ints - for example if an out of bounds integer was set in an int8 series. (Currrently this upcasts to int16 / int32 etc. as appropriate)

2. Similarly - What about setting pd.NA or np.nan to an int series (non-nullable). Currently this upcasts to float

[MarcoGorelli]

Thanks @asishm for taking a look! Setting an out-of-bounds value in an int8 Series, or np.nan in any 'int' (non-nullable) Series, would also raise

[MarcoGorelli]

Thanks for your reviews!

Right, I think it's time for yet another @pandas-dev/pandas-core @pandas-dev/pandas-triage ping...would love to hear people's thoughts on this

[Dr-Irv]

I think you should be more complete on all the operations that would be affected. s.loc[1] = 1.5 would also raise under this proposal. In addition, setting values of a column in a DataFrame would also raise.

Also, we have discussed the possibility of eliminating inplace = True. If we did that first, wouldn't then the only places where this PDEP would apply be in Series.__setitem__() and Series.loc.__setitem__() (and the corresponding DataFrame places as well)?

[WillAyd]

I like the ideas here and would be in favor of the strict option for the float / int case. I think its easy enough to work around if that's really what you want and would enforce more consistent behavior within our own library

[rhshadrach]

Looks good!

[jreback]

is the intention to follow for other routines which can change dtype (either other pdep or as a follow up)

eg fillna, where?

[twoertwein]

I'm happy to contribute to pandas (and pandas-stubs) and will voice my opinion when I have a strong one, but I wouldn't mind forfeiting my voting privileges.

Restricting the voting pool to more engaged people might make voting easier. If there are/will be PDEPs where you are short on votes, please ping me (again) if I haven't voted yet.

[mroeschke]

+1

[lukemanley]

+1

[datapythonista]

+1Thanks for working on this @MarcoGorelli, very nice improvement.

Just two minor comments. When this is implemented, I'd provide in the error message how to change the type of the column (it's not present in the sample error message in the abstract).

About setting float in an int column, of the 3 listed options I'd go with 1 too, but I think another option to always raise should be available, and that would be my preference. Technically floats are approximations, and the concept of the decimal part being 0 is not accurate, which brings questions about which precision to use. No big deal, happy to move forward with option 1 too, and see this PDEP move forward which will be a great improvement. But something to consider during implementation and for the future.

[Dr-Irv]

Added one comment. There are some other comments I've made that are still unresolved.

[jbrockmendel]

+1

[Dr-Irv]

+1

[MarcoGorelli]

Right, that's 12 upvotes, and no downvotes. Lot's of great discussion here, thanks all for the attention to detail, much appreciated!

Merging this today then

[attack68]

+1