Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PDEP-6: Ban upcasting in setitem-like operations #50424

Merged
merged 33 commits into from
Apr 21, 2023
Merged
Changes from 2 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
1fb0adb
[skip ci] pdep6 draft
Dec 22, 2022
5456787
[skip ci] reword
Dec 24, 2022
02ff735
[skip ci] compare with DataFrames.jl
Dec 24, 2022
e3cc381
[skip ci] note about loss of precision
Dec 24, 2022
f6298e9
[skip ci] add examples of operations which would raise
Dec 30, 2022
dffef42
[skip ci] note about DataFrame.__setitem__
Dec 30, 2022
9fa7675
[skip ci] notes about dataframe case
Dec 30, 2022
2ce6ff0
[skip ci] remove special-casing of full slice
Jan 3, 2023
4626831
[skip ci] minor fixups
Jan 3, 2023
1217a4e
[skip ci] add examples with boolean masks
Jan 3, 2023
6798a2d
[skip ci] Merge remote-tracking branch 'upstream/main' into pdep6
Jan 13, 2023
a930df1
use more generic indexer in example, clarify the enlargement is out o…
Mar 17, 2023
02bab00
dont call workaround "easy"
Mar 20, 2023
875cf4c
define indexer
Mar 20, 2023
9dcf8d4
clarify
Mar 24, 2023
d15a7c1
wip
Mar 24, 2023
2c214c7
split up examples, assorted cleanups, clarify scope
Mar 24, 2023
1868f3c
mention df.index.intersection
Mar 30, 2023
80841d2
make explicit that option 1 was chosen in this pdep
Apr 6, 2023
0c4bdff
clarify option 3
MarcoGorelli Apr 6, 2023
e6f0c7f
clarify option 2
MarcoGorelli Apr 6, 2023
368ad20
correct "risk annoy" to "risk annoying" so as to not risk annoying re…
MarcoGorelli Apr 6, 2023
35c3f37
Merge remote-tracking branch 'upstream/main' into pdep6
Apr 7, 2023
a0ae1fd
add faq
Apr 8, 2023
af7e0e4
Merge remote-tracking branch 'upstream/main' into pdep6
Apr 9, 2023
a002831
Merge branch 'pdep6' of github.com:MarcoGorelli/pandas into pdep6
Apr 9, 2023
3e220f0
add example with 16.000000000000001 to faq
Apr 11, 2023
0b02317
minor clarification (when constructing it)
Apr 11, 2023
50f0a41
Update web/pandas/pdeps/0006-ban-upcasting.md
MarcoGorelli Apr 11, 2023
cc77562
add example of maybe_convert_to_int function
Apr 11, 2023
9cee2c7
Merge branch 'pdep6' of github.com:MarcoGorelli/pandas into pdep6
Apr 11, 2023
0ff14f1
Merge remote-tracking branch 'upstream/main' into pdep6
Apr 21, 2023
8211f37
change status to accepted
Apr 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 28 additions & 4 deletions web/pandas/pdeps/0006-ban-upcasting.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ In[10]: ser[2] = "2000-01-04" # works, is converted to datetime64
In[11]: ser[2] = "2000-01-04x" # typo - but pandas does not error, it upcasts to object
```

The scope of this PDEP is limited to setitem-like operations on Series.
The scope of this PDEP is limited to setitem-like operations on Series (and DataFrame columns).
For example, starting with
```python
df = DataFrame({"a": [1, 2, np.nan], "b": [4, 5, 6]})
Expand All @@ -71,7 +71,7 @@ then the following would all raise:
- ``df.loc[indexer, 'a'] = 'foo'``
MarcoGorelli marked this conversation as resolved.
Show resolved Hide resolved
- ``ser[indexer] = 'foo'``

It may be desirable to expand the top list to ``replace`` and ``update``,
It may be desirable to expand the top list to ``Series.replace`` and ``Series.update``,
but to keep the scope of the PDEP down, they are excluded for now.

Examples of operations which would not raise are:
MarcoGorelli marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -122,11 +122,29 @@ The trickiest part of this proposal concerns what to do when setting a float in
```python
In[1]: ser = pd.Series([1, 2, 3])

In[2]: ser[0] = 1.5
In [2]: ser
Out[2]:
0 1
1 2
2 3
dtype: int64

In[3]: ser[0] = 1.5 # what should this do?
```

The current behaviour is to upcast to 'float64':
```python
In [4]: ser
Out[4]:
0 1.5
1 2.0
2 3.0
dtype: float64
```

This is not necessarily a sign of a bug, because the user might just be thinking of their ``Series`` as being
numeric (without much regard for ``int`` vs ``float``) - ``'int64'`` is just what pandas happened to infer.
numeric (without much regard for ``int`` vs ``float``) - ``'int64'`` is just what pandas happened to infer
when constructing it.

Possible options could be:
MarcoGorelli marked this conversation as resolved.
Show resolved Hide resolved
1. only accept round floats (e.g. ``1.0``) and raise on anything else (e.g. ``1.01``);
Expand Down Expand Up @@ -190,6 +208,12 @@ at all. To keep this proposal focused, it is intentionally excluded from the sco
**A**: The current behavior would be to upcast to ``int32``. So under this PDEP,
it would instead raise.

**Q: What happens in setting ``16.000000000000001`` in an `int8`` Series?**

**A**: As far as Python is concerned, ``16.000000000000001`` and ``16.0`` are the
same number. So, it would be inserted as ``1`` and the dtype would not change
MarcoGorelli marked this conversation as resolved.
Show resolved Hide resolved
(just like what happens now, there would be no change here).

## Timeline

Deprecate sometime in the 2.x releases (after 2.0.0 has already been released), and enforce in 3.0.0.
Expand Down