Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PDEP-6: Ban upcasting in setitem-like operations #50424

Merged
merged 33 commits into from
Apr 21, 2023
Merged
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
1fb0adb
[skip ci] pdep6 draft
Dec 22, 2022
5456787
[skip ci] reword
Dec 24, 2022
02ff735
[skip ci] compare with DataFrames.jl
Dec 24, 2022
e3cc381
[skip ci] note about loss of precision
Dec 24, 2022
f6298e9
[skip ci] add examples of operations which would raise
Dec 30, 2022
dffef42
[skip ci] note about DataFrame.__setitem__
Dec 30, 2022
9fa7675
[skip ci] notes about dataframe case
Dec 30, 2022
2ce6ff0
[skip ci] remove special-casing of full slice
Jan 3, 2023
4626831
[skip ci] minor fixups
Jan 3, 2023
1217a4e
[skip ci] add examples with boolean masks
Jan 3, 2023
6798a2d
[skip ci] Merge remote-tracking branch 'upstream/main' into pdep6
Jan 13, 2023
a930df1
use more generic indexer in example, clarify the enlargement is out o…
Mar 17, 2023
02bab00
dont call workaround "easy"
Mar 20, 2023
875cf4c
define indexer
Mar 20, 2023
9dcf8d4
clarify
Mar 24, 2023
d15a7c1
wip
Mar 24, 2023
2c214c7
split up examples, assorted cleanups, clarify scope
Mar 24, 2023
1868f3c
mention df.index.intersection
Mar 30, 2023
80841d2
make explicit that option 1 was chosen in this pdep
Apr 6, 2023
0c4bdff
clarify option 3
MarcoGorelli Apr 6, 2023
e6f0c7f
clarify option 2
MarcoGorelli Apr 6, 2023
368ad20
correct "risk annoy" to "risk annoying" so as to not risk annoying re…
MarcoGorelli Apr 6, 2023
35c3f37
Merge remote-tracking branch 'upstream/main' into pdep6
Apr 7, 2023
a0ae1fd
add faq
Apr 8, 2023
af7e0e4
Merge remote-tracking branch 'upstream/main' into pdep6
Apr 9, 2023
a002831
Merge branch 'pdep6' of github.com:MarcoGorelli/pandas into pdep6
Apr 9, 2023
3e220f0
add example with 16.000000000000001 to faq
Apr 11, 2023
0b02317
minor clarification (when constructing it)
Apr 11, 2023
50f0a41
Update web/pandas/pdeps/0006-ban-upcasting.md
MarcoGorelli Apr 11, 2023
cc77562
add example of maybe_convert_to_int function
Apr 11, 2023
9cee2c7
Merge branch 'pdep6' of github.com:MarcoGorelli/pandas into pdep6
Apr 11, 2023
0ff14f1
Merge remote-tracking branch 'upstream/main' into pdep6
Apr 21, 2023
8211f37
change status to accepted
Apr 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 17 additions & 14 deletions web/pandas/pdeps/0006-ban-upcasting.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,25 +50,28 @@ In [10]: ser[2] = '2000-01-04' # works, is converted to datetime64
In [11]: ser[2] = '2000-01-04x' # almost certainly a typo - but pandas doesn't error, it upcasts to object
```

The scope of this PDEP is limited to setitem-like operations which would operate inplace, such as:
- ``ser[0] = 2.5``;
The scope of this PDEP is limited to setitem-like operations on Series.
MarcoGorelli marked this conversation as resolved.
Show resolved Hide resolved
For example, starting with
```python
df = DataFrame({'a': [1, 2, np.nan], 'b': [4, 5, 6]})
ser = df['a'].copy()
```
then the following would all raise:
MarcoGorelli marked this conversation as resolved.
Show resolved Hide resolved
- ``ser[0] = 'foo'``;
- ``ser.fillna('foo', inplace=True)``;
- ``ser.where(ser.isna(), 'foo', inplace=True)``
- ``ser.iloc[0] = 2.5``
- ``ser.loc[0] = 2.5``
- ``ser[:] = 2.5``
- ``ser.iloc[0] = 'foo'``
- ``ser.loc[0] = 'foo'``
- ``df.loc[0, 'a'] = 'foo'``

There may be more. What is explicitly excluded from this PDEP is any operation would have no change
of operating inplace to begin with, such as:
Examples of operations which would not raise are:
MarcoGorelli marked this conversation as resolved.
Show resolved Hide resolved
- ``ser.diff()``;
- ``pd.concat([ser, other])``;
- ``pd.concat([ser, ser.astype(object)])``;
- ``ser.mean()``;
- ``df.loc[0, 'col1'] = 2.5`` (if ``df`` is not a single block).
- ``df.loc[:, 'a'] = 'foo'`` (debatable, as is the one below)
- ``ser[:] = 'foo'``
MarcoGorelli marked this conversation as resolved.
Show resolved Hide resolved

These would keep being allowed to change Series' dtypes. Note that setting element of a column of a
``DataFrame`` would not raise, as that sets the elements in a new block manager (rather than in the
original one),
see https://github.com/pandas-dev/pandas/blob/4e4be0bfa8f74b9d453aa4163d95660c04ffea0c/pandas/core/internals/managers.py#L1361-L1362.
These would keep being allowed to change Series' dtypes.

## Detailed description

Expand All @@ -91,7 +94,7 @@ For a start, this would involve:
else:
```

2. making a similar change in ``Block.where``, ``Block.putmask``, and likewise for ``EABlock`` (and possibly in more places).
2. making a similar change in ``Block.where``, ``Block.putmask``, ``EABackedBlock.where``, and ``EABackedBlock.putmask``.

The above would already require several hundreds of tests to be adjusted.

Expand Down