-
-
Notifications
You must be signed in to change notification settings - Fork 18.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CLN: enforce deprecation of NDFrame.interpolate
with ffill/bfill/pad/backfill
methods
#57869
CLN: enforce deprecation of NDFrame.interpolate
with ffill/bfill/pad/backfill
methods
#57869
Conversation
pandas/core/generic.py
Outdated
@@ -7806,14 +7791,9 @@ def interpolate( | |||
fillna_methods = ["ffill", "bfill", "pad", "backfill"] | |||
if method.lower() in fillna_methods: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you just remove this entire if
branch? A bad method
should raise later on
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for reviewing this PR. I removed this if
branch and changed the error message in the if
branch below. Just one question: we didn't have a test for the previous ValueError
message:
"'fill_value' is not a valid keyword for "
f"{type(self).__name__}.interpolate with method from "
f"{fillna_methods}"
Should I add the test?
pandas/core/generic.py
Outdated
obj, should_transpose = (self.T, True) if axis == 1 else (self, False) | ||
# GH#53631 | ||
if np.any(obj.dtypes == object): | ||
raise TypeError( | ||
f"{type(self).__name__} cannot interpolate with object dtype." | ||
) | ||
|
||
if method in fillna_methods and "fill_value" in kwargs: | ||
else: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this case isn't needed either anymore either. The obj._mgr.interpolate
call below should raise on invalid arguments. (Also note the # TODO(3.0): remove this case
comment below which you can address in this PR)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks, I moved this check for the bad methods
below and removed the useless check (the one with the note # TODO(3.0): remove this case
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to be clear, here's the diff of the change I think you can fully clean up (I think we don't need any logic dealing with checking for a fillna_method
):
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -7797,30 +7797,11 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
if not isinstance(method, str):
raise ValueError("'method' should be a string, not None.")
- fillna_methods = ["ffill", "bfill", "pad", "backfill"]
- if method.lower() in fillna_methods:
- # GH#53581
- warnings.warn(
- f"{type(self).__name__}.interpolate with method={method} is "
- "deprecated and will raise in a future version. "
- "Use obj.ffill() or obj.bfill() instead.",
- FutureWarning,
- stacklevel=find_stack_level(),
- )
- obj, should_transpose = self, False
- else:
- obj, should_transpose = (self.T, True) if axis == 1 else (self, False)
- f"{type(self).__name__} cannot interpolate with object dtype."
- )
-
- if method in fillna_methods and "fill_value" in kwargs:
- raise ValueError(
- "'fill_value' is not a valid keyword for "
- f"{type(self).__name__}.interpolate with method from "
- f"{fillna_methods}"
+ obj, should_transpose = (self.T, True) if axis == 1 else (self, False)
+ # GH#53631
+ if np.any(obj.dtypes == object):
+ raise TypeError(
+ f"{type(self).__name__} cannot interpolate with object dtype."
)
if isinstance(obj.index, MultiIndex) and method != "linear":
@@ -7830,34 +7811,16 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
limit_direction = missing.infer_limit_direction(limit_direction, method)
- if method.lower() in fillna_methods:
- # TODO(3.0): remove this case
- # TODO: warn/raise on limit_direction or kwargs which are ignored?
- # as of 2023-06-26 no tests get here with either
- if not self._mgr.is_single_block and axis == 1:
- # GH#53898
- if inplace:
- raise NotImplementedError()
- obj, axis, should_transpose = self.T, 1 - axis, True
-
- new_data = obj._mgr.pad_or_backfill(
- method=method,
- axis=self._get_block_manager_axis(axis),
- limit=limit,
- limit_area=limit_area,
- inplace=inplace,
- )
- else:
- index = missing.get_interp_index(method, obj.index)
- new_data = obj._mgr.interpolate(
- method=method,
- index=index,
- limit=limit,
- limit_direction=limit_direction,
- limit_area=limit_area,
- inplace=inplace,
- **kwargs,
- )
+ index = missing.get_interp_index(method, obj.index)
+ new_data = obj._mgr.interpolate(
+ method=method,
+ index=index,
+ limit=limit,
+ limit_direction=limit_direction,
+ limit_area=limit_area,
+ inplace=inplace,
+ **kwargs,
+ )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ahhh, I think I understand now. I removed any logic dealing with checking for a fillna_methods
from NDFrame.interpolate
.
I think we need to check if method
is valid in interpolate
in class Block
and to do the same check in get_interp_index
. I corrected these methods and updated error messages in tests.
pandas/core/internals/blocks.py
Outdated
@@ -1383,6 +1383,9 @@ def interpolate( | |||
limit_area: Literal["inside", "outside"] | None = None, | |||
**kwargs, | |||
) -> list[Block]: | |||
valid = missing.NP_METHODS + missing.SP_METHODS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A better place for this check (and the one you added in get_interp_index
) in is probably in _interpolate_scipy_wrapper
.
We want to make these checks at the lowest level possible
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks, I moved this check from pandas/core/internals/blocks.py
to _interpolate_scipy_wrapper
in pandas/core/missing.py
.
I left the check in get_interp_index
, otherwise, several tests fail.
pandas/core/missing.py
Outdated
valid = NP_METHODS + SP_METHODS | ||
if method not in valid: | ||
raise ValueError(f"Can not interpolate with method={method}.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of this, the terp = alt_methods[method]
below should be changed to:
terp = alt_methods.get(method, None):
if terp is None:
raise ValueError(f"Can not interpolate with method={method}.")
Thanks for sticking with this @natmokval! |
Thanks @mroeschke for helping me with this PR. Enforcing this deprecation was more complicated than I expected. |
…ad/backfill` methods (pandas-dev#57869) * enforce deprecation of interpolate with ffill, bfill-pad, backfill methods * remove redundant if branch * remove unuseful cheek from interpolate * move checking for a fillna_method from NDFrame.interpolate to Block.interpolate, correct tests * remove the check from Block.interpolate * add a note to v3.0.0 * correct def _interpolate_scipy_wrapper: use alt_methods instead of valid
xref #53607
enforced deprecation of
NDFrame.interpolate
with"ffill” / "bfill” / “pad” / “backfill"
methods