Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLN: enforce deprecation of NDFrame.interpolate with ffill/bfill/pad/backfill methods #57869

Conversation

natmokval
Copy link
Contributor

xref #53607

enforced deprecation of NDFrame.interpolate with "ffill” / "bfill” / “pad” / “backfill" methods

@natmokval natmokval added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Clean labels Mar 17, 2024
@natmokval natmokval marked this pull request as ready for review March 17, 2024 21:16
@@ -7806,14 +7791,9 @@ def interpolate(
fillna_methods = ["ffill", "bfill", "pad", "backfill"]
if method.lower() in fillna_methods:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you just remove this entire if branch? A bad method should raise later on

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for reviewing this PR. I removed this if branch and changed the error message in the if branch below. Just one question: we didn't have a test for the previous ValueError message:

"'fill_value' is not a valid keyword for "
f"{type(self).__name__}.interpolate with method from "
f"{fillna_methods}"

Should I add the test?

obj, should_transpose = (self.T, True) if axis == 1 else (self, False)
# GH#53631
if np.any(obj.dtypes == object):
raise TypeError(
f"{type(self).__name__} cannot interpolate with object dtype."
)

if method in fillna_methods and "fill_value" in kwargs:
else:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this case isn't needed either anymore either. The obj._mgr.interpolate call below should raise on invalid arguments. (Also note the # TODO(3.0): remove this case comment below which you can address in this PR)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I moved this check for the bad methods below and removed the useless check (the one with the note # TODO(3.0): remove this case)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear, here's the diff of the change I think you can fully clean up (I think we don't need any logic dealing with checking for a fillna_method):

--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -7797,30 +7797,11 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
         if not isinstance(method, str):
             raise ValueError("'method' should be a string, not None.")
 
-        fillna_methods = ["ffill", "bfill", "pad", "backfill"]
-        if method.lower() in fillna_methods:
-            # GH#53581
-            warnings.warn(
-                f"{type(self).__name__}.interpolate with method={method} is "
-                "deprecated and will raise in a future version. "
-                "Use obj.ffill() or obj.bfill() instead.",
-                FutureWarning,
-                stacklevel=find_stack_level(),
-            )
-            obj, should_transpose = self, False
-        else:
-            obj, should_transpose = (self.T, True) if axis == 1 else (self, False)
-                    f"{type(self).__name__} cannot interpolate with object dtype."
-                )
-
-        if method in fillna_methods and "fill_value" in kwargs:
-            raise ValueError(
-                "'fill_value' is not a valid keyword for "
-                f"{type(self).__name__}.interpolate with method from "
-                f"{fillna_methods}"
+        obj, should_transpose = (self.T, True) if axis == 1 else (self, False)
+        # GH#53631
+        if np.any(obj.dtypes == object):
+            raise TypeError(
+                f"{type(self).__name__} cannot interpolate with object dtype."
             )
 
         if isinstance(obj.index, MultiIndex) and method != "linear":
@@ -7830,34 +7811,16 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
 
         limit_direction = missing.infer_limit_direction(limit_direction, method)
 
-        if method.lower() in fillna_methods:
-            # TODO(3.0): remove this case
-            # TODO: warn/raise on limit_direction or kwargs which are ignored?
-            #  as of 2023-06-26 no tests get here with either
-            if not self._mgr.is_single_block and axis == 1:
-                # GH#53898
-                if inplace:
-                    raise NotImplementedError()
-                obj, axis, should_transpose = self.T, 1 - axis, True
-
-            new_data = obj._mgr.pad_or_backfill(
-                method=method,
-                axis=self._get_block_manager_axis(axis),
-                limit=limit,
-                limit_area=limit_area,
-                inplace=inplace,
-            )
-        else:
-            index = missing.get_interp_index(method, obj.index)
-            new_data = obj._mgr.interpolate(
-                method=method,
-                index=index,
-                limit=limit,
-                limit_direction=limit_direction,
-                limit_area=limit_area,
-                inplace=inplace,
-                **kwargs,
-            )
+        index = missing.get_interp_index(method, obj.index)
+        new_data = obj._mgr.interpolate(
+            method=method,
+            index=index,
+            limit=limit,
+            limit_direction=limit_direction,
+            limit_area=limit_area,
+            inplace=inplace,
+            **kwargs,
+        )

Copy link
Contributor Author

@natmokval natmokval Mar 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhh, I think I understand now. I removed any logic dealing with checking for a fillna_methods from NDFrame.interpolate.
I think we need to check if method is valid in interpolate in class Block and to do the same check in get_interp_index. I corrected these methods and updated error messages in tests.

@@ -1383,6 +1383,9 @@ def interpolate(
limit_area: Literal["inside", "outside"] | None = None,
**kwargs,
) -> list[Block]:
valid = missing.NP_METHODS + missing.SP_METHODS
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A better place for this check (and the one you added in get_interp_index) in is probably in _interpolate_scipy_wrapper.

We want to make these checks at the lowest level possible

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I moved this check from pandas/core/internals/blocks.py to _interpolate_scipy_wrapper in pandas/core/missing.py.

I left the check in get_interp_index, otherwise, several tests fail.

Comment on lines 592 to 594
valid = NP_METHODS + SP_METHODS
if method not in valid:
raise ValueError(f"Can not interpolate with method={method}.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of this, the terp = alt_methods[method] below should be changed to:

terp = alt_methods.get(method, None):
if terp is None:
    raise ValueError(f"Can not interpolate with method={method}.")

@mroeschke mroeschke added this to the 3.0 milestone Mar 22, 2024
@mroeschke mroeschke merged commit 677a4ea into pandas-dev:main Mar 22, 2024
46 checks passed
@mroeschke
Copy link
Member

Thanks for sticking with this @natmokval!

@natmokval
Copy link
Contributor Author

Thanks @mroeschke for helping me with this PR. Enforcing this deprecation was more complicated than I expected.

pmhatre1 pushed a commit to pmhatre1/pandas-pmhatre1 that referenced this pull request May 7, 2024
…ad/backfill` methods (pandas-dev#57869)

* enforce deprecation of interpolate with ffill, bfill-pad, backfill methods

* remove redundant if branch

* remove unuseful cheek from interpolate

* move checking for a fillna_method from NDFrame.interpolate to Block.interpolate, correct tests

* remove the check from Block.interpolate

* add a note to v3.0.0

* correct def _interpolate_scipy_wrapper: use alt_methods instead of valid
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Clean Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants