BUG: Handle nonexistent end dates when resampling #59471
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rhshadrach
requested changes
Aug 11, 2024
| expected = Series([len(ts)], index=DatetimeIndex([ts.index[0]], freq="7D")) | ||
|
|
||
| result = ts.resample("7D").sum() | ||
| print(f"{result=}") |
There was a problem hiding this comment.
Oops, sorry about that!
doc/source/whatsnew/v3.0.0.rst
Outdated
| @@ -615,6 +615,7 @@ Groupby/resample/rolling | |||
| - Bug in :meth:`DataFrameGroupBy.cumsum` where it did not return the correct dtype when the label contained ``None``. (:issue:`58811`) | |||
| - Bug in :meth:`DataFrameGroupby.transform` and :meth:`SeriesGroupby.transform` with a reducer and ``observed=False`` that coerces dtype to float when there are unobserved categories. (:issue:`55326`) | |||
| - Bug in :meth:`Rolling.apply` where the applied function could be called on fewer than ``min_period`` periods if ``method="table"``. (:issue:`58868`) | |||
| - Bug in :meth:`Series.resample` could raise when the the date range ended shortly before DST. (:issue:`58380`) | |||
There was a problem hiding this comment.
Not just before DST, but more generally before a non-existent time, right?
mroeschke
reviewed
Aug 12, 2024
| @@ -2466,7 +2466,7 @@ def _get_timestamp_range_edges( | |||
| ) | |||
| if isinstance(freq, Day): | |||
| first = first.tz_localize(index_tz) | |||
| last = last.tz_localize(index_tz) | |||
| last = last.tz_localize(index_tz, nonexistent="shift_forward") | |||
There was a problem hiding this comment.
Should nonexistent be dependent on closed?
There was a problem hiding this comment.
I don't think last here is used as the last element in the series. Instead, last needs to be strictly after all the data points:
Lines 893 to 894 in 0fadaa9
Shifting backwards should raise when running near a nonexistent hour, in the rare case where the nanosecond before is in the index:
import pandas as pd
almost_a_day = pd.Timedelta(days=1) - pd.Timedelta(nanoseconds=1)
timestamp = pd.to_datetime("2024-04-25").tz_localize("Africa/Cairo")
ts = pd.Series(
1,
[timestamp, timestamp + almost_a_day],
)
ts.resample("1D", closed="right").sum()However, shifting forward runs with the following output:
2024-04-24 00:00:00+02:00 1
2024-04-25 00:00:00+02:00 1
Freq: D, dtype: int64
(I'm not sure why it gives a bin on 2024-04-24.)
mroeschke
approved these changes
Aug 19, 2024
|
Thanks @rob-sil! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
doc/source/whatsnew/v3.0.0.rstfile if fixing a bug or adding a new feature.