Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resample function wrong interpolation #21418

Open
peppusnight opened this issue Jun 11, 2018 · 2 comments
Open

Resample function wrong interpolation #21418

peppusnight opened this issue Jun 11, 2018 · 2 comments
Labels
Bug Resample resample method

Comments

@peppusnight
Copy link

Code Sample, a copy-pastable example if possible

import pandas as pd
from datetime import datetime
import pytz
import numpy as np
import matplotlib.pyplot as plt

def convert_2_datetime(timestamp, timezoneid):
    """

    :param timestamp: UTC format in milliseconds (data.index.values)
    :param timezoneid: timezone object from CTX (for example pytz.timezone(ctx.inp.assets[0].properties['timezoneid']))
    :return: vector of datetimes
    """

    if isinstance(timestamp,int) or isinstance(timestamp,float):
        utctime = datetime.utcfromtimestamp(timestamp / 1000).replace(tzinfo=pytz.utc)
        output = utctime.astimezone(pytz.timezone(timezoneid.zone))
    else:
        utctime = [datetime.utcfromtimestamp(i / 1000).replace(tzinfo=pytz.utc) for i in timestamp]
        output = [i.astimezone(pytz.timezone(timezoneid.zone)) for i in utctime]

    return output

# minute sampled data
v1 = [0.0,0.0,0.0,0.0,0.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,0.0,0.0,0.0,0.0,0.0]
data = pd.DataFrame({'v1':np.array(v1)},index=np.arange(start=1,stop=len(v1)+1)*60000)

data['ts']= convert_2_datetime(timestamp=data.index.values,timezoneid=pytz.timezone('UTC'))
data.set_index('ts',inplace=True)
data07 = data.resample(rule='700L',closed={'right','left'}).interpolate(method='linear')
data06 = data.resample(rule='600L',closed={'right','left'}).interpolate(method='linear')

plt.show()

data07.v1.plot(style='.',label='700 ms')
data06.v1.plot(style='.', label = '600 ms')
data.v1.plot(style='x',label='original')

plt.legend()

print('Finish!')

Problem description

In the code above, data07 does not start from the same time data06 (and data) starts. The plot is showing that also the behavior of interpolation on data07 is very different from the one with data06.
I upgraded pandas to the last version but the problem persists.

This is the link where the question has been asked on Stackoverflow:
https://stackoverflow.com/questions/50724231/pandas-resample-function-issue-from-minute-to-millisecond-ersampling

Expected Output

Expected output should be equal to data06 (interpolation between points of data).

Output of pd.show_versions()

I tried with 0.19.2 and 0.23.0 (here below I put the output of 0.23.0).

INSTALLED VERSIONS

commit: None
python: 3.6.1.final.0
python-bits: 32
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 63 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.23.0
pytest: 3.0.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.14.4
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 5.3.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.2.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@oricou
Copy link

oricou commented Feb 17, 2021

I just wrote a fix to this issue in Time Series Interpolation is wrong #21351

If someone could make a Pull request, it would be nice.

@MarcoGorelli
Copy link
Member

If someone could make a Pull request, it would be nice.

Thanks @oricou - I haven't looked at this yet, but why not make a pull request yourself? Here's the contributing guide

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Resample resample method
Projects
None yet
Development

No branches or pull requests

5 participants