Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GroupBy(..., as_index=True).agg() drops index when #28641

Closed
adamhooper opened this issue Sep 26, 2019 · 2 comments · Fixed by #33098
Closed

GroupBy(..., as_index=True).agg() drops index when #28641

adamhooper opened this issue Sep 26, 2019 · 2 comments · Fixed by #33098
Labels
Groupby Needs Tests Unit test(s) needed to prevent regressions
Milestone

Comments

@adamhooper
Copy link
Contributor

adamhooper commented Sep 26, 2019

Code Sample, a copy-pastable example if possible

In [1]: import pandas as pd

In [2]: pd.DataFrame({"A": [1997], "B": pd.Series(["b"], dtype="category").cat.as_ordered()}).groupby("A", as_index=True).agg({"B": "size"})
Out[2]:
      B
A
1997  1

In [3]: pd.DataFrame({"A": [1997], "B": pd.Series(["b"], dtype="category").cat.as_ordered()}).groupby("A", as_index=True).agg({"B": "min"})
Out[3]:
   B
0  b

Problem description

When aggregating min, max or first of a categorical column, .agg() returns a dataframe with a default index instead of the index returned by groupby().

(This may be related to #13416 ... but I think it's a clear, well-defined bug so maybe easier to resolve?)

In my case, I think I can work around this problem with a hack: if I .agg({"B": ["min", "size"]}) and then ignore the ("B", "size") output column, Pandas will output a dataframe with the correct index.

Expected Output

Out[3]:
      B
A
1997  b

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : None python : 3.7.4.final.0 python-bits : 64 OS : Linux OS-release : 5.2.11-200.fc30.x86_64 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 0.25.0
numpy : 1.16.1
pytz : 2018.9
dateutil : 2.8.0
pip : 19.0.3
setuptools : 40.8.0
Cython : 0.29.5
pytest : 4.5.0
hypothesis : None
sphinx : 2.1.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.2.5
html5lib : 1.0.1
pymysql : None
psycopg2 : 2.8.3 (dt dec pq3 ext lo64)
jinja2 : 2.10.1
IPython : 7.2.0
pandas_datareader: None
bs4 : 4.6.3
bottleneck : None
fastparquet : 0.2.1
gcsfs : None
lxml.etree : 4.2.5
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.14.1
pytables : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

adamhooper added a commit to CJWorkbench/groupby that referenced this issue Sep 26, 2019
I discovered pandas-dev/pandas#28641
while testing ... and I fixed it.
@ZOUG
Copy link

ZOUG commented Nov 3, 2019

I've encountered a related issue, in which the aggregation function is called directly and a Categorical object is returned rather than a dataframe.

df = pd.DataFrame({"A": [1997], "B": pd.Series(["b"], dtype="category").cat.as_ordered()})
df.groupby("A")["B"].first()

Output:

[b]
Categories (1, object): [b]

@jbrockmendel jbrockmendel added the Apply Apply, Aggregate, Transform, Map label Feb 25, 2020
phofl added a commit to phofl/pandas that referenced this issue Mar 28, 2020
@jreback jreback added this to the 1.1 milestone Mar 31, 2020
@simonjayhawkins
Copy link
Member

xref #33098

While searching for related issue for my other pull request, I found that this issue must have been fixed with another commit in the past (don't know when this issue was fixed).

@simonjayhawkins simonjayhawkins added Needs Tests Unit test(s) needed to prevent regressions and removed Apply Apply, Aggregate, Transform, Map Groupby labels May 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Groupby Needs Tests Unit test(s) needed to prevent regressions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants