[Discourse] Using pivot_table with non-numerical data

[github-actions]

What would be the best way to do a pivot_table on a dataframe containing non-numerical data? I can do it with pandas for smaller dataframes, but my current data is too large to fit in memory.

Example:

raw_data_pd = pd.DataFrame({"ID": [1, …

---

Would you like to know more?

Read the full article on the following website:

https://dask.discourse.group/t/using-pivot-table-with-non-numerical-data/267

[scharlottej13]

sticking some notes here for myself:

• you can get close w/ ddf.groupby(['Col_ID', 'ID']).aggregate('first').compute()
• I don't think Dask supports a multi-index yet, but seems like there is work being done on this?
• ddf.map_partitions(lambda x: x.pivot_table(index="ID", columns="Col_ID", values="value", aggfunc="first")).compute() throws ValueError: The columns in the computed data do not match the columns in the provided metadata Extra: ['B'] Missing: [] I think b/c of the lack of support for a multi-index

[pavithraes]

@scharlottej13 Thanks for answering this! I asked Ian about this last week (because we seem to have worked on the same issue, oops! -- I'll close that one!), and in addition to your answer, we can encourage them to open an issue on dask/dask requesting first and last for pivot_table!

[scharlottej13]

because we seem to have worked on the same issue, oops! -- I'll close that one!)

Sorry about that! I will be better about checking first to make sure it's not a duplicate! Thank you for closing the other one.

and in addition to your answer, we can encourage them to open an issue on dask/dask requesting first and last for pivot_table!

Ah great idea, I'll reply with this as well.

[scharlottej13]

• before closing, open up issue in dask/dask for first and last if the question poster does not

[pavithraes]

The user opened this issue: https://github.com/dask/dask/issues/8618 🎉