Explode Series with Dask-cuDF

[sarahyurick]

Addresses feature request #8660. Invoking explode() on a cuDF or Dask Series outputs a DataFrame.

Similar to #8729, except now instead of having to call a_cudf_series.struct.explode(), we can do just a_cudf_series.explode(). Added Dask functionality allows for a_dask_cudf_series.compute().explode().

Minor detail - I'm not a huge fan of how I dealt with datatypes here. In order to obtain the values within the Series, I use to_arrow() (as seen in #8675), but I need to convert the PyArrow datatypes back to the original datatypes. So I convert them to strings and after creating the DataFrame, iterate through the columns to recast them.

[beckernick]

What is the structure of the out you want here? The device to host transfer + nested loop may be quite slow

[sarahyurick]

the idea was that it would essentially look the same as the struct.explode() functionality that's already been implemented, so like:

[beckernick]

You may be able to accomplish that without going to the CPU:

import cudf
​
s = cudf.Series([
    {"a":5, "b":10},
    {"a":3, "b":7},
    {"a":-3, "b":11},
])
​
results = []
for key in s.dtype.fields:
    results.append(s.struct.field(key))
​
out = cudf.concat(results, axis=1)
out.columns = s.dtype.fields
print(out)
   a   b
0  5  10
1  3   7
2 -3   11

Separately, we may want to think more broadly about what we want the behavior to be for functionality that "kind of" exists in pandas. Pandas doesn't have struct columns, but does allow exploding of an object column containing dictionaries. However, the explode does not behave like exploding a struct column in Hive, Spark, etc. Instead, it behaves like exploding a list column (which it doesn't technically have either), where every element becomes a new row in a single column. This is a traditional, rather than a lateral, explode.

Because of that, we might need to special case an explode operator in dask-cuDF anyway, rather than rely on Dask to appropriately delegate to the cuDF explode from the existing one in Dask.DataFrame.

For now, I'd suggest we consider holding off on series.explode() natively doing a "lateral explode" for struct columns, and instead building the lateral view explode functionality as dask_series.struct.explode() once we land #8658

[beckernick]

cc @shwina @VibhuJawa (as they might disagree)

[shwina]

I agree, I don't think we want the Series.explode() to support this just yet (unless Pandas does so). dask_series.struct.explode() can work off of series.struct.explode().

[sarahyurick]

in Pandas, you just get something like:

import pandas as pd

s = pd.Series([{"a": 1, "b": "x"},
                {"a": 2, "b": "y"},
                {"a": 3, "b": "z"},
                {"a": 4, "b": "a"}])
s.explode()
0    a
0    b
1    a
1    b
2    a
2    b
3    a
3    b
dtype: object

[beckernick]

Thanks for the example 👍 .

This is the "traditional explode" mentioned above and what we do with lists. The "lateral explode" is particularly common for structs, as the field names often map to actual features.

We may want to support this kind of traditional explode for structs, but in general this would be less common

[sarahyurick]

sounds good - I'll close this PR then, as it has a different functionality than the desired dask_series.struct.explode()