ENH: ColumnTransformer

[TomAugspurger]

This PR implements a daskified column transformer.

There are two issues preventing us from just using scikit-learn's

1. dask DataFrame doesn't implement .shape ( Add (lazy) shape property to dataframe and series dask#3212)
2. sklearn.compose.compose._column_transformer._hstack doesn't handle dask objects (or pandas dataframes). Just sparse objects and ndarrays. The _hstack implemented here handles arrays (dask or numpy) dataframes (dask or numpy) and sparse matricies.

In [1]: import dask.dataframe as dd
   ...: import pandas as pd
   ...: import sklearn.compose
   ...: import sklearn.preprocessing
   ...: from sklearn.base import clone
   ...:
   ...: import dask_ml.compose
   ...: import dask_ml.preprocessing
   ...:
   ...: df = pd.DataFrame({"A": pd.Categorical(["a", "a", "b", "a"]), "B": [1.0, 2, 4, 5]})
   ...: ddf = dd.from_pandas(df, npartitions=2)
   ...:
   ...:

In [2]:     b = dask_ml.compose.make_column_transformer(
   ...:         (["A"], dask_ml.preprocessing.OneHotEncoder(sparse=False)),
   ...:         (["B"], dask_ml.preprocessing.StandardScaler()),
   ...:     )
   ...:
   ...:

In [3]: result = b.fit_transform(ddf)

In [4]: result
Out[4]:
Dask DataFrame Structure:
                   A_a      A_b        B
npartitions=2
0              float64  float64  float64
2                  ...      ...      ...
3                  ...      ...      ...
Dask Name: concat-indexed, 22 tasks

In [5]: result.compute()
Out[5]:
   A_a  A_b         B
0  1.0  0.0 -1.264911
1  1.0  0.0 -0.632456
2  0.0  1.0  0.632456
3  1.0  0.0  1.264911

Long-term, it'd be nice to remove this class entirely, but that'll probably require a lot of work upstream (scipy adopting pydata/sparse, NumPy implementing and libraries adopting __array_function__).

Medium-term, _hstack could become a staticmethod on ColumnTransformer. Then this subclass would just override _hstack, and everything else could be removed.

cc @jorisvandenbossche @ogrisel for that last point. Should I open an issue on scikit-learn to discuss that further?

[ogrisel]

Medium-term, _hstack could become a staticmethod on ColumnTransformer. Then this subclass would just override _hstack, and everything else could be removed.

cc @jorisvandenbossche @ogrisel for that last point. Should I open an issue on scikit-learn to discuss that further?

+1 for opening an issue on scikit-learn and discuss your suggestion (or even a pull request).

[ogrisel]

If you do a quick sklearn PR it could be part of the 0.20 release.

[TomAugspurger]

Will do quick.

…

On Thu, Jul 26, 2018 at 11:21 AM Olivier Grisel ***@***.***> wrote: If you do a quick sklearn PR it could be part of the 0.20 release. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#315 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIlMN2FOutbEgJOYPUBFs2erbTXH1ks5uKeyWgaJpZM4Vgwlc> .

[TomAugspurger]

Scikit-Learn PR at scikit-learn/scikit-learn#11689

This passes locally for me, but won't pass here till dask/dask#3212 and scikit-learn/scikit-learn#11689 are done.

[TomAugspurger]

The upstream PRs are in. Merging later today.

[TomAugspurger]

Ignoring the coverage failure, since coverage isn't run against sklearn dev.