Any plan to include cuDF as backend?

[argenisleon]

Hi,

Is there any plan to include cuDF as an Ibis backend option?

[gforsyth]

Hi @argenisleon -- we don't have it on our roadmap, currently. No opposition to the idea at all, and a cudf backend could likely take advantage of a bunch of the existing stuff in the pandas backend, but we're not currently working on that. (One tricky bit here would be setting up GPU-enabled CI for such a backend)

[ogrisel]

Maybe you should contact the cuDF developers themselves? Arguably they have the CI infra at end and probably know very well the performance details impacts of mapping high level ibis operations to concrete GPU specific implementations.

I would be really interested in seeing how GPU-enabled ibis-cuDF fairs against the highly CPU optimized duckdb or clickhouse backends for common analytical workfloads.

[rjzamora]

I wonder if it might be possible to support cudf through the existing Dask backend without adding a new cudf backend. For example, it seems like one can already use Ibis to interact with a dask_cudf DataFrame:

In [1]: import ibis
   ...: import dask
   ...: import dask_cudf
In [2]: ddf = dask.datasets.timeseries()
   ...: ddf = dask_cudf.from_dask_dataframe(ddf)
In [3]: # Can only handle numerical data (for now)
   ...: ddf = ddf.reset_index(drop=True)[["id", "x", "y"]]
In [4]: con = ibis.dask.connect({"df": ddf})
In [5]: table = con.table("df")
In [6]: agg = table.group_by("id").aggregate(table["x"].sum().name("x_sum"))
In [7]: agg.execute()
Out[7]: 
       id       x_sum
0    1124   -0.348288
1    1137    0.720640
2     869    0.221074
3    1098    2.916699
4    1031  159.277703
..    ...         ...
290  1011 -162.496827
291   942   57.536960
292  1051  -11.014909
293   993   99.983848
294   907   -5.328226

[295 rows x 2 columns]

Do any Ibis developers know how much work it would be to make the Dask backend (mostly) pandas/cudf agnostic? (Disclaimer: I am completely new to Ibis, so this question may be totally naive)

cc @jcrist @cpcloud @kszucs (Sorry for the ping, but you all seem active in the Dask backend)