feat(api): add `to_pandas_batches` #7123

cpcloud · 2023-09-11T19:39:58Z

Adds a to_pandas_batches for returning an iterator of pandas DataFrames from expression execution. Closes #6805.

cpcloud · 2023-09-12T15:52:05Z

~~I'd like to add some more tests before this is merged.~~

Added some more tests, fixed up issues.

jcrist · 2023-09-28T15:02:49Z

ibis/backends/snowflake/__init__.py

+        params: Mapping[ir.Scalar, Any] | None = None,
+        limit: int | str | None = None,
+        **_: Any,
+    ) -> pa.ipc.RecordBatchReader:


Incorrect type annotation

jcrist · 2023-09-28T15:02:59Z

ibis/expr/types/core.py

+        params: Mapping[ir.Value, Any] | None = None,
+        chunk_size: int = 1_000_000,
+        **kwargs: Any,
+    ) -> pa.ipc.RecordBatchReader:


jcrist · 2023-09-28T15:04:41Z

ibis/backends/base/__init__.py

+        schema = expr.schema()
+        yield from (
+            orig_expr.__pandas_result__(
+                self.pandas_converter.convert_table(batch.to_pandas(), schema)


IIUC given this default definition, the PandasData converter should work for dask as well, since it's always working on direct pandas data. No need for the pandas_converter attribute at all, can just use PandasData directly.

Yep, removed the attribute.

jcrist · 2023-09-28T15:07:00Z

ibis/backends/base/__init__.py

@@ -223,6 +224,8 @@ def _ipython_key_completions_(self) -> list[str]:


 class _FileIOHandler:
+    pandas_converter = PandasData


Slight preference for this to be a private attribute if we keep this, just so it doesn't show up in tab-completion for users.

Removed the attribute entirely.

cpcloud · 2023-09-28T15:12:42Z

Clouds are passing:

…/ibis on  to-pandas-batches is 📦 v6.1.0 via 🐍 v3.10.12 via ❄️  impure (ibis-3.10.12-env)
❯ pytest -m 'bigquery or snowflake' --snapshot-update --allow-snapshot-deletion -n auto -q
bringing up nodes...
........xx..x..x..........x........xx.xx....xxx..x.x.....x.........xxx....x........xx.............xx..x..x......xx...xx.x............xx......x.xxx.............x.x.x.............................x.x.. [  7%]
...........x.x...x...x...x..x...xx.x.........x......x...xxx.............x..x.x.....x..x..x..x......x...x........x...x........x......x..x......x.....xx.....x.......x....x....x....x...............x... [ 14%]
........xx....x.....x...................x.x...x...x.x.xxxxx.xxxxxx.xssssx.x.x.....x.sx...x.xxx...x......x..........x........x......x...x........x..x..............xx..............x......x..x..xx...x. [ 21%]
.................xx.........xx.x..............x.xx.....x..x.xx..x..........................x.x..x...x................................................................................................. [ 28%]
......x....x.....x.........x...x.............xxx.......x....x...x.......xx...x..........xx..xxx...x.......x........x....x.....x..........xx......................xx......xx.x..x.......x..xx.......... [ 35%]
...................x.x..........x........x.....x........x...x............x...x....xx......x...................xx..x.......sss...............x.........x....x.x..x............x..xx.x.......x....x..... [ 42%]
..........x..x............xxx.....................x...x...x..x............x.......x.x..x...x.x.x..sx...xx.x.x.x..x..x..........x..x............x.xx.xx.....x.x.....x.x.xxxx......x..x.....x..........x [ 49%]
x................xx...x....xxx...xxx.x...x....x...................x..x...x.....x........x....xx..x..xx.xx..xx...x.x...x.x..x..........x......x...xx.xxxx.xxx...xx.xxxx....x.........x..xxxx.xxxxxxxsss [ 56%]
ssssss.....x....x..........................x......x...x.x.x....x...xxxx.......x...xx.x.ss......xx.....x....s.......x..........x....sx.xs..x..........x.x........x.......x..x......x......x.x....x..... [ 63%]
...s...xx....x..x..xsx.x.xs.x...x.......x....................................s..x...........x...xx..........................sx....x..........xx....ssss.xx..xx.x.x..x..x...x...xx.xx.xxxxx.......x.... [ 70%]
......xxxx.............x...x.xx..xxxs.....x......................................x.............x..xss.....x...........x.xx.x.x........x.x......x......x............................................... [ 77%]
...................x................................................................x................................................x....x..........................x...x...x.........x..........x... [ 84%]
........................................x..x.x..........................x..........x..x...............x..x.x..x..........s....x......x...xx.xxx.....................xx.......x.....x.................. [ 91%]
..............................x...x.................................................................................................................................................s..s............... [ 98%]
.....................................xx...........x.                                                                                                                                                   [100%]
2349 passed, 38 skipped, 438 xfailed in 425.92s (0:07:05)

gforsyth

documentation nits, but this looks good to me

ibis/backends/base/__init__.py

ibis/expr/types/core.py

cpcloud · 2023-09-28T15:40:38Z

Clouds are good after the most recent commit:

…/ibis on  to-pandas-batches is 📦 v6.1.0 via 🐍 v3.10.12 via ❄️  impure (ibis-3.10.12-env)
❯ pytest -m 'bigquery or snowflake' --snapshot-update --allow-snapshot-deletion -n auto -q
bringing up nodes...
ssssxxxx..........xx.xxxxxxxxxxxxx.sxxxxxxxxxxxxxxsxxx....xxxxxxxxxxxxxxxxxxxxxxxxxxss..x............x..x.....x.xx....x..............x...........x......x......xx.....xx..x...x.xx..xxx.x...x....s.... [  7%]
.x...x........x.x.x..............x...x.sx..xxxx...xxxx....x..x.xxxxx......x.x.x.xx.x....xx.xx.x...x....x..xx......x....xx.....x..x.x.xxx.xx.....x.xx.....xx.....................x..x..x.xx............ [ 14%]
.......x...x..x...............xxx..x.....x......x...x...x..x.x.x..x...s.x.x.xxssss..x..............x.x.x....x..x.x..x................x..............x..........x.....x............x..xx.............ss [ 21%]
ss.............x....x.x.x....x....................................x....x.......xx.........x...............................x...x.........s.xxx.x.x..x.....xx...x.....x...xxx...x...x.....x...x....s...x [ 28%]
........x...........x.xx.x.x..s...x..................x....x...................x..............x......x...x..............x..........x.............x......................x..xx......x..x...x.x...x...... [ 35%]
.........xx.x...x.......x.x...x.....x...........x.x.....x.x..............ssss.x....x....ssxx.....s....x........x..xx..............x.......x.......x..x................xx...x....xx.x.xxxx...xx....x... [ 42%]
......x....x....x.....x..x...........x......x...x...x.......x.....x.s....x....x...x..x.....x....s...x...x......xx.......x.x...xx..x..x......xx......xx...........xx.........xs........x......x.x...... [ 49%]
..x....x.sssx..x.....x...............................x.x.x.x..xx...x..x....xx.....xx..s.x..........x.....x..........x..xx..x.x........x.x........................x....x.........xx...x...xxx..x.x..... [ 56%]
..........x.x......x.........xxx....x.......x..............x......x....xx.....x.....x.......x..x...................................x........x.......x..x......x..........x...........x......xx......x. [ 63%]
......x.....x.x.x....xx..............x.................x.x.x..........xx........x....x.........x.x.x...x.x............x....x....................x............x.........x.......x.x.x.....x............ [ 70%]
...x.....x....x.....x...xx.x.x..............x............................x.....x...x.x.x..................x...........................................................x............................... [ 77%]
......x.........................x..x........x.x..x...x.............................x......s.................x.........x............................x.................................................. [ 84%]
.......................x...............................................................................x................x...............x.xx..x..x..............................x...................xx [ 91%]
.................................................x.................................................................................................................................................... [ 98%]
.........sx........................x.......x.........                                                                                                                                                  [100%]
2349 passed, 38 skipped, 438 xfailed in 447.58s (0:07:27)

Co-authored-by: Gil Forsyth <[email protected]>

cpcloud added this to the 7.0 milestone Sep 11, 2023

cpcloud added feature Features or general enhancements ux User experience related issues labels Sep 11, 2023

cpcloud force-pushed the to-pandas-batches branch from 7c8766e to 45001a7 Compare September 11, 2023 19:56

cpcloud changed the title ~~to pandas batches~~ feat(api): add to_pandas_batches Sep 11, 2023

cpcloud marked this pull request as draft September 12, 2023 15:51

cpcloud removed this from the 7.0 milestone Sep 28, 2023

cpcloud force-pushed the to-pandas-batches branch from 45001a7 to 7c23aa9 Compare September 28, 2023 14:52

cpcloud marked this pull request as ready for review September 28, 2023 14:52

cpcloud force-pushed the to-pandas-batches branch from 7c23aa9 to 08bf5bd Compare September 28, 2023 14:55

cpcloud requested a review from jcrist September 28, 2023 14:55

cpcloud added this to the 7.0 milestone Sep 28, 2023

jcrist reviewed Sep 28, 2023

View reviewed changes

cpcloud force-pushed the to-pandas-batches branch from 20770a4 to 0b6f637 Compare September 28, 2023 15:17

jcrist approved these changes Sep 28, 2023

View reviewed changes

gforsyth reviewed Sep 28, 2023

View reviewed changes

ibis/backends/base/__init__.py Outdated Show resolved Hide resolved

ibis/backends/base/__init__.py Outdated Show resolved Hide resolved

ibis/backends/base/__init__.py Outdated Show resolved Hide resolved

ibis/expr/types/core.py Outdated Show resolved Hide resolved

cpcloud and others added 2 commits September 28, 2023 11:52

feat(api): add to_pandas_batches

c551166

chore: fix docstrings for to_pandas_batches

f8432d7

Co-authored-by: Gil Forsyth <[email protected]>

cpcloud force-pushed the to-pandas-batches branch from 924b3d8 to f8432d7 Compare September 28, 2023 15:52

cpcloud merged commit ad88d8a into ibis-project:master Sep 28, 2023
84 checks passed

cpcloud deleted the to-pandas-batches branch September 28, 2023 16:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(api): add `to_pandas_batches` #7123

feat(api): add `to_pandas_batches` #7123

cpcloud commented Sep 11, 2023

cpcloud commented Sep 12, 2023 •

edited

Loading

jcrist Sep 28, 2023

cpcloud Sep 28, 2023

jcrist Sep 28, 2023

cpcloud Sep 28, 2023

jcrist Sep 28, 2023

cpcloud Sep 28, 2023

jcrist Sep 28, 2023

cpcloud Sep 28, 2023

cpcloud commented Sep 28, 2023

gforsyth left a comment

cpcloud commented Sep 28, 2023

		@@ -223,6 +224,8 @@ def _ipython_key_completions_(self) -> list[str]:


		class _FileIOHandler:
		pandas_converter = PandasData

feat(api): add to_pandas_batches #7123

feat(api): add to_pandas_batches #7123

Conversation

cpcloud commented Sep 11, 2023

cpcloud commented Sep 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Sep 28, 2023

gforsyth left a comment

Choose a reason for hiding this comment

cpcloud commented Sep 28, 2023

feat(api): add `to_pandas_batches` #7123

feat(api): add `to_pandas_batches` #7123

cpcloud commented Sep 12, 2023 •

edited

Loading