docs: blog for the 1 billion row challenge #8004

lostmygithubaccount · 2024-01-17T02:26:18Z

Description of changes

work in progress, just throwing up the code

Issues closed

cpcloud · 2024-01-17T14:17:20Z

docs/posts/1brc/index.qmd

+    .agg(
+        min_temp=ibis._.temperature.min(),
+        mean_temp=ibis._.temperature.mean(),
+        max_temp=ibis._.temperature.max(),


good place to show off selectors:

.agg(s.across(_.temp, {"min": _.min(), "mean": _.mean(), "max": _.max()}))

not sure I've written this correctly but getting an error:

TypeError Traceback (most recent call last) [/Users/cody/repos/ibis/docs/posts/1brc/index.qmd](https://file+.vscode-resource.vscode-cdn.net/Users/cody/repos/ibis/docs/posts/1brc/index.qmd) in line 6 [288](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=287) t = ibis.read_csv("1brc/data/measurements.txt", **kwargs) [289](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=288) res = ( [290](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=289) t [291](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=290) .group_by(ibis._.station) [292](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=291) .agg( ----> [293](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=292) s.across( [294](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=293) ibis._.temperature, [295](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=294) { [296](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=295) "min": ibis._.min(), [297](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=296) "mean": ibis._.mean(), [298](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=297) "max": ibis._.max(), [299](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=298) }, [300](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=299) ) [301](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=300) ) [302](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=301) .order_by(ibis._.station.desc()) [303](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=302) ) [305](file:///Users/cody/repos/ibis/docs/posts/1brc/index.qmd?line=304) res File [~/repos/ibis/ibis/selectors.py:498](https://file+.vscode-resource.vscode-cdn.net/Users/cody/repos/ibis/~/repos/ibis/ibis/selectors.py:498), in across(selector, func, names) [496](file:///Users/cody/repos/ibis/ibis/selectors.py?line=495) funcs = dict(func if isinstance(func, Mapping) else {None: func}) [497](file:///Users/cody/repos/ibis/ibis/selectors.py?line=496) if not isinstance(selector, Selector): --> [498](file:///Users/cody/repos/ibis/ibis/selectors.py?line=497) selector = c(*util.promote_list(selector)) ... --> [396](file:///Users/cody/repos/ibis/ibis/selectors.py?line=395) names = frozenset(col if isinstance(col, str) else col.get_name() for col in names) [398](file:///Users/cody/repos/ibis/ibis/selectors.py?line=397) def func(col: ir.Value) -> bool: [399](file:///Users/cody/repos/ibis/ibis/selectors.py?line=398) schema = col.op().table.schema TypeError: unhashable type: 'Deferred'

cpcloud · 2024-01-17T14:18:25Z

docs/posts/1brc/index.qmd

+        header=False,
+        columns={"station": "VARCHAR", "temperature": "DOUBLE"},
+    )
+elif ibis.get_backend().name == "polars":


Can you instead collect the kwargs into a dict and then have a single call to ibis.read_csv? It's pretty noisy with the repetition.

cpcloud · 2024-01-17T14:18:48Z

docs/posts/1brc/index.qmd

+```
+
+```{python}
+t


You're already showing t in the previous block of code.

ncclementi · 2024-01-17T14:28:48Z

Dropping couple of links I found in the python submissions, that might help as reference on what people did:

DuckDB solution: https://github.com/ifnesi/1brc/blob/main/calculateAverageDuckDB.py
Polars: seems like scan_csv performs better: 1BRC in Python gunnarmorling/1brc#62 (reply in thread), script is here https://github.com/ifnesi/1brc/blob/main/calculateAveragePolars.py

Here are the dask and spark solutions, idk if we want to get into try to run those with Ibis but just in case, gunnarmorling/1brc#450 (comment)

cpcloud · 2024-01-17T14:36:02Z

ibis.read_csv ultimately calls pl.scan_csv if possible: https://github.com/ibis-project/ibis/blob/main/ibis/backends/polars/__init__.py#L169

cpcloud · 2024-01-17T14:38:46Z

docs/posts/1brc/index.qmd

+        separator=";",
+        has_header=False,
+        new_columns=["station", "temperature"],
+        schema={"station": pl.datatypes.Utf8, "temperature": pl.datatypes.Float64},


Suggested change

schema={"station": pl.datatypes.Utf8, "temperature": pl.datatypes.Float64},

schema={"station": pl.Utf8, "temperature": pl.Float64},

ncclementi · 2024-01-17T14:39:23Z

Interesting, I haven't used polars much, I was just reading the thread on what people tried, looks like Ritchie chimed in that thread explaining why read_csv in their case is slower than scan_csv:

If you read csv, you force polars to materialize the whole dataset at once. It is expected to be slower as you don't allow polars to determine how to process the query.

gunnarmorling/1brc#62 (reply in thread)

lostmygithubaccount · 2024-01-17T14:53:18Z

this code is so simple I think we should try to show it across as many backends as possible -- DuckDB, Polars, DataFusion, SQLite (?), Postgres, Clickhouse should all work pretty easily? then we can have a fancy title like "Using one Python dataframe API to take the billion row challenge with DuckDB, Polars, DataFusion, Clickhouse, SQLite, Postgres"

cpcloud · 2024-01-17T14:57:20Z

I would show it across the columnar local backends, only to avoid having to write a bunch of ingestion code for backends that don't support read_csv (SQLite for example).

lostmygithubaccount · 2024-01-17T15:12:56Z

so far:

27s for DuckDB
290s for Polars, not sure what's going on there
76s for DataFusion

ncclementi · 2024-01-17T15:34:30Z

@lostmygithubaccount what are the specs of the machine you are running this on? The results for an M1 with 32GB RAM for polars reported here way different https://github.com/ifnesi/1brc/tree/main#performance-on-a-macbook-pro-m1-32gb

lostmygithubaccount · 2024-01-17T16:23:29Z

also a macbook pro M1 32GB -- just pushed duckdb/polars/datafusion, need to be away for a bit, will try to finish this up later. unclear how to pass in the right kwargs for Clickhouse right now

lostmygithubaccount · 2024-01-17T16:32:15Z

docs/posts/1brc/index.qmd

+}
+
+# kwargs = duckdb_kwargs if ibis.get_backend().name == "duckdb" else polars_kwargs
+match ibis.get_backend().name:


I think this is my first use of a match statement

ibis-docs-bot · 2024-01-18T02:25:36Z

Docs preview: https://pr-8004-468d140881a7eb53c3af71aa9430f96cb0e3b85e--ibis-quarto.netlify.app

ibis-docs-bot · 2024-01-18T16:25:08Z

Docs preview: https://pr-8004-a5eedffa028fa79d9b4a49aa85e96fc72d851796--ibis-quarto.netlify.app

docs/posts/1brc/index.qmd

ibis-docs-bot · 2024-01-18T17:41:07Z

Docs preview: https://pr-8004-3451950b6598bb4778c3ae90c72298eea4725890--ibis-quarto.netlify.app

ibis-docs-bot · 2024-01-18T18:50:44Z

Docs preview: https://pr-8004-e2493a97dfadfb223716309786eecb8fc7f3bb63--ibis-quarto.netlify.app

deepyaman · 2024-01-18T21:37:25Z

docs/posts/1brc/index.qmd

+
+::: {.panel-tabset}
+
+## DuckDb


Suggested change

## DuckDb

## DuckDB

arggghhhh thanks...rerendering for another 20 minutes 😂

docs/posts/1brc/index.qmd

ncclementi · 2024-01-19T15:56:08Z

I think the post looks great @lostmygithubaccount .

The last small comment I have is, whether the conclusion section should be before the Bonus content. I think the Bonus is neat, but feels disconnected from the rest, and people might skip it and not read the conclusion. But this is just an opinion, feel free to ignore.

ibis-docs-bot · 2024-01-19T17:41:52Z

Docs preview: https://pr-8004-9a98a5071bc1a7539fa328544537c726ddb3e4b7--ibis-quarto.netlify.app

lostmygithubaccount · 2024-01-22T14:10:49Z

I think this is good to merge today

ncclementi

LGTM

lostmygithubaccount added 2 commits January 16, 2024 21:25

docs: blog for the 1 billion row challenge

b5d7350

add schema to polars, didn't help

fa812b2

cpcloud reviewed Jan 17, 2024

View reviewed changes

save for now

39eb6a6

lostmygithubaccount commented Jan 17, 2024

View reviewed changes

update blog

468d140

lostmygithubaccount added docs Documentation related issues or PRs docs-preview Add this label to trigger a docs preview labels Jan 18, 2024

ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Jan 18, 2024

fix whitespace

b1feec7

lostmygithubaccount mentioned this pull request Jan 18, 2024

bug: incorrect SQL generated on main (but not TES) for generating random numbers #8013

Closed

1 task

blog updates

a5eedff

lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Jan 18, 2024

ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Jan 18, 2024

ncclementi reviewed Jan 18, 2024

View reviewed changes

docs/posts/1brc/index.qmd Outdated Show resolved Hide resolved

add newline

b3aaaae

lostmygithubaccount marked this pull request as ready for review January 18, 2024 16:52

add repo link

3451950

lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Jan 18, 2024

ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Jan 18, 2024

hide output for some reason

e2493a9

lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Jan 18, 2024

ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Jan 18, 2024

deepyaman reviewed Jan 18, 2024

View reviewed changes

docs/posts/1brc/index.qmd Show resolved Hide resolved

lostmygithubaccount added 2 commits January 18, 2024 16:48

freeze

a1c7d75

add portability tag

ec634bf

swap conclusion and bonus sections

64fbf15

lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Jan 19, 2024

ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Jan 19, 2024

forgot the freeze

9a98a50

lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Jan 19, 2024

ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Jan 19, 2024

lostmygithubaccount mentioned this pull request Jan 20, 2024

bug: investigate discrepancies between Polars Python native code vs Ibis w/ the Polars backend #8050

Open

1 task

lostmygithubaccount requested a review from cpcloud January 22, 2024 14:10

ncclementi approved these changes Jan 22, 2024

View reviewed changes

cpcloud approved these changes Jan 22, 2024

View reviewed changes

cpcloud merged commit 141edea into ibis-project:main Jan 22, 2024
19 checks passed

cpcloud added this to the 8.0 milestone Jan 22, 2024

kszucs mentioned this pull request Feb 1, 2024

chore: check builds for the-epic-split after rebase #8178

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: blog for the 1 billion row challenge #8004

docs: blog for the 1 billion row challenge #8004

lostmygithubaccount commented Jan 17, 2024

cpcloud Jan 17, 2024

lostmygithubaccount Jan 18, 2024

cpcloud Jan 17, 2024

cpcloud Jan 17, 2024

ncclementi commented Jan 17, 2024

cpcloud commented Jan 17, 2024 •

edited

Loading

cpcloud Jan 17, 2024

ncclementi commented Jan 17, 2024

lostmygithubaccount commented Jan 17, 2024

cpcloud commented Jan 17, 2024

lostmygithubaccount commented Jan 17, 2024

ncclementi commented Jan 17, 2024

lostmygithubaccount commented Jan 17, 2024

lostmygithubaccount Jan 17, 2024

ibis-docs-bot bot commented Jan 18, 2024

ibis-docs-bot bot commented Jan 18, 2024

ibis-docs-bot bot commented Jan 18, 2024

ibis-docs-bot bot commented Jan 18, 2024

deepyaman Jan 18, 2024

lostmygithubaccount Jan 18, 2024

ncclementi commented Jan 19, 2024

ibis-docs-bot bot commented Jan 19, 2024

lostmygithubaccount commented Jan 22, 2024

ncclementi left a comment

	schema={"station": pl.datatypes.Utf8, "temperature": pl.datatypes.Float64},
	schema={"station": pl.Utf8, "temperature": pl.Float64},

docs: blog for the 1 billion row challenge #8004

docs: blog for the 1 billion row challenge #8004

Conversation

lostmygithubaccount commented Jan 17, 2024

Description of changes

Issues closed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncclementi commented Jan 17, 2024

cpcloud commented Jan 17, 2024 • edited Loading

Choose a reason for hiding this comment

ncclementi commented Jan 17, 2024

lostmygithubaccount commented Jan 17, 2024

cpcloud commented Jan 17, 2024

lostmygithubaccount commented Jan 17, 2024

ncclementi commented Jan 17, 2024

lostmygithubaccount commented Jan 17, 2024

Choose a reason for hiding this comment

ibis-docs-bot bot commented Jan 18, 2024

ibis-docs-bot bot commented Jan 18, 2024

ibis-docs-bot bot commented Jan 18, 2024

ibis-docs-bot bot commented Jan 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncclementi commented Jan 19, 2024

ibis-docs-bot bot commented Jan 19, 2024

lostmygithubaccount commented Jan 22, 2024

ncclementi left a comment

Choose a reason for hiding this comment

cpcloud commented Jan 17, 2024 •

edited

Loading