Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add Python + SQL section to why ibis #8526

Merged
merged 2 commits into from
Mar 7, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions docs/why.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,74 @@ and robust framework for data manipulation in Python.
In the long-term, we aim for a standard query plan Intermediate Representation
(IR) like [Substrait](https://substrait.io) to simplify this further.

## Python + SQL: better together

For most backends, Ibis works by compiling Python expressions into SQL:

```{python}
g = t.group_by(["species", "island"]).agg(count=t.count()).order_by("count")
ibis.to_sql(g)
```

You can mix and match Python and SQL code:

```{python}
sql = """
SELECT
species,
island,
COUNT(*) AS count
FROM penguins
GROUP BY species, island
""".strip()
```

::: {.panel-tabset}

## DuckDB

```{python}
con = ibis.duckdb.connect()
t = con.read_parquet("penguins.parquet")
g = t.alias("penguins").sql(sql)
g
```

```{python}
g.order_by("count")
```

## DataFusion

```{python}
con = ibis.datafusion.connect()
t = con.read_parquet("penguins.parquet")
g = t.alias("penguins").sql(sql)
g
```

```{python}
g.order_by("count")
```

## PySpark

```{python}
con = ibis.connect("pyspark://")
t = con.read_parquet("penguins.parquet")
g = t.alias("penguins").sql(sql)
g
```

```{python}
g.order_by("count")
```

:::

This allows you to combine the flexibility of Python with the scale and
performance of modern SQL.

## Scaling up and out

Out of the box, Ibis offers a great local experience for working with many file
Expand Down
Loading