refactor(trino): port to sqlglot #7871

cpcloud · 2023-12-31T22:24:08Z

Port the trino backend to sqlglot. I need to create some follow-up issues.

github-actions · 2023-12-31T22:24:34Z

ACTION NEEDED

Ibis follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

cpcloud · 2024-01-03T18:05:21Z

Snowflake tests are passing:

cloud in 🌐 falcon in …/ibis on  tes-trino is 📦 v7.2.0 via 🐍 v3.10.13 via ❄️  impure (ibis-3.10.13-env)
❯ pytest -m snowflake -n 8 --dist loadgroup --snapshot-update -q
bringing up nodes...
......x..x............x....x.............x.xx..............x..x.........x......................x....xx........x...x.......x..xx..x.............x.........x.xx.x.xx..........x...........x....x [ 13%]
.xxx..x.x.......xx...x.xx..x......x.............x....xx..........xx.....xx..........xx..x...x.x.xxx..x.x.xx......x........xx....x...x...x.x......x.......x..x...............x...x..x.x.......x [ 26%]
.............................x......s.x..x.....x............x.........x.......x.....x.x.x........xxxx....x.x.....xx.x............xx................x........x....x.....xx...x..xx....xxxxx.xxx [ 39%]
xx.x..x.x......x.............................x.........xx.x....x...............................x.x........xx...x................x..................x...........xx..............xx......x...... [ 52%]
.....x.x....................x........x.....x.x.......xx.x..x.....................x.......................x..........................................x.....x...xx......x....x..x.x.....x........ [ 65%]
.xx....x.......x.....x..xx.x............................xx........x............x..x..x..........x.x..x.......x.....x.....x........x...s......xx.........................x.......x......x.x.... [ 78%]
...........xx...x..x..x.x....x.........x.xx.x................................................................................................................................................. [ 91%]
......................................s...x.xx.....x..............................................................................                                                             [100%]
1260 passed, 3 skipped, 198 xfailed in 131.07s (0:02:11)

ibis/backends/base/sqlglot/__init__.py

ibis/formats/pandas.py

kszucs · 2024-01-04T09:11:26Z

ibis/expr/rewrites.py

@@ -91,6 +92,48 @@ def rewrite_dropna(_):
    return ops.Filter(_.parent, tuple(preds))


+@replace(p.WindowFunction(p.First(x, y)))


Are these specific to trino?

No, other backends that lack a proper First/Last aggregate (but implement it for scalar aggs in Ibis) can reuse this.

Can you please move these rules to sqlglot/rewrites.py?

Later we can do a consolidation pass over all the different rewrites.py

ibis/backends/tests/test_struct.py

ibis/backends/tests/test_array.py

ibis/backends/duckdb/compiler.py

cpcloud · 2024-01-06T09:37:16Z

Snowflake tests look good:

cloud in 🌐 falcon in …/ibis on  tes-trino is 📦 v7.2.0 via 🐍 v3.10.13 via ❄️  impure (ibis-3.10.13-env)
❯ pytest -m snowflake -n 8 --dist loadgroup --snapshot-update -q
bringing up nodes...
.......x.x.........xxx...xx.x...xx.......x...xx.x..x........s......x.x..xx........x......x..x.x.......x...............x......x..x.....x..x.....x......xxx..x.x.x.........x.............x...... [ 13%]
...............x...x.....x.x.x...............x.....x....xx............xx.....x..x.x..xx.x...x..x........x........x.............x...x......x..............x....x......x.xxx...x.x..x.x......... [ 26%]
.xxx...xx......xx.xx..x.....x..x...x...x..x........x......x....x..x.x...x....x..........x..x......xx......x.....x..x....x..........x.......................x...........x.......x...........xx. [ 39%]
...xxxx......x...x............................x............................x...........................................x................x.............x..................x.....x.....x.x...... [ 52%]
.x.x.x...x..x.x....xx...xx.........x..x..x....xx.x.........x...x.........xx.......x............x................x.x............x.x...........x..............x...x.....x..x......xx...........x [ 65%]
.......x...x.x..xx........x.x..x.xxx..x.xxxxxxxxx......x..x....x.x.....x.x...x....x.x....................x..x........x....x..........x......x.........................................x....... [ 78%]
.........................................x......................................................x.s...............................xx.......................................................... [ 91%]
.....................................................s.............................................................................                                                            [100%]
1260 passed, 3 skipped, 198 xfailed in 125.41s (0:02:05)

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

…is-project#7904) This PR pulls out some changes from ibis-project#7871, including some removal of redundant translation rules and fixing some xpassing tests

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

…is-project#7904) This PR pulls out some changes from ibis-project#7871, including some removal of redundant translation rules and fixing some xpassing tests

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

…is-project#7904) This PR pulls out some changes from ibis-project#7871, including some removal of redundant translation rules and fixing some xpassing tests

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

…is-project#7904) This PR pulls out some changes from ibis-project#7871, including some removal of redundant translation rules and fixing some xpassing tests

cpcloud changed the base branch from master to the-epic-split December 31, 2023 22:24

cpcloud changed the title ~~tes trino~~ refactor(trino): port to sqlglot Dec 31, 2023

cpcloud marked this pull request as ready for review December 31, 2023 22:34

cpcloud marked this pull request as draft December 31, 2023 22:35

cpcloud force-pushed the tes-trino branch 2 times, most recently from 595ad37 to c625487 Compare January 1, 2024 16:45

cpcloud added refactor Issues or PRs related to refactoring the codebase trino The Trino backend labels Jan 1, 2024

cpcloud force-pushed the tes-trino branch from c625487 to 2085c68 Compare January 3, 2024 11:53

cpcloud marked this pull request as ready for review January 3, 2024 11:56

cpcloud force-pushed the tes-trino branch 5 times, most recently from a90625f to 2636060 Compare January 3, 2024 17:24

cpcloud force-pushed the tes-trino branch 2 times, most recently from 0f76d65 to 04eaf7f Compare January 3, 2024 18:27

kszucs reviewed Jan 3, 2024

View reviewed changes

ibis/backends/base/sqlglot/__init__.py Outdated Show resolved Hide resolved

kszucs reviewed Jan 4, 2024

View reviewed changes

ibis/formats/pandas.py Outdated Show resolved Hide resolved

kszucs reviewed Jan 4, 2024

View reviewed changes

ibis/backends/tests/test_struct.py Show resolved Hide resolved

kszucs reviewed Jan 4, 2024

View reviewed changes

ibis/backends/tests/test_array.py Outdated Show resolved Hide resolved

kszucs reviewed Jan 4, 2024

View reviewed changes

ibis/backends/duckdb/compiler.py Outdated Show resolved Hide resolved

cpcloud force-pushed the the-epic-split branch from eb31002 to 61e34b9 Compare January 4, 2024 12:18

cpcloud force-pushed the tes-trino branch 3 times, most recently from d22fb51 to d2a7e5a Compare January 4, 2024 13:16

cpcloud mentioned this pull request Jan 4, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups #7904

Merged

cpcloud force-pushed the tes-trino branch from 4ccbd98 to ed76624 Compare January 6, 2024 09:31

cpcloud added 3 commits January 6, 2024 04:31

feat(trino): port to sqlglot

8fdeb27

fix(polars): force null sorting to match the rest of ibis

cc5d283

test(pandas): ignore array size warning

6ed20fa

cpcloud force-pushed the tes-trino branch from ed76624 to 6ed20fa Compare January 6, 2024 09:32

kszucs approved these changes Jan 6, 2024

View reviewed changes

kszucs enabled auto-merge (rebase) January 6, 2024 09:35

kszucs merged commit 605a873 into ibis-project:the-epic-split Jan 6, 2024
32 checks passed

cpcloud deleted the tes-trino branch January 6, 2024 09:43

cpcloud added a commit that referenced this pull request Jan 12, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

b229673

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

cpcloud added a commit that referenced this pull request Jan 13, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

bdfb11e

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

cpcloud added a commit that referenced this pull request Jan 17, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

91fbbbd

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

kszucs pushed a commit that referenced this pull request Feb 5, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

53a0248

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

kszucs pushed a commit that referenced this pull request Feb 6, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

5a3b00f

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

kszucs pushed a commit that referenced this pull request Feb 6, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

4df79e6

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

cpcloud added a commit that referenced this pull request Feb 12, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

72e854d

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

cpcloud added a commit that referenced this pull request Feb 12, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

16bc7f9

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

kszucs pushed a commit that referenced this pull request Feb 12, 2024

refactor(sqlglot): various sqlglot compiler and backend clean ups (#7904

db45e41

) This PR pulls out some changes from #7871, including some removal of redundant translation rules and fixing some xpassing tests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(trino): port to sqlglot #7871

refactor(trino): port to sqlglot #7871

cpcloud commented Dec 31, 2023 •

edited

Loading

github-actions bot commented Dec 31, 2023

cpcloud commented Jan 3, 2024 •

edited

Loading

kszucs Jan 4, 2024

cpcloud Jan 4, 2024 •

edited

Loading

kszucs Jan 6, 2024

cpcloud commented Jan 6, 2024

		@@ -91,6 +92,48 @@ def rewrite_dropna(_):
		return ops.Filter(_.parent, tuple(preds))


		@replace(p.WindowFunction(p.First(x, y)))

refactor(trino): port to sqlglot #7871

refactor(trino): port to sqlglot #7871

Conversation

cpcloud commented Dec 31, 2023 • edited Loading

github-actions bot commented Dec 31, 2023

cpcloud commented Jan 3, 2024 • edited Loading

kszucs Jan 4, 2024

Choose a reason for hiding this comment

cpcloud Jan 4, 2024 • edited Loading

Choose a reason for hiding this comment

kszucs Jan 6, 2024

Choose a reason for hiding this comment

cpcloud commented Jan 6, 2024

cpcloud commented Dec 31, 2023 •

edited

Loading

cpcloud commented Jan 3, 2024 •

edited

Loading

cpcloud Jan 4, 2024 •

edited

Loading