Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ir): support converting limit(1) inputs to scalar subqueries #8223

Closed
wants to merge 161 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
161 commits
Select commit Hold shift + click to select a range
69cba23
test(ir): reorganize ibis/tests/expr to enable running the tests with…
kszucs Dec 13, 2023
4986c91
test(sql): move sql tests requiring a functional backend from `ibis/t…
cpcloud Dec 5, 2023
4eba860
test(backends): move backends dependent benchmarks to ibis/backends/t…
kszucs Dec 13, 2023
a38aae8
test(ir): ensure that no backends are required to run the core tests
kszucs Dec 13, 2023
2927d8f
chore(ci): skip running backend tests on the-epic-split branch
kszucs Dec 13, 2023
6b225ee
chore(ci): change the core testing command since the core marker is c…
kszucs Dec 13, 2023
b90f2c4
chore(ci): temporarily disable test_doctests job
kszucs Dec 13, 2023
26c1321
chore(ci): add todo note about restoring the previous ci-check command
kszucs Dec 14, 2023
631349a
test(ir): ensure that no backends are required to run the core tests
kszucs Dec 13, 2023
c805eb9
refactor(ir): split the relational operations
kszucs Nov 8, 2023
0ecc96f
refactor(ir): wrap `JoinChain.first` in `ops.SelfReference` similar t…
kszucs Dec 21, 2023
a162718
test(ir): cover constructing reductions in the core test suite
kszucs Dec 21, 2023
c988ab9
refactor(ir): add `JoinTable` operation unique to `JoinChain` instead…
kszucs Dec 22, 2023
881a938
fix(decompile): ensure that `SelfReference` is decompiled with a call…
cpcloud Dec 23, 2023
5980895
refactor(ir): support join of joins while avoiding nesting
kszucs Dec 23, 2023
405f1ba
feat(sql): lower expressions to SQL-like relational operations
kszucs Dec 24, 2023
413543d
refactor(duckdb): initial cut of sqlglot DuckDB compiler
gforsyth Jul 10, 2023
da7745e
refactor(duckdb/clickhouse): implement sqlglot backends and re-enable ci
cpcloud Dec 22, 2023
31252a9
feat(datafusion): port to new sqlglot backend
cpcloud Dec 24, 2023
8bd248a
refactor(compilers): conslidate StringJoin impl
cpcloud Dec 26, 2023
bce36bd
feat(common): add `Dispatched` base class for convenient visitor patt…
kszucs Dec 28, 2023
ac25d9e
refactor(duckdb): remove the need for a specialized `_to_geodataframe…
cpcloud Dec 29, 2023
a3cca0b
fix(duckdb): ensure that create_schema and create_database are actual…
cpcloud Dec 29, 2023
dffbda1
refactor(ir): stricter scalar subquery integrity checks
kszucs Dec 29, 2023
5f89cc0
feat(common): add a memory efficient `Node.map()` implementation
kszucs Dec 29, 2023
1726e6d
fix(common): intermediate result removal fails if there are duplicate…
kszucs Dec 29, 2023
d547d33
refactor(api): revamp `asof` join predicates
kszucs Dec 29, 2023
1d1c541
fix(ir): self reference fields were incorrectly dereferenced to the p…
kszucs Jan 2, 2024
78d38c7
fix(rewrites): add missing filter arguments for `node.replace()` calls
kszucs Jan 2, 2024
2f7640f
refactor(snowflake): use sqlglot for the snowflake backend
cpcloud Dec 26, 2023
24abf1e
refactor(common): support union types as well as forward references i…
kszucs Jan 2, 2024
b89273e
ci(snowflake): enable for `the-epic-split` branch
cpcloud Jan 3, 2024
5ee9c26
fir(ir): `asof` join `tolerance` parameter should post-filter and pos…
kszucs Jan 2, 2024
218061e
feat(duckdb): support `asof` joins including `tolerance` parameter
kszucs Jan 3, 2024
e266585
ci: remove `merge_group` (#7899)
kszucs Jan 4, 2024
52c5051
refactor(pandas): port the pandas backend with an improved execution …
kszucs Jan 4, 2024
a466684
chore(deps): relock
cpcloud Jan 4, 2024
3fd91f0
ci: comment out sys-deps step
cpcloud Jan 4, 2024
c1f4a9a
chore: remove duplicate `distinct` decompile rule
cpcloud Jan 4, 2024
ace6e20
ci: install decompiler as extra not as its own dependency (#7901)
cpcloud Jan 4, 2024
3474dc9
fix(conversion): convert decimals to the exact precision and scale re…
cpcloud Jan 4, 2024
0a629e6
test(snowflake): fix expected decimal results
cpcloud Jan 4, 2024
544925e
test(duckdb): relax exact type check in decimal literal assertion
cpcloud Jan 4, 2024
4a3952c
refactor(sqlglot): various sqlglot compiler and backend clean ups (#7…
cpcloud Jan 4, 2024
45a42b9
refactor(polars): update the polars backend to use the new relational…
kszucs Jan 4, 2024
aefc3f7
feat(trino): port to sqlglot
cpcloud Dec 29, 2023
c293b23
fix(polars): force null sorting to match the rest of ibis
cpcloud Jan 5, 2024
c8563b1
test(pandas): ignore array size warning
cpcloud Jan 5, 2024
2876676
refactor(postgres): port to sqlglot (#7877)
cpcloud Jan 6, 2024
abdebfd
refactor(mysql): port to sqlglot (#7926)
cpcloud Jan 6, 2024
15ba46c
refactor(sqlglot): remove duplicated simple compilation rules and sort
cpcloud Jan 9, 2024
22493ce
chore(deps): bump sqlglot and regen sql
cpcloud Jan 12, 2024
3f5a04b
fix(duckdb): add `flip_coordinates` translation to sqlglot duckdb bac…
cpcloud Jan 12, 2024
fb0aa60
fix(snowflake): use `_safe_raw_sql` for `insert` implementation
cpcloud Jan 12, 2024
c282137
fix(mysql): remove not-allowed frame clause from rank window function
cpcloud Jan 12, 2024
3c9dbd2
test(postgres): use DBAPI instead of sqlalchemy apis in timezone test
cpcloud Jan 12, 2024
9228754
test(postgres): remove test that no longer works
cpcloud Jan 12, 2024
f28f1a5
test(pandas): use the correct error type when xfailing for compound-s…
cpcloud Jan 12, 2024
1f92b2b
refactor(ir): give unbound tables namespaces
cpcloud Jan 12, 2024
7a69243
chore(duckdb/mysql): remove dead code and comment
cpcloud Jan 12, 2024
b4fecd5
refactor(sqlglot): remove duplicate `StringAscii` definitions
cpcloud Jan 12, 2024
a5e86dc
chore(sqlglot): deduplicate pad functions
cpcloud Jan 12, 2024
69885db
refactor(sqlglot): make anonymous functions easier to use and remove …
cpcloud Jan 12, 2024
bedfce6
test(backends): make null results try_cast test agnostic to nan vs None
cpcloud Jan 12, 2024
2b7c415
refactor(sqlglot): use a more backend-agnostic expression for non-fin…
cpcloud Jan 13, 2024
dbce293
refactor(sqlglot): clean up `explode` usage
cpcloud Jan 13, 2024
55dcf57
refactor(pyspark): reimplement the backend using the new relational o…
kszucs Jan 8, 2024
03162f2
feat(pyspark): add support for PySpark 3.5
kszucs Jan 15, 2024
eb95cf3
refactor(pyspark): remove sqlalchemy dependency from pyspark
cpcloud Jan 16, 2024
89e7ef6
chore(deps): bump pyspark to 3.5 in poetry lock file
cpcloud Jan 16, 2024
d231cf2
fix(ir): only dereference comparisons not generic binary operations
cpcloud Jan 15, 2024
6c55148
chore: rename to `dereference_comparison`
cpcloud Jan 16, 2024
da8b3ef
chore(deps): relock
cpcloud Jan 17, 2024
e09e733
test(generic): clean up try_cast to null test
cpcloud Jan 17, 2024
5c96a9f
fix(polars): user newer `drop` API to avoid deprecation warning
cpcloud Jan 17, 2024
4edd78b
fix(polars): user newer `drop` API in asof join implementation
cpcloud Jan 17, 2024
e0ab99b
refactor(druid): port to sqlglot
cpcloud Jan 16, 2024
d0378bb
refactor(impala): port to sqlglot
cpcloud Jan 14, 2024
71bb3db
refactor(pandas): simplify pandas helpers
kszucs Jan 18, 2024
a78e9ee
chore(impala): remove unused imports
cpcloud Jan 18, 2024
56794e6
refactor(bigquery): port to sqlglot
cpcloud Jan 9, 2024
ac5b75b
chore: add docstring for null ordering transform
cpcloud Jan 18, 2024
6ea9c6a
chore(bigquery-datatypes): fix type annotations and raise uniform err…
cpcloud Jan 18, 2024
090a723
ci(bigquery): install geospatial extra
cpcloud Jan 19, 2024
4509839
test: remove unused `spread_type` function
cpcloud Jan 19, 2024
e370b26
test: account for new error type
cpcloud Jan 19, 2024
43eea9d
test(bigquery): skip geospatial execution test when geopandas not ins…
cpcloud Jan 19, 2024
1f649f6
fix(snowflake): handle udf function naming
cpcloud Jan 19, 2024
336092f
refactor(exasol): port to sqlglot (#8032)
cpcloud Jan 19, 2024
8ca3e00
ci(exasol): run ci serially (#8042)
cpcloud Jan 19, 2024
e31428f
feat(sql): extract common table expressions
kszucs Jan 19, 2024
a76686e
chore(sql): regenerate snapshots for clickhouse, duckdb and postgres
kszucs Jan 19, 2024
96002f4
chore(sql): regenerate snapshots for snowflake
kszucs Jan 19, 2024
5179b81
chore(sql): regenerate snapshots for bigquery
kszucs Jan 19, 2024
b8d06ce
chore(sql): regenerate snapshots for pyspark
kszucs Jan 19, 2024
6827be0
chore(sql): regenerate snapshots for trino
kszucs Jan 19, 2024
35c9390
chore(impala): regen snapshots
cpcloud Jan 20, 2024
127d51c
fix(trino): compile property literal values directly instead of going…
cpcloud Jan 20, 2024
44b7da2
ci(impala): run tests in series
cpcloud Jan 20, 2024
5731e2a
chore(exasol): avoid complex websocket callback for inserting memtables
cpcloud Jan 20, 2024
2022201
test(exasol): account for unordered results in window function tests
cpcloud Jan 20, 2024
af54a18
test(duckdb): test that column name case is preserved when inserting
cpcloud Jan 20, 2024
b2ec733
refactor(oracle): port to sqlglot (#8020)
gforsyth Jan 22, 2024
81ed41a
chore(deps): remove sqlalchemy dependencies from oracle extra
cpcloud Jan 23, 2024
59ddaef
refactor(mssql): port to sqlglot
cpcloud Jan 6, 2024
abe2d93
fix(sql): don't generate table aliases for `ops.JoinLink`
kszucs Jan 22, 2024
e4e2dc8
chore(impala): regen snapshots
cpcloud Jan 22, 2024
1f4d0c8
test(markers): add tests for custom markers
gforsyth Jan 23, 2024
2b4f0d3
fix(duckdb): allow passing both overwrite and temp to create_table
gforsyth Jan 24, 2024
1b46a21
refactor(polars): allow passing temp=False to polars create_table
gforsyth Jan 24, 2024
f9f8a68
refactor(exasol): add temp kwarg to create_table for api consistency
gforsyth Jan 24, 2024
6557210
test(backends): add test for overwrite and temp intersection in creat…
gforsyth Jan 24, 2024
fa991e4
fix(oracle): enable dropping temporary tables
gforsyth Jan 24, 2024
515ad2b
fix(oracle): clean up memtables at exit
gforsyth Jan 24, 2024
e1412ea
fix(oracle): allow passing both overwrite and temp to create_table
gforsyth Jan 24, 2024
4f770f5
refactor(oracle): simplify oracle timestamp overrides
gforsyth Jan 23, 2024
1f6595e
fix(api): forbid using `asc`/`desc` in selections
jcrist Jan 25, 2024
bcb6905
fix(api): support passing literal booleans to `filter`
jcrist Jan 26, 2024
d6e1611
test(api): add union aliasing test
cpcloud Jan 27, 2024
f10ecbd
fix(polars): reference the correct field in the `ops.SelfReference` rule
cpcloud Jan 27, 2024
b1fcdb6
test(polars): enable xpassing test
cpcloud Jan 27, 2024
f77b9b9
test(duckdb): move tests to specific backend test suites
cpcloud Jan 29, 2024
c9ab21d
test(duckdb): run test in subprocess to avoid setting the default bac…
cpcloud Jan 29, 2024
0a088e5
refactor(sqlite): port to SQLGlot (#8154)
jcrist Jan 31, 2024
06ee3a1
refactor(sql): remove temporary table creation when using inline sql …
cpcloud Jan 31, 2024
b5fecc6
refactor(sql): reorganize sqlglot rewrites
kszucs Jan 28, 2024
3c8eb56
chore(deps): relock
kszucs Feb 1, 2024
52c0d84
chore(oracle): fix ops.ArrayColum reference to be ops.Array
kszucs Feb 1, 2024
2cefd16
chore(polars): enable xpassing .sql test
kszucs Feb 1, 2024
61e39a3
chore(sqlite): fix expected errors for array test cases
kszucs Feb 1, 2024
2942069
chore(duckdb): generate snapshots for geospatial tests
kszucs Feb 1, 2024
ab8e3f2
chore(pandas): xfail test_unnest_range()
kszucs Feb 1, 2024
d7feef1
chore(impala): rename ops.ArrayColumn to ops.Array
kszucs Feb 1, 2024
5c233ec
chore(mssql): rename ops.ArrayColumn to ops.Array
kszucs Feb 1, 2024
e6e4081
chore(pyspark): enable xpassing tests
kszucs Feb 1, 2024
4451109
chore(mssql): disable test cases for unsupported ops.Array, ops.HexDi…
kszucs Feb 1, 2024
eb345ef
chore(exasol): enable xpassing tests
kszucs Feb 1, 2024
f093047
style(tests): reformat the pytest marker blocks
kszucs Feb 1, 2024
c3e3da0
chore(impala): add early failure in the type system for complex datat…
cpcloud Feb 1, 2024
f7a904e
chore(impala): disable array position
cpcloud Feb 1, 2024
bdc58db
test: fix array, string and uuid tests
cpcloud Feb 1, 2024
4ca96b2
test: fix more array tests
cpcloud Feb 1, 2024
7313cc0
fix(common): don't match an `Object` pattern with more positional arg…
kszucs Jan 28, 2024
954228f
refactor(risingwave): port to sqlglot (#8171)
gforsyth Feb 1, 2024
5915b05
refactor(dask): port the dask backend to the new execution model (#8005)
kszucs Feb 1, 2024
acd2a47
fix(polars): remove deprecated `pl.count()` and `with_time_unit`
kszucs Feb 2, 2024
c68cc96
test(snowflake): enable xpassing `test_dot_sql::test_order_by_no_proj…
kszucs Feb 2, 2024
28ce539
style: remove extra newline from docstrings
kszucs Feb 2, 2024
1d9e263
test(pandas): skip tests for older pandas
kszucs Feb 2, 2024
45a76f9
test(dask): skip tests for older pandas
kszucs Feb 2, 2024
8504d87
fix(mssql): don't use the removed `sge.TRUE` and `sge.FALSE` literals
kszucs Feb 2, 2024
0aa9199
fix(pyspark): don't use the removed `sge.NULL`, `sge.TRUE` and `sge.F…
kszucs Feb 2, 2024
9982e8b
fix(sqlite): don't use the removed `sge.NULL` literal
kszucs Feb 2, 2024
6b73d2e
test(ir): remove outdated old-style pytest hookwrapper causing warnings
kszucs Feb 2, 2024
7915237
chore(deps): remove `sqlalchemy-views` and `sqlalchemy-risingwave`
kszucs Feb 2, 2024
db39886
test(backends): restore deleted `test_benchmarks.py`
kszucs Feb 2, 2024
e4df99b
test(duckdb): account for other errors when running in the nix sandbox
cpcloud Feb 2, 2024
d5e256f
feat(api): support the inner join convenience to not repeat fields kn…
kszucs Feb 2, 2024
77052c7
fix(polars): columns are picked from the correct side in case of conf…
kszucs Feb 2, 2024
e68000c
chore: quote everything (#8172)
cpcloud Feb 2, 2024
5c421ca
fix(ir): support converting limit(1) inputs to scalar subqueries
kszucs Feb 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 2 additions & 5 deletions .github/renovate.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
"addLabels": ["bigquery"]
},
{
"matchPackagePatterns": ["duckdb", "duckdb-engine"],
"matchPackagePatterns": ["duckdb"],
"addLabels": ["duckdb"]
},
{
Expand Down Expand Up @@ -82,10 +82,7 @@
"addLabels": ["pyspark"]
},
{
"matchPackagePatterns": [
"snowflake-connector-python",
"snowflake-sqlalchemy"
],
"matchPackagePatterns": ["snowflake-connector-python"],
"addLabels": ["snowflake"]
},
{
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/check-generated-files.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ on:
- poetry.lock
- pyproject.toml
- requirements-dev.txt
merge_group:

concurrency:
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
Expand Down
21 changes: 18 additions & 3 deletions .github/workflows/ibis-backends-cloud.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ on:
- ".envrc"
branches:
- main
- the-epic-split

permissions:
# this allows extractions/setup-just to list releases for `just` at a higher
Expand Down Expand Up @@ -42,10 +43,24 @@ jobs:
- "3.9"
- "3.11"
backend:
- name: bigquery
title: BigQuery
- name: snowflake
title: Snowflake
extras:
- snowflake
include:
- python-version: "3.9"
backend:
name: bigquery
title: BigQuery
extras:
- bigquery
- python-version: "3.11"
backend:
name: bigquery
title: BigQuery
extras:
- bigquery
- geospatial
steps:
- name: checkout
uses: actions/checkout@v4
Expand All @@ -61,7 +76,7 @@ jobs:
cache: poetry

- name: install ibis
run: poetry install --without dev --without docs --extras ${{ matrix.backend.name }}
run: poetry install --without dev --without docs --extras "${{ join(matrix.backend.extras, ' ') }}"

- uses: extractions/setup-just@v1
env:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/ibis-backends-skip-helper.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ on:
branches:
- main
- "*.x.x"
- the-epic-split
pull_request:
paths:
- "docs/**"
Expand All @@ -23,6 +24,7 @@ on:
branches:
- main
- "*.x.x"
- the-epic-split
merge_group:
jobs:
test_backends:
Expand Down
Loading
Loading