perf(backends): speed up most memtable existence checks #10067

cpcloud · 2024-09-09T18:09:26Z

Adds some slightly faster checks for memtable existence. At some point we may want to expand this to non-memtables, but for now these are only implemented for memtables.

ibis/backends/sqlite/__init__.py

jcrist

Just a couple questions. Once #9695 lands we could also change the default implementation to

try:
    self.table(name)
    return True
except TableNotFound:
    return False

ibis/backends/sqlite/__init__.py

ibis/backends/trino/__init__.py

cpcloud · 2024-09-10T20:13:42Z

~~Going to leave off postgres for now.~~ Looking into it.

cpcloud · 2024-09-11T11:48:21Z

Ok, I managed to audit all the backends except for Druid and Flink. Those can be tackled in a follow-up.

cpcloud · 2024-09-11T12:50:18Z

ibis/backends/pyspark/__init__.py

@@ -411,11 +411,18 @@ def _register_udfs(self, expr: ir.Expr) -> None:
            self._session.udf.register(f"unwrap_json_{typ.__name__}", unwrap_json(typ))
        self._session.udf.register("unwrap_json_float", unwrap_json_float)

+    def _in_memory_table_exists(self, name: str) -> bool:
+        sql = f"SHOW TABLES IN {self.current_database} LIKE '{name}'"


Somehow self._session.catalog.tableExists doesn't give the same answer as the code I have here, even when passing a fully quoted default.$memtable (the second dbName parameter is deprecated and the deprecation warning suggests passing the fully qualified name instead.)

… TABLE`

cpcloud · 2024-09-11T13:15:38Z

Clouds are passing:

…/ibis on  faster-memtable-existence-checks is 📦 v9.4.0 via 🐍 v3.10.14 via ❄️   impure (ibis-3.10.14-env)
❯ pytest -m 'snowflake or bigquery' -n 8 --dist loadgroup --snapshot-update -q
bringing up nodes...
x.....................................s....x.....x..s.........................x..........x........................x........xx....x..........................x..xx..................xx.............xx.........x [  5%]
.....................x....x...x.....x..............x........................x..............x...x..........................x.....xx.xx.........................x...........x.......x.....x................x.... [ 10%]
...........xx.....x.x........xx...x....x..x...x....x.x.........xx.xx.xxxxxxxxxx.x.x..x..x...xx..xxxx...x.x.xx..xxx..xx.x.....xxx.x.....xxx.xx.xxx.x..xxxx.x...x....x.x.xxx.xx.xx.....xx.x...xx.xx..xxx..xxx... [ 15%]
x.x.x.x.x..xx.x......................x.xx.........x.xx.x.s.ssssssssssssssssssss..xx.......x........................................x..................x..........x.x..............................x........... [ 20%]
x.................xx...................x..x....x......x......x...x.x.....x...........x.........x..s..x........x......x.....................x.................x......x............x.......x.x.x.xx....xxx.xs..x [ 25%]
..xx...x................................x........s...x.......ssssssssssssssssssssssssssssssssssssssssssssssssssss.ssssssssssssssssssssssssssss.sssssssssssssssssss........x.............s..............x...... [ 30%]
.....x.....................................x........x........................x............................................................x..x................x..........x..xx..xx........x...x............... [ 35%]
....................................................................x...........x..xx...x....x.x...........................................x................................x.......................x......... [ 40%]
.x....................x.....................x..x.....................................xxx...x........................................x...x...x.....x.x............x.x.........x.......xx.............x......... [ 45%]
....x......s........x............x............x...x.x...x.................x.............x.................x.......x......x...x.....x.x.........x.xxx....................x.x.........x.......................x. [ 50%]
.x.x...x..........x............x........................X......x........x.xxx.........x........x....x...x.......x.x.x.x...x......xxx.X........x..x.x..x.x.....xx.x....xxX..X....x........X....x...xxxxxx.xx.xx [ 56%]
x.xxxx.xx.xxxx.xxx.xxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx..x.x.xx.....................x............x..x...x......................x.x... [ 61%]
........x......x.x.x.....x...x..............................x...x.x..................x......x...............xx..x...................xx.x.........x.x.s..s...x.......x.........xxxXx.xs...s........x.x......... [ 66%]
..x...............x....x.x.xxx...xx.xx.x..x...x...x...x.x..........................s....ss.......x................................x..............x....x..xx.........x.......x.....x............x.xx.x.....x.xx [ 71%]
...x..xx...x....xxx..x.....x............x..xx.x.....x.x.........xx.x..x..............x.........x...............x........................................x..................................................... [ 76%]
.......x.....x...........................................xx....x.............xx................................................x..x..........x....................x.x..............x.......x..x.....xx.x..xxxx [ 81%]
xxxxx.....xx.xx.x.xx....x.xxxxx..x..x..xx...............x....x....xx.....x.............................x..................................................................................x................... [ 86%]
....x...................................................................................x..xx.....................................x........................................................................... [ 91%]
.............................................................................................................................................s........x..x.................................................... [ 96%]
....................................................................s...s.......x........................................s.....                                                                                [100%]
3330 passed, 138 skipped, 567 xfailed, 6 xpassed in 922.55s (0:15:22)

…#10067)

cpcloud added this to the 9.5 milestone Sep 9, 2024

cpcloud added postgres The PostgreSQL backend sqlite The SQLite backend pyspark The Apache PySpark backend snowflake The Snowflake backend risingwave The RisingWave backend labels Sep 9, 2024

cpcloud requested review from jcrist and gforsyth September 9, 2024 18:09

cpcloud added trino The Trino backend performance Issues related to ibis's performance labels Sep 9, 2024

cpcloud commented Sep 9, 2024

View reviewed changes

ibis/backends/sqlite/__init__.py Outdated Show resolved Hide resolved

jcrist approved these changes Sep 9, 2024

View reviewed changes

ibis/backends/sqlite/__init__.py Outdated Show resolved Hide resolved

ibis/backends/trino/__init__.py Outdated Show resolved Hide resolved

cpcloud force-pushed the faster-memtable-existence-checks branch 2 times, most recently from faf78a6 to 89d4253 Compare September 11, 2024 11:46

cpcloud added clickhouse The ClickHouse backend mysql The MySQL backend mssql The Microsoft SQL Server backend oracle The Oracle backend exasol Issues related to the exasol backend labels Sep 11, 2024

cpcloud force-pushed the faster-memtable-existence-checks branch 2 times, most recently from 4a0537b to 4f2ab29 Compare September 11, 2024 12:45

cpcloud commented Sep 11, 2024

View reviewed changes

cpcloud added 4 commits September 11, 2024 08:56

perf(snowflake): speed up memtable existence check by using `DESCRIBE…

7e33a3e

… TABLE`

perf(trino): use DESCRIBE for faster table existence checking

7a3797e

perf(postgres): avoid listing all tables to query table existence

5d5e1ab

perf(risingwave): use faster check for existence

0794a5b

cpcloud force-pushed the faster-memtable-existence-checks branch from 4f2ab29 to 0188256 Compare September 11, 2024 12:56

cpcloud changed the title ~~perf(backends): speed up some memtable existence checks~~ perf(backends): speed up most memtable existence checks Sep 11, 2024

cpcloud added 7 commits September 11, 2024 09:15

perf(sqlite): use faster table existence check

66922ce

perf(pyspark): faster memtable existence check

69d6e22

perf(clickhouse): improve table existence check

a5a0ba7

perf(mysql): improve table existence check

30cc6c4

perf(exasol): speed up existence check

1c7c29c

perf(mssql): speed up existence check

7ea0ee4

perf(oracle): speed up existence check

bcbf310

cpcloud force-pushed the faster-memtable-existence-checks branch from 0188256 to bcbf310 Compare September 11, 2024 13:15

cpcloud enabled auto-merge (squash) September 11, 2024 13:27

cpcloud merged commit a205ab7 into ibis-project:main Sep 11, 2024
81 checks passed

cpcloud deleted the faster-memtable-existence-checks branch September 11, 2024 13:40

ncclementi mentioned this pull request Sep 13, 2024

refactor: standardize _in_memory_table_exists #10123

Open

jitingxu1 mentioned this pull request Sep 19, 2024

bug(trino): cannot create table with large size data in trino #10178

Closed

1 task

ncclementi pushed a commit to ncclementi/ibis that referenced this pull request Sep 24, 2024

perf(backends): speed up most memtable existence checks (ibis-project…

20e0bb7

…#10067)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(backends): speed up most memtable existence checks #10067

perf(backends): speed up most memtable existence checks #10067

cpcloud commented Sep 9, 2024

jcrist left a comment

cpcloud commented Sep 10, 2024 •

edited

Loading

cpcloud commented Sep 11, 2024

cpcloud Sep 11, 2024 •

edited

Loading

cpcloud commented Sep 11, 2024

perf(backends): speed up most memtable existence checks #10067

perf(backends): speed up most memtable existence checks #10067

Conversation

cpcloud commented Sep 9, 2024

jcrist left a comment

Choose a reason for hiding this comment

cpcloud commented Sep 10, 2024 • edited Loading

cpcloud commented Sep 11, 2024

cpcloud Sep 11, 2024 • edited Loading

Choose a reason for hiding this comment

cpcloud commented Sep 11, 2024

cpcloud commented Sep 10, 2024 •

edited

Loading

cpcloud Sep 11, 2024 •

edited

Loading