Distributed incremental materialization #172

gladkikhtutu · 2023-07-21T08:29:01Z

Created distributed incremental materialization with docs.

To add tests, clickhouse with cluster should be added in docker compose.

# Conflicts: # dbt/include/clickhouse/macros/materializations/incremental.sql

gfunc · 2023-07-25T02:08:09Z

IMO it is crucial to enable insert_distributed_sync in ClickHouse for distributed table inserts for dbt usage ( the default is disabled for ClickHouse 22.12). Otherwise, downstream calculation could be wrong if the distributed insert is not finished in time. Maybe add a check SELECT value FROM system.settings WHERE name = 'insert_distributed_sync'; as prehook, or add SET insert_distributed_sync = 1 before inserting SQL run, or a notice in README?

gladkikhtutu · 2023-07-26T07:57:40Z

IMO it is crucial to enable insert_distributed_sync in ClickHouse for distributed table inserts for dbt usage ( the default is disabled for ClickHouse 22.12). Otherwise, downstream calculation could be wrong if the distributed insert is not finished in time. Maybe add a check SELECT value FROM system.settings WHERE name = 'insert_distributed_sync'; as prehook, or add SET insert_distributed_sync = 1 before inserting SQL run, or a notice in README?

Yes, I added warning about insert_distributed_sync to README and exception in materialization if insert_distributed_sync != 1. Don't know, if SET insert_distributed_sync = 1 before inserting SQL is a correct solution, maybe should think about it in seperate PR.

genzgd

Thanks for the hard work here! My comments are generally minor and before releasing I will probably add some notes to the documentation that this is "experimental" and not currently part of our test suite, but I think it's pretty close.

README.md

dbt/include/clickhouse/macros/materializations/distributed_table.sql

* distributed table materialization * fix rebase * PR fixes * distributed incremental materialization * fix * fix * add insert_distributed_sync to README.md * add checks on insert_distributed_sync * add checks on insert_distributed_sync * review fixes

* fix: SYSTEM SYNC REPLICA for on_cluster_clause (ClickHouse#156) * fix SYSTEM SYNC REPLICA * add schema * Update version and pypi job * Fix incompatible return type (ClickHouse#162) * Distributed table materialization (ClickHouse#163) * distributed table materialization * fix rebase * PR fixes * Bump version * Tweak PyPI build Python release * Add space to exchange_tables_atomic macro (ClickHouse#168) * Add space to exchange_tables_atomic macro This changes the SYSTEM SYNC REPLICA query to have a space between the ON CLUSTER clause and the table name. * Move whitespace to on_cluster_clause * Fix bad logging/error handling (ClickHouse#170) * Distributed incremental materialization (ClickHouse#172) * distributed table materialization * fix rebase * PR fixes * distributed incremental materialization * fix * fix * add insert_distributed_sync to README.md * add checks on insert_distributed_sync * add checks on insert_distributed_sync * review fixes * Update version and tweak docs * Lw delete set fix (ClickHouse#174) * Move lightweight delete settings to per query for HTTP stickiness fix * Minor cleanup and doc updates * Fix legacy incremental materialization (ClickHouse#178) * fix: distributed_table materialization issue (ClickHouse#184) * Bump version and changelog (ClickHouse#185) * cluster names containing dash characters (ClickHouse#198) (ClickHouse#200) Co-authored-by: the4thamigo-uk <the4thamigo-uk> * Add basic error test, fix minor merge conflict (ClickHouse#202) * Cluster setting and Distributed Table tests (ClickHouse#186) * added can_on_cluster var in ClickhouseRelation * add tests for cluster * fix lint issue * debug set cluster env variable * debug test * debug and add tests * skip distributed table grant test * debug workflow * debug workflow * debug test * add tests fro distributed_incremental * fix zk path error * fix wrong alias for distributed materializations update aliase test * update base on review * Update version and CHANGELOG, incorporate cluster name fix (ClickHouse#203) * Release 1 5 0 (ClickHouse#210) * Initial 1.5.0 commit * Reorganize basic tests * Fix lint * Add case sensitive cache * Fix s3 bucket bug * Checkpoint for constraints/contracts * Fix native column query * Loosen replication test * Checkpoint for constraints tests * Checkpoint for constraints tests * Add rendering of model level CHECK constraints * Fix lint * Reorganize test files * Add one hooks test * Fix lint * Update test and dependency versions. (ClickHouse#211) * Adjust the wrapper parenthesis around the table materialization sql code (ClickHouse#212) * Update for 1.5.1 bug fix * Fix creation of replicated tables when using legacy materialization (ClickHouse#208) * On cluster sync cleanup * Bug fixes related to model settings. (ClickHouse#214) * Add materialization macro for materialized view (ClickHouse#207) * Add materialization macro for materialized view * fix isort issues in materialized view test * Release 1 6 0 (ClickHouse#215) * Initial dbt 1.6 update * Add skipped clone test * Clean up MV PR * Release 1 6 1 (ClickHouse#217) * Identifier quoting checkpoint * Identifier quoting checkpoint * Fix distributed table local quoting * Fix issues with deduplication settings * Release 1 6 2 (ClickHouse#219) * Limited fix to completely broken `on_schema_change` * Tweak changelog * Release 1 7 0 (ClickHouse#220) * Initial dependency updates for 1.7.x * Initial dependency updates for 1.7.x * Correctly warn or error if light weight deletes not available * Wrap columns_in_query query in select statement (ClickHouse#222) * Wrap columns_in_query query in select statement * formatting * Update changelog * allows to add a comment in table's or view's metadata * add settings_section flag as comment for code using settings * override test sql macro and add limit-placer macro * update CHANGELOG.md * fix: use correct schema for MV target tables (ClickHouse#244) * fix: use correct schema when updating MVs The existing implementation passes just the name for `target_table`, which ultimately means that the target schema is not included when the final SQL is generated. By passing the entire relation object, the correct target schema will be present in the final SQL. * update MV tests Provide a custom schema to make sure that the full target table name (schema + relation name) is included in the CREATE MATERIALIZED VIEW statement * Update changelog * rename end of query flag * Bug/223 relationship test with limit (ClickHouse#245) * add settings_section flag as comment for code using settings * override test sql macro and add limit-placer macro * update CHANGELOG.md * rename end of query flag * Revert "Bug/223 relationship test with limit (ClickHouse#245)" (ClickHouse#247) This reverts commit d8afb93. * always return --end_of_sql when asking for settings * Add model settings based on materialization type * support setting clause on view creation * edit CHANGELOG.md * Bump version and tweak changelog * change list syntax to satisfy lint test * change list syntax to satisfy lint test * change imports order to satisfy lint test * Add typing to satisfy lint * Add snapshot materialization to default settings * Fix tests - add distributed_table and distributed_incremental materializations * Fix tests - make sure to call the get_model_settings only when materialization is view * clean up recent changelog * Add materialization macro for dictionaries * address lint issue in dictionary test * address lint issue with enum * Fix model settings with custom materialization * Release 1.7.4 housekeeping (ClickHouse#261) * Bump black from 23.11.0 to 24.3.0 (ClickHouse#259) Bumps [black](https://github.com/psf/black) from 23.11.0 to 24.3.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](psf/black@23.11.0...24.3.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Release 1 7 5 (ClickHouse#265) * Release 1.7.5 housekeeping * Upgrade setuptools requirement for clickhouse_driver install * Remove flake8 checks for the moment * Update workflow actions * Fix black comma * fix(clients): add newlines around subquery when retrieving columns to avoid a syntax error (ClickHouse#262) * Fix lint * lazy load agate (ClickHouse#263) * feat: add TTL support (ClickHouse#254) * Fix lint * Update table relation after exchange command (ClickHouse#230) Related to ClickHouse#226 * feat: allow to add connection overrides for dictionaries (ClickHouse#267) * Housekeeping for 1.7.6 release (ClickHouse#268) * Revert "allows to add a comment in table's or view's metadata" * Fix bool_or behavior (ClickHouse#270) * feat: support column codecs * Use Column.data_type in ClickHouseAdapter.format_columns * Always apply query_settings in clickhouse__insert_into macro * Add ClickHouseColumn.is_low_cardinality * Update column type test cases for LowCardinality * Omit empty dictionary connection_overrides from materialization DDL --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Schum <[email protected]> Co-authored-by: Geoff Genz <[email protected]> Co-authored-by: Sergey Reshetnikov <[email protected]> Co-authored-by: gladkikhtutu <[email protected]> Co-authored-by: Damir Basic Knezevic <[email protected]> Co-authored-by: Zhenbang <[email protected]> Co-authored-by: Andy <[email protected]> Co-authored-by: gfunc <[email protected]> Co-authored-by: Kristof Szaloki <[email protected]> Co-authored-by: Steven Reitsma <[email protected]> Co-authored-by: Rory Sawyer <[email protected]> Co-authored-by: ptemarvelde <[email protected]> Co-authored-by: Dmitrii Tcimokha <[email protected]> Co-authored-by: bentsileviav <[email protected]> Co-authored-by: Dmitriy Sokolov <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: triou <[email protected]> Co-authored-by: Daniel Reeves <[email protected]> Co-authored-by: Cristhian Garcia <[email protected]> Co-authored-by: Thomas Schmidt <[email protected]> Co-authored-by: scrawfor <[email protected]> Co-authored-by: Robin Norgren <[email protected]>

gladkikhtutu and others added 8 commits June 19, 2023 19:36

distributed table materialization

b1f34c2

fix rebase

4579214

PR fixes

c280bfe

distributed incremental materialization

ab25314

Merge branch 'ClickHouse:main' into main

18c7642

Merge branch 'main' into distributed_incremental_mat

489d001

# Conflicts: # dbt/include/clickhouse/macros/materializations/incremental.sql

fix

61a9a65

fix

55903b1

gladkikhtutu marked this pull request as ready for review July 21, 2023 08:44

gladkikhtutu changed the title ~~Distributed incremental mat~~ Distributed incremental materialization Jul 21, 2023

gladkikhtutu mentioned this pull request Jul 24, 2023

Support for Distributed table engine #14

Closed

gladkikhtutu added 3 commits July 26, 2023 09:48

add insert_distributed_sync to README.md

852db1f

add checks on insert_distributed_sync

3777c38

add checks on insert_distributed_sync

3c94de5

genzgd requested changes Jul 26, 2023

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

dbt/include/clickhouse/macros/materializations/distributed_table.sql Outdated Show resolved Hide resolved

review fixes

c3d0ef0

gladkikhtutu requested a review from genzgd July 27, 2023 09:57

genzgd merged commit 8a2731f into ClickHouse:main Jul 27, 2023

genzgd mentioned this pull request Nov 30, 2023

First version of distributed materialization #15

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distributed incremental materialization #172

Distributed incremental materialization #172

gladkikhtutu commented Jul 21, 2023 •

edited

Loading

gfunc commented Jul 25, 2023 •

edited

Loading

gladkikhtutu commented Jul 26, 2023 •

edited

Loading

genzgd left a comment

Distributed incremental materialization #172

Distributed incremental materialization #172

Conversation

gladkikhtutu commented Jul 21, 2023 • edited Loading

gfunc commented Jul 25, 2023 • edited Loading

gladkikhtutu commented Jul 26, 2023 • edited Loading

genzgd left a comment

Choose a reason for hiding this comment

gladkikhtutu commented Jul 21, 2023 •

edited

Loading

gfunc commented Jul 25, 2023 •

edited

Loading

gladkikhtutu commented Jul 26, 2023 •

edited

Loading