Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/apply incremental schema changes #332

Merged

Conversation

canbekley
Copy link
Contributor

@canbekley canbekley commented Aug 2, 2024

Summary

implementation of schema changes for incremental models are currently missing. detecting schema changes during materialization would fallback to the legacy incremental strategy. this is not user-friendly and not feasible for very large models. schema changes for distributed_incremental materialization were not supported at all.

this pr contains

  • support for applying schema changes for incremental and distributed_incremental materializations
  • implementation of the on_schema_change='sync_all_column' strategy as per official dbt specs
  • fix of an issue in distributed_incremental materialzation and on_schema_change='ignore', where new columns in the target relation would break the dbt run during creation of the __dbt_new_data_ tables
  • fix of an issue in distributed_incremental materialzation, where column order changes would break the dbt run (or insert data to the wrong columns)

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG
  • For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

@CLAassistant
Copy link

CLAassistant commented Aug 2, 2024

CLA assistant check
All committers have signed the CLA.

@BentsiLeviav
Copy link
Contributor

Github changed its docker-compose command so all the tests failed. I adjusted it in #334 and merged it to main.
@canbekley Could you please merge main to your branch?

@BentsiLeviav BentsiLeviav merged commit a9042ad into ClickHouse:main Aug 21, 2024
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants