Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support start task with inconsistent upstream table schema for Optimsitic Sharding DDL #3786

Closed
GMHDBJD opened this issue Dec 8, 2021 · 0 comments · Fixed by #3903
Closed
Assignees
Labels
area/dm Issues or PRs related to DM. subject/new-feature Denotes an issue or pull request adding a new feature.

Comments

@GMHDBJD
Copy link
Contributor

GMHDBJD commented Dec 8, 2021

Is your feature request related to a problem?

When users have some inconsistent sharding tables, they can not start a task either in pessimistic mode or optimistic mode.

Describe the feature you'd like

start task with inconsistent upstream table schema for Optimsitic Sharding DDL
e.g.

table1(a int primary, b int)
table2(a int primary, c int)

Describe alternatives you've considered

No response

Teachability, Documentation, Adoption, Migration Strategy

To support different upstream table schema when tasks start, master need to know all upstream schema for a optimstic DDL.
The main logic may like

start-task
worker put all upstream table names which needed for the task to etcd
master watch table names and save to memory
worker fetch table infos from dump files or upstream databases and then flush to downstream checkpoint when entering sync unit
master receive a DDL and then fetch all sharding table infos from downstream for that DDL
master joined table base on all table infos
master return ddl to worker

When stop-task for a source, all the table infos/ddl infos/operations for that source in memory or etcd will be removed.

@GMHDBJD GMHDBJD added the subject/new-feature Denotes an issue or pull request adding a new feature. label Dec 8, 2021
@lichunzhu lichunzhu added the area/dm Issues or PRs related to DM. label Dec 10, 2021
@GMHDBJD GMHDBJD self-assigned this Dec 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dm Issues or PRs related to DM. subject/new-feature Denotes an issue or pull request adding a new feature.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants