Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Concurrent DDL Execution #18396

Closed
3 tasks
lonng opened this issue Jul 7, 2020 · 9 comments
Closed
3 tasks

Support Concurrent DDL Execution #18396

lonng opened this issue Jul 7, 2020 · 9 comments
Labels
feature/accepted This feature request is accepted by product managers priority/P1 The issue has P1 priority. sig/sql-infra SIG: SQL Infra type/feature-request Categorizes issue or PR as related to a new feature.

Comments

@lonng
Copy link
Contributor

lonng commented Jul 7, 2020

Feature Request

Currently, TiDB initial support for concurrent DDL
(The DDL of "add index" and the other types of DDL can be executed concurrently when they are on the different tables. Related to #6955). When more services are connected to the same cluster, the Same type of DDL statement is usually blocked between different services.

In this case, the DDL capabilities will block more service to use the TiDB database. But TiDB can do more things to extend the DDL capabilities, this PR proposes to support execute DDL parallelly

For example:
Teams A and B share the same TiDB cluster. Team A cannot do any add index operation when team B is executing a DDL alter table t1 add index idx_ab(a,b), which will block the team A upgrading service especially when the table t1 is huge (eg: greater than 1B rows).

But there are some risks if we support add index parallelly, eg: the cluster load will increase if add index parallelly, which will impact the cluster performance.

Category

Feature/Improvement

Value

3: Make TiDB support more scenarios.

TODO list

@lonng lonng added the type/feature-request Categorizes issue or PR as related to a new feature. label Jul 7, 2020
@zimulala zimulala added the sig/sql-infra SIG: SQL Infra label Jul 7, 2020
@IANTHEREAL IANTHEREAL added the priority/P1 The issue has P1 priority. label Jul 8, 2020
@zz-jason zz-jason removed their assignment Jul 9, 2020
@zhangjinpeng87
Copy link
Contributor

@lonng Would you mind to give more description for the necessary of DDL parallel execution, for example there already is a huge table's add index DDL job which is running, and when I want to add an index for a small table, I must wait for the previous add index job finished, this is not acceptable.

@zz-jason zz-jason changed the title Feature Request: Support DDL parallel execution Support Parallel DDL Execution Jul 14, 2020
@scsldb scsldb added the feature/accepted This feature request is accepted by product managers label Jul 21, 2020
@ghost
Copy link

ghost commented Sep 12, 2020

Would it be more correct to call this "concurrent ddl"? This distinguishes it from #19386 which is to implement parallelism across the cluster.

@lonng
Copy link
Contributor Author

lonng commented Sep 13, 2020

Would it be more correct to call this "concurrent ddl"? This distinguishes it from #19386 which is to implement parallelism across the cluster.

done

@bb7133
Copy link
Member

bb7133 commented Sep 14, 2020

Hi @nullnotnil , this feature is planned to finish in 2020 Q4.

@ghost ghost changed the title Support Parallel DDL Execution Support Concurrent DDL Execution Sep 14, 2020
@blacktear23
Copy link
Contributor

I think we also need a parallel level control and rate limitation policy to prevent DDL job eat all disk IO. And there also should be a method to online reduce the parallel level.

@morgo
Copy link
Contributor

morgo commented Dec 2, 2020

For online reconfiguration, may I suggest using SET GLOBAL sys_var = newvalue?

In #21424 I am working on a way for changing sessionvars to execute arbitrary code. It is a little bit more complicated for globalvars because the semantics need to be worked out for the local instance + other tidb instances.

@kolbe
Copy link
Contributor

kolbe commented Feb 16, 2021

Hello! Is there any update to the ETA for this feature? It seems that it is still not possible to concurrently add indexes to separate tables.

@lonng
Copy link
Contributor Author

lonng commented Feb 18, 2021

@kolbe Hi, this feature won't be scheduled to release in the short-term.

@bb7133
Copy link
Member

bb7133 commented Jul 6, 2023

Close this after #32031

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/accepted This feature request is accepted by product managers priority/P1 The issue has P1 priority. sig/sql-infra SIG: SQL Infra type/feature-request Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

No branches or pull requests