ddl: support alter algorithm INPLACE/INSTANT #8811

winkyao · 2018-12-25T08:29:51Z

What problem does this PR solve?

For #8428 , this is the first step to support EXPLAIN DDL(#8429).

MySQL support COPY/INPLACE/INSTANT/DEFAULT alter algorithm, you can get the details in https://dev.mysql.com/doc/refman/8.0/en/alter-table.html#alter-table-performance

In TiDB, we only support INPLACE/INSTANT, and the most algorithm of the alter operations is the INSTANT, which means

Operations only modify metadata in the data dictionary. No exclusive metadata locks are taken on the table during preparation and execution, and table data is unaffected, making operations instantaneous. Concurrent DML is permitted.

The add index using INPLACE algorithm, but not exactly the same with MySQL, the difference with MySQL is that TiDB never blocks the DML and never have a metadata lock on the table:

Operations avoid copying table data but may rebuild the table in place. An exclusive metadata lock on the table may be taken briefly during the preparation and execution phases of the operation. Typically, concurrent DML is supported.

If user specifies the unsupported algorithm of the alter operation, a wanrning ERROR 1846 (0A000): ALGORITHM=COPY is not supported. Reason: Cannot alter table by COPY. Try ALGORITHM=INSTANT. will be returned:

mysql> create table t(a int);
Query OK, 0 rows affected (0.00 sec)

mysql> alter table t comment = 'test', ALGORITHM=COPY;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> show warnings;
+-------+------+---------------------------------------------------------------------------------------------+
| Level | Code | Message                                                                                     |
+-------+------+---------------------------------------------------------------------------------------------+
| Error | 1846 | ALGORITHM=COPY is not supported. Reason: Cannot alter table by COPY. Try ALGORITHM=INSTANT. |
+-------+------+---------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Need to merge pingcap/parser#93 first.

Tests

Unit test

Code changes

Has exported function/method change

Side effects

Increased code complexity
Breaking backward compatibility

Related changes

Need to update the documentation
Need to be included in the release note

This change is

@zimulala @crazycs520 @morgo PTAL

winkyao · 2018-12-25T08:38:44Z

The circleci will be fixed after the pingcap/parser#93 merged.

shenli · 2018-12-26T01:11:44Z

ddl/ddl_algorithm_test.go

@@ -0,0 +1,107 @@
+// Copyright 2015 PingCAP, Inc.


2015 -> 2018

shenli · 2018-12-26T01:11:55Z

ddl/ddl_algorithm.go

+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSES/QL-LICENSE file.
+
+// Copyright 2015 PingCAP, Inc.


2015 -> 2018

shenli · 2018-12-26T01:12:35Z

go.mod

@@ -86,3 +86,5 @@ require (
 	sourcegraph.com/sourcegraph/appdash v0.0.0-20180531100431-4c381bd170b4
 	sourcegraph.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67
 )
+
+replace github.com/pingcap/parser => github.com/winkyao/parser v0.0.0-20181221090435-0eaad9528419


Should we remove this line?

this line will be removed after the pingcap/parser#93 is merged, but now we need to replace the parser to pass the ci.

morgo · 2019-01-03T21:01:04Z

@winkyao I spoke to MySQL users about the error on ALGORITHM=COPY behavior, and they reminded me that TiDB might be replicating from a MySQL system. A warning is better behavior, since it won't break replication if an upstream MySQL server sends this DDL event.

IANTHEREAL · 2019-01-04T02:45:31Z

@winkyao what do you think？if you are worried about warning messages are easy to ignore, we can also transform it(CPOY=>INSTANT) in replication app. But a warning message is definitely simpler if no other affects

winkyao · 2019-01-04T03:18:13Z

@morgo Well, return warning is also acceptable for me. I'll change this PR later.

…port

… into ddl_algorithm_support

winkyao · 2019-01-04T08:22:27Z

@morgo Done.

ddl/ddl_api.go

ddl/ddl_algorithm.go

crazycs520

LGTM

tiancaiamao · 2019-02-06T03:27:10Z

LGTM

…port

zimulala · 2019-02-19T02:28:50Z

ddl/ddl_api.go

@@ -1407,10 +1407,16 @@ func getCharsetAndCollateInTableOption(startIdx int, options []*ast.TableOption)
 	return
 }

-func (d *ddl) AlterTable(ctx sessionctx.Context, ident ast.Ident, specs []*ast.AlterTableSpec) (err error) {
-	// Only handle valid specs.
+// resolveAlterTableSpec resolve alter table algorithm and remove ignore table spec in specs.


s/resolve/resolves
s/remove/removes

zimulala · 2019-02-19T02:29:19Z

ddl/ddl_api.go

-func (d *ddl) AlterTable(ctx sessionctx.Context, ident ast.Ident, specs []*ast.AlterTableSpec) (err error) {
-	// Only handle valid specs.
+// resolveAlterTableSpec resolve alter table algorithm and remove ignore table spec in specs.
+// returns valied specs, and the occured error.


s/returns/Return

zimulala · 2019-02-19T02:32:18Z

ddl/ddl_api.go

 	for _, spec := range specs {
+		if spec.Tp == ast.AlterTableAlgorithm {
+			// find the last AlterTableAlgorithm.


s/find/Find

zimulala · 2019-02-19T02:47:18Z

ddl/db_integration_test.go

@@ -1210,3 +1210,75 @@ func (s *testIntegrationSuite) TestAlterColumn(c *C) {
 	_, err = s.tk.Exec("alter table mc modify column a bigint auto_increment") // Adds auto_increment should throw error
 	c.Assert(err, NotNil)
 }
+
+func (s *testIntegrationSuite) assertWarningExec(c *C, sql string) {


I think adding the warning message as an argument can be beneficial for calls to other functions.

…port

winkyao · 2019-02-19T08:25:04Z

@zimulala Done, PTAL

zimulala

LGTM

winkyao · 2019-02-19T08:47:11Z

/run-all-tests

winkyao · 2019-02-19T09:00:22Z

/run-common-test

winkyao · 2019-02-19T10:03:36Z

/run-common-test tidb-test=pr/741

winkyao · 2019-02-19T10:06:53Z

/run-common-test tidb-test=pr/741

winkyao · 2019-02-20T01:57:21Z

/run-integration-common-test tidb-test=pr/741

codecov-io · 2019-02-20T02:14:16Z

Codecov Report

Merging #8811 into master will increase coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #8811      +/-   ##
==========================================
+ Coverage   67.07%   67.15%   +0.08%     
==========================================
  Files         372      373       +1     
  Lines       78085    78113      +28     
==========================================
+ Hits        52373    52459      +86     
+ Misses      21043    20966      -77     
- Partials     4669     4688      +19

Impacted Files	Coverage Δ
ddl/ddl_algorithm.go	`100% <100%> (ø)`
ddl/ddl_api.go	`77.24% <100%> (+2.41%)`	⬆️
ddl/ddl.go	`89.58% <100%> (+0.04%)`	⬆️
util/systimemon/systime_mon.go	`80% <0%> (-20%)`	⬇️
ddl/delete_range.go	`75.13% <0%> (-4.24%)`	⬇️
store/tikv/lock_resolver.go	`41.7% <0%> (-0.95%)`	⬇️
executor/distsql.go	`72.24% <0%> (+0.45%)`	⬆️
executor/join.go	`79.42% <0%> (+0.52%)`	⬆️
expression/schema.go	`94.53% <0%> (+0.78%)`	⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7dc915...b6e2b28. Read the comment docs.

ddl: support alter algorithm INPLACE/INSTANT

470cbdd

winkyao added the type/new-feature label Dec 25, 2018

winkyao mentioned this pull request Dec 25, 2018

parser: add 'INSTANT' alter algorithm pingcap/parser#93

Merged

tidy

dd8fa6d

shenli added the component/DDL-need-LGT3 label Dec 26, 2018

shenli reviewed Dec 26, 2018

View reviewed changes

winkyao added 2 commits December 27, 2018 10:42

modify the copyright

ee0288c

Merge branch 'master' into ddl_algorithm_support

aa82a65

winkyao added 3 commits January 4, 2019 16:19

return warning instead of error.

5de30c5

Merge remote-tracking branch 'upstream/master' into ddl_algorithm_sup…

47c7d15

…port

Merge branch 'ddl_algorithm_support' of https://github.com/winkyao/tidb…

cf84689

… into ddl_algorithm_support

fix ci

c2c4d55

crazycs520 reviewed Jan 4, 2019

View reviewed changes

ddl/ddl_api.go Outdated Show resolved Hide resolved

crazycs520 reviewed Jan 4, 2019

View reviewed changes

ddl/ddl_algorithm.go Show resolved Hide resolved

winkyao added 2 commits January 4, 2019 16:28

remove replace

d3e57b8

update parser

cf056bb

crazycs520 reviewed Jan 4, 2019

View reviewed changes

tiancaiamao added the status/LGT2 Indicates that a PR has LGTM 2. label Feb 6, 2019

Merge remote-tracking branch 'upstream/master' into ddl_algorithm_sup…

ab0a613

…port

zimulala reviewed Feb 19, 2019

View reviewed changes

winkyao added 3 commits February 19, 2019 15:56

update parser

f0abc89

Merge remote-tracking branch 'upstream/master' into ddl_algorithm_sup…

25f0b20

…port

address comments

d4b9de2

zimulala previously approved these changes Feb 19, 2019

View reviewed changes

zimulala added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Feb 19, 2019

update parser

52125ab

winkyao dismissed zimulala’s stale review via 52125ab February 19, 2019 08:42

Merge branch 'master' into ddl_algorithm_support

cdf3652

zimulala approved these changes Feb 19, 2019

View reviewed changes

winkyao added the status/all tests passed label Feb 20, 2019

Merge branch 'master' into ddl_algorithm_support

b6e2b28

winkyao merged commit dec3ab6 into pingcap:master Feb 20, 2019

winkyao deleted the ddl_algorithm_support branch February 20, 2019 02:22

you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: support alter algorithm INPLACE/INSTANT #8811

ddl: support alter algorithm INPLACE/INSTANT #8811

winkyao commented Dec 25, 2018 •

edited

Loading

winkyao commented Dec 25, 2018 •

edited

Loading

shenli Dec 26, 2018

shenli Dec 26, 2018

shenli Dec 26, 2018

winkyao Dec 26, 2018 •

edited

Loading

morgo commented Jan 3, 2019

IANTHEREAL commented Jan 4, 2019 •

edited

Loading

winkyao commented Jan 4, 2019

winkyao commented Jan 4, 2019

crazycs520 left a comment

tiancaiamao commented Feb 6, 2019

zimulala Feb 19, 2019

zimulala Feb 19, 2019

zimulala Feb 19, 2019

zimulala Feb 19, 2019

winkyao commented Feb 19, 2019

zimulala left a comment

winkyao commented Feb 19, 2019

winkyao commented Feb 19, 2019

winkyao commented Feb 19, 2019

winkyao commented Feb 19, 2019

winkyao commented Feb 20, 2019

codecov-io commented Feb 20, 2019

ddl: support alter algorithm INPLACE/INSTANT #8811

ddl: support alter algorithm INPLACE/INSTANT #8811

Conversation

winkyao commented Dec 25, 2018 • edited Loading

What problem does this PR solve?

winkyao commented Dec 25, 2018 • edited Loading

shenli Dec 26, 2018

Choose a reason for hiding this comment

shenli Dec 26, 2018

Choose a reason for hiding this comment

shenli Dec 26, 2018

Choose a reason for hiding this comment

winkyao Dec 26, 2018 • edited Loading

Choose a reason for hiding this comment

morgo commented Jan 3, 2019

IANTHEREAL commented Jan 4, 2019 • edited Loading

winkyao commented Jan 4, 2019

winkyao commented Jan 4, 2019

crazycs520 left a comment

Choose a reason for hiding this comment

tiancaiamao commented Feb 6, 2019

zimulala Feb 19, 2019

Choose a reason for hiding this comment

zimulala Feb 19, 2019

Choose a reason for hiding this comment

zimulala Feb 19, 2019

Choose a reason for hiding this comment

zimulala Feb 19, 2019

Choose a reason for hiding this comment

winkyao commented Feb 19, 2019

zimulala left a comment

Choose a reason for hiding this comment

winkyao commented Feb 19, 2019

winkyao commented Feb 19, 2019

winkyao commented Feb 19, 2019

winkyao commented Feb 19, 2019

winkyao commented Feb 20, 2019

codecov-io commented Feb 20, 2019

Codecov Report

winkyao commented Dec 25, 2018 •

edited

Loading

winkyao commented Dec 25, 2018 •

edited

Loading

winkyao Dec 26, 2018 •

edited

Loading

IANTHEREAL commented Jan 4, 2019 •

edited

Loading