ddl, executor : add auto_random column option #13127

tangenta · 2019-11-05T01:09:33Z

What problem does this PR solve?

A column attribute named auto_random is added, which is used to mitigate the hot spots on a single server when high-volume writes happen in the table with integer primary key.

If the primary key constraint of a table is specified on an integer column c, the row values in c are directly used as the row id instead of implicit allocation during TiDB's key-value construction.
In this case, we say PKIsHandle.

This PR adds a keyword, auto_random. The usage is as follows:

create table t(a bigint auto_random(3) primary key, b varchar(10))
create table t(a int auto_random(3), b char, primary key (a))
create table t(a int auto_random, b char, primary key (a))

The number n following auto_random means the highest nth bits of the column value is reserved for sharding. Omitting this value is equivalent to setting it to 5. In other words, auto_random has the same meaning as auto_random(5).

Similar to auto_increment, users can also omit the column value when inserting a row, at which point the value is allocated by TiDB automatically.

There are some features about these allocated values:

The values are NOT guaranteed to be monotonically increasing or continuous.
Each of these values is unique.
The highest nth bits of each value is hashed according to the StartTS of a transaction.

Users can also specify the column value explicitly. In this case, TiDB will save the value intact.

Note: before using auto_random, allow-auto-random needs to be set to true in the [experimental] section of the configuration file.

Limitations:

Modification on the column type with auto_random is not supported.
Modification on the value of auto_random is strongly not recommended.
The addition or removal of auto_random column attribute is not supported.
auto_increment and auto_random are not allowed to be specified in the same column at the same time.

What is changed and how it works?

Change the configuration definition
Add some checks in buildTableInfo and getModifiableColumnJob.
Adjust row value when doing an insertion.

Check List

Tests

Unit test
Integration test

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change

Side effects

Possible performance regression
Increased code complexity

Related changes

Need to update the documentation

Release note

codecov · 2019-11-11T16:17:10Z

Codecov Report

❗ No coverage uploaded for pull request base (master@09cb919). Click here to learn what that means.
The diff coverage is 77.7777%.

@@             Coverage Diff             @@
##             master     #13127   +/-   ##
===========================================
  Coverage          ?   80.1654%           
===========================================
  Files             ?        483           
  Lines             ?     121626           
  Branches          ?          0           
===========================================
  Hits              ?      97502           
  Misses            ?      16351           
  Partials          ?       7773

AilinKid

Haven't finished review yet

config/config.go

AilinKid · 2019-11-17T08:03:27Z

executor/insert_common.go

@@ -792,6 +807,62 @@ func getAutoRecordID(d types.Datum, target *types.FieldType, isInsert bool) (int
 	return recordID, nil
 }

+func (e *InsertValues) adjustAutoShardDatum(ctx context.Context, d types.Datum, hasValue bool, c *table.Column) (types.Datum, error) {
+	if !hasValue || d.IsNull() {


should take retry into consideration.

Can we simply ignore the retry, just allocate another id instead?

It's better to cache the auto shard id in retryInfo, but we can do that in another PR, after all this PR is very large already.

Maybe can plus lazy batch alloc than alloc one by one in next pr, by now, it's ok here.

executor/insert_common.go

config/config.go

ddl/db_integration_test.go

config/config.go

executor/ddl_test.go

ddl/ddl_api.go

bb7133 · 2019-11-21T05:25:10Z

executor/insert_common.go

@@ -792,6 +807,62 @@ func getAutoRecordID(d types.Datum, target *types.FieldType, isInsert bool) (int
 	return recordID, nil
 }

+func (e *InsertValues) adjustAutoShardDatum(ctx context.Context, d types.Datum, hasValue bool, c *table.Column) (types.Datum, error) {
+	if !hasValue || d.IsNull() {


It's better to cache the auto shard id in retryInfo, but we can do that in another PR, after all this PR is very large already.

executor/insert_common.go

table/tables/tables.go

crazycs520

LGTM

crazycs520 · 2019-12-23T05:27:18Z

/run-all-tests

sre-bot · 2019-12-23T05:33:43Z

Your auto merge job has been accepted, waiting for 14177

sre-bot · 2019-12-23T05:39:18Z

/run-all-tests

sre-bot · 2019-12-23T05:42:16Z

@tangenta merge failed.

tangenta · 2019-12-23T05:45:27Z

/run-unit-test

tangenta · 2019-12-23T05:52:15Z

/run-unit-test

tangenta · 2019-12-23T06:47:52Z

/run-unit-test

tangenta added component/DDL-need-LGT3 type/new-feature labels Nov 5, 2019

wjhuang2016 added the sig/execution SIG execution label Nov 5, 2019

tangenta force-pushed the auto-shard branch 6 times, most recently from 84e1b0d to c5fd251 Compare November 11, 2019 09:53

zimulala self-requested a review November 11, 2019 12:06

tangenta force-pushed the auto-shard branch 3 times, most recently from d7e632b to 15887eb Compare November 11, 2019 14:20

tangenta mentioned this pull request Nov 11, 2019

parser: support syntax for AUTO_RANDOM pingcap/parser#571

Merged

tangenta force-pushed the auto-shard branch 2 times, most recently from b502cb4 to cb7933d Compare November 11, 2019 16:17

AilinKid changed the title ~~ddl,executor: add auto_shard_bits column option~~ ddl, executor : add auto_shard_bits column option. Nov 12, 2019

AilinKid self-requested a review November 12, 2019 05:56

tangenta force-pushed the auto-shard branch from cb7933d to b75725b Compare November 13, 2019 04:49

tangenta mentioned this pull request Nov 14, 2019

Is there a complete solution to the problem of autoincrement primary keys? #13437

Closed

AilinKid reviewed Nov 17, 2019

View reviewed changes

bb7133 reviewed Nov 18, 2019

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

config/config.go Outdated Show resolved Hide resolved

config/config.go Outdated Show resolved Hide resolved

bb7133 reviewed Nov 18, 2019

View reviewed changes

ddl/db_integration_test.go Outdated Show resolved Hide resolved

bb7133 reviewed Nov 18, 2019

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

tangenta force-pushed the auto-shard branch 2 times, most recently from c610f8e to 0ed14db Compare November 18, 2019 08:50

crazycs520 reviewed Nov 18, 2019

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

bb7133 reviewed Nov 20, 2019

View reviewed changes

executor/ddl_test.go Outdated Show resolved Hide resolved

executor/ddl_test.go Outdated Show resolved Hide resolved

bb7133 reviewed Nov 21, 2019

View reviewed changes

tangenta requested a review from crazycs520 December 22, 2019 11:55

tangenta force-pushed the auto-shard branch from 8ae5f75 to e4861d2 Compare December 23, 2019 02:37

ddl,executor: add auto_random column option

917aafc

tangenta force-pushed the auto-shard branch from e4861d2 to 917aafc Compare December 23, 2019 02:46

crazycs520 approved these changes Dec 23, 2019

View reviewed changes

Merge branch 'master' into auto-shard

09cb919

bb7133 added status/LGT3 The PR has already had 3 LGTM. status/can-merge Indicates a PR has been approved by a committer. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Dec 23, 2019

Merge branch 'master' into auto-shard

8647d52

tangenta changed the title ~~ddl, executor : add auto_random column option.~~ ddl, executor : add auto_random column option Dec 23, 2019

tangenta merged commit fce9870 into pingcap:master Dec 23, 2019

tangenta mentioned this pull request Jan 20, 2020

ddl,executor: add auto_random column option (#13127, #14489) #14555

Merged

This was referenced Feb 4, 2020

reference: add user documentation for auto_random feature pingcap/docs-cn#2209

Merged

*: add auto_random id cache for statement retrying and table recover #14711

Merged

tangenta added a commit to tangenta/tidb that referenced this pull request Feb 12, 2020

ddl,executor: add auto_random column option (pingcap#13127)

17d9bc8

bb7133 pushed a commit that referenced this pull request Feb 12, 2020

ddl,executor: add auto_random column option (#13127, #14489) (#14555)

1516653

you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020

wjhuang2016 mentioned this pull request Mar 22, 2020

conf, roles: add the config of auto-random, expression and new-collation pingcap/tidb-ansible#1217

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl, executor : add auto_random column option #13127

ddl, executor : add auto_random column option #13127

tangenta commented Nov 5, 2019 •

edited by bb7133

Loading

codecov bot commented Nov 11, 2019 •

edited

Loading

AilinKid left a comment

AilinKid Nov 17, 2019

tangenta Nov 17, 2019 •

edited

Loading

bb7133 Nov 21, 2019

AilinKid Dec 5, 2019

bb7133 Nov 21, 2019

crazycs520 left a comment

crazycs520 commented Dec 23, 2019

sre-bot commented Dec 23, 2019

sre-bot commented Dec 23, 2019

sre-bot commented Dec 23, 2019

tangenta commented Dec 23, 2019

tangenta commented Dec 23, 2019

tangenta commented Dec 23, 2019

ddl, executor : add auto_random column option #13127

ddl, executor : add auto_random column option #13127

Conversation

tangenta commented Nov 5, 2019 • edited by bb7133 Loading

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Nov 11, 2019 • edited Loading

Codecov Report

AilinKid left a comment

Choose a reason for hiding this comment

AilinKid Nov 17, 2019

Choose a reason for hiding this comment

tangenta Nov 17, 2019 • edited Loading

Choose a reason for hiding this comment

bb7133 Nov 21, 2019

Choose a reason for hiding this comment

AilinKid Dec 5, 2019

Choose a reason for hiding this comment

bb7133 Nov 21, 2019

Choose a reason for hiding this comment

crazycs520 left a comment

Choose a reason for hiding this comment

crazycs520 commented Dec 23, 2019

sre-bot commented Dec 23, 2019

sre-bot commented Dec 23, 2019

sre-bot commented Dec 23, 2019

tangenta commented Dec 23, 2019

tangenta commented Dec 23, 2019

tangenta commented Dec 23, 2019

tangenta commented Nov 5, 2019 •

edited by bb7133

Loading

codecov bot commented Nov 11, 2019 •

edited

Loading

tangenta Nov 17, 2019 •

edited

Loading