expression: add builtin function `tidb_decode_key()` #12193

crazycs520 · 2019-09-16T05:52:33Z

What problem does this PR solve?

Add tidb_decode_key() function to decode key.

What is changed and how it works?

-- decode region key.
mysql>select region_id, tidb_decode_key(start_key) as k from `TIKV_REGION_STATUS` where tidb_decode_key(start_key) like 'tableID=%' limit 3;
+-----------+---------------------------------------------------------------+
| region_id | k                                                             |
+-----------+---------------------------------------------------------------+
| 107       | tableID=43, handle=300000000                                  |
| 128       | tableID=43, indexID=1, indexValues=0380000000165A0BC003000000 |
| 154       | tableID=43, indexID=1, indexValues=03800000000EE6B28003000000 |
+-----------+---------------------------------------------------------------+

mysql>select tidb_decode_key( '7480000000000000FF2E5F698000000000FF0000010380000000FF05F5E10003000000FF0000000000000000F7')
+----------------------------------------------------------------------------------------------------------------+
| tidb_decode_key( '7480000000000000FF2E5F698000000000FF0000010380000000FF05F5E10003000000FF0000000000000000F7') |
+----------------------------------------------------------------------------------------------------------------+
| tableID=46, indexID=1, indexValues=038000000005F5E10003000000                                                  |
+----------------------------------------------------------------------------------------------------------------+

mysql>select tidb_decode_key( '7480000000000000FF2E5F728000000011FFE1A3000000000000FA')
+----------------------------------------------------------------------------+
| tidb_decode_key( '7480000000000000FF2E5F728000000011FFE1A3000000000000FA') |
+----------------------------------------------------------------------------+
| tableID=46, handle=300000000                                               |
+----------------------------------------------------------------------------+

-- if the key is not record/index kye, just do nothing.
mysql>select tidb_decode_key( '7480000000000000FF2E5F698000000000FF0000010380000000FF05F5E10003000000FF0000000000000000F')
+---------------------------------------------------------------------------------------------------------------+
| tidb_decode_key( '7480000000000000FF2E5F698000000000FF0000010380000000FF05F5E10003000000FF0000000000000000F') |
+---------------------------------------------------------------------------------------------------------------+
| 7480000000000000FF2E5F698000000000FF0000010380000000FF05F5E10003000000FF0000000000000000F                     |
+---------------------------------------------------------------------------------------------------------------+

Other discuss

The decoded key format currently maybe not very well, If you have any suggestions, Please comment, Thanks very much.

Maybe we can use the format in show table region.

+-----------+-------------------------------------+-------------------------------------+-----------+-----------------+---------------+------------+---------------+------------+----------------------+------------------+
| REGION_ID | START_KEY                           | END_KEY                             | LEADER_ID | LEADER_STORE_ID | PEERS         | SCATTERING | WRITTEN_BYTES | READ_BYTES | APPROXIMATE_SIZE(MB) | APPROXIMATE_KEYS |
+-----------+-------------------------------------+-------------------------------------+-----------+-----------------+---------------+------------+---------------+------------+----------------------+------------------+
| 95        | t_43_r                              | t_43_r_100000000                    | 112       | 9               | 98, 112, 111  | 0          | 129           | 0          | 1                    | 0                |
| 99        | t_43_r_100000000                    | t_43_r_200000000                    | 113       | 3               | 101, 113, 165 | 0          | 0             | 0          | 1                    | 0                |
| 103       | t_43_r_200000000                    | t_43_r_300000000                    | 114       | 9               | 105, 106, 114 | 0          | 126           | 0          | 1                    | 0                |
| 107       | t_43_r_300000000                    | t_43_r_400000000                    | 116       | 3               | 108, 116, 115 | 0          | 35            | 0          | 1                    | 0                |
| 2         | t_43_r_400000000                    |                                     | 73        | 8               | 73, 79, 162   | 0          | 0             | 0          | 1                    | 0                |
| 146       | t_43_i_1_                           | t_43_i_1_03800000000773594003000000 | 149       | 9               | 148, 149, 166 | 0          | 0             | 0          | 1                    | 0                |
| 150       | t_43_i_1_03800000000773594003000000 | t_43_i_1_03800000000ee6b28003000000 | 159       | 8               | 152, 153, 159 | 0          | 35            | 0          | 1                    | 0                |
| 154       | t_43_i_1_03800000000ee6b28003000000 | t_43_i_1_0380000000165a0bc003000000 | 160       | 1               | 155, 160, 161 | 0          | 35            | 0          | 1                    | 0                |
| 128       | t_43_i_1_0380000000165a0bc003000000 | t_43_i_2_                           | 145       | 3               | 131, 145, 144 | 0          | 0             | 0          | 1                    | 0                |
| 91        | t_43_i_2_                           | t_43_r                              | 93        | 8               | 92, 93, 94    | 0          | 35            | 0          | 1                    | 0                |
+-----------+-------------------------------------+-------------------------------------+-----------+-----------------+---------------+------------+---------------+------------+----------------------+------------------+

Check List

Tests

Unit test

Code changes

Side effects

Related changes

Release note

Add tidb_decode_key build-in function to decode the key of TiDB used.

crazycs520 · 2019-09-16T05:53:45Z

@winkyao PTAL

codecov · 2019-09-16T05:58:49Z

Codecov Report

Merging #12193 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #12193   +/-   ##
===========================================
  Coverage   81.1245%   81.1245%           
===========================================
  Files           454        454           
  Lines         99351      99351           
===========================================
  Hits          80598      80598           
  Misses        12950      12950           
  Partials       5803       5803

winkyao

LGTM

bb7133

If tidb_decode_key() failed to decode the input, IMHO it's better to return a warning to the client, for example, given the following query:

>select region_id, tidb_decode_key(start_key) as k from `TIKV_REGION_STATUS` where tidb_decode_key(start_key) like 'tableID=%' limit 3

If the decoding failed for some unexpected reason, the end-user sees nothing result. In such cases, a warning can remind the user that something is wrong because of the failed decodes, rather than the real result does not exist actually.

expression/builtin_info.go

…encode_key

zz-jason · 2019-09-17T13:37:14Z

@crazycs520 Please supplement the release-note section in the PR description. BTW, we also need to update the document about this new builtin function.

zz-jason

Maybe tidb_decode_region_key is preciser, although tidb_decode_key() is simpler,

expression/builtin_info.go

Co-Authored-By: Zhang Jian <[email protected]>

crazycs520 · 2019-09-23T02:37:49Z

@zz-jason , tidb_decode_region_key is too narrow, this function not only use to decode the region key, sometime in the TiDB log/TiKV log maybe record the conflict key, then we can use this function to decode too.

bb7133

LGTM

sre-bot · 2019-09-23T03:53:34Z

Your auto merge job has been accepted, waiting for 12254

sre-bot · 2019-09-23T04:01:17Z

/run-all-tests

expression/builtin.go

expression/builtin_info.go

Co-Authored-By: Zhang Jian <[email protected]>

zz-jason

LGTM

zz-jason · 2019-09-24T01:49:52Z

/run-all-tests

winoros · 2019-12-24T11:36:08Z

expression/builtin_info.go

+	// Try to decode it as a record key.
+	tableID, handle, err := tablecodec.DecodeRecordKey(key)
+	if err == nil {
+		return "tableID=" + strconv.FormatInt(tableID, 10) + ", _tidb_rowid=" + strconv.FormatInt(handle, 10)


Why we don't return json object?

Great idea.

I will change this tomorrow.

crazycs520 added 2 commits September 16, 2019 12:04

ast/functions: add tidb_decode_key() function

011ff71

efine code

df69702

crazycs520 added the type/usability label Sep 16, 2019

Merge branch 'master' into encode_key

d8be767

crazycs520 requested a review from imtbkcat September 16, 2019 06:00

crazycs520 changed the title ~~Encode key~~ ast/functions: add tidb_decode_key() function to decode key Sep 16, 2019

winkyao reviewed Sep 16, 2019

View reviewed changes

Merge branch 'master' into encode_key

cae01dc

bb7133 reviewed Sep 16, 2019

View reviewed changes

expression/builtin_info.go Show resolved Hide resolved

crazycs520 added 2 commits September 17, 2019 15:07

add warning

c6069ff

Merge branch 'encode_key' of https://github.com/crazycs520/tidb into …

bc55f0e

…encode_key

sre-bot mentioned this pull request Sep 17, 2019

ddl: fix drop index failed when index contain auto_increment column and more than 2 index contain auto_increment_column #12230

Merged

zz-jason added the component/expression label Sep 17, 2019

zz-jason changed the title ~~ast/functions: add tidb_decode_key() function to decode key~~ expression: add builtin function tidb_decode_key() Sep 17, 2019

zz-jason reviewed Sep 17, 2019

View reviewed changes

expression/builtin_info.go Outdated Show resolved Hide resolved

expression/builtin_info.go Outdated Show resolved Hide resolved

expression/builtin_info.go Outdated Show resolved Hide resolved

expression/builtin_info.go Outdated Show resolved Hide resolved

crazycs520 and others added 5 commits September 21, 2019 23:00

Update expression/builtin_info.go

9b0fad8

Co-Authored-By: Zhang Jian <[email protected]>

Update expression/builtin_info.go

5b81959

Co-Authored-By: Zhang Jian <[email protected]>

Update expression/builtin_info.go

066b793

Co-Authored-By: Zhang Jian <[email protected]>

address comment

e7fb3bd

Merge branch 'master' into encode_key

e023194

bb7133 reviewed Sep 23, 2019

View reviewed changes

bb7133 previously approved these changes Sep 23, 2019

View reviewed changes

update parser go.mod

ff781ee

crazycs520 dismissed bb7133’s stale review via ff781ee September 23, 2019 03:29

fix ci

c63fede

Merge branch 'master' into encode_key

5d0ce6b

crazycs520 added status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. labels Sep 23, 2019

Merge branch 'master' into encode_key

948a6f2

crazycs520 added 3 commits September 23, 2019 13:27

Merge branch 'master' into encode_key

90b9ea5

Merge branch 'master' into encode_key

ae539ca

Merge branch 'master' into encode_key

932c2e6

zz-jason reviewed Sep 23, 2019

View reviewed changes

expression/builtin.go Show resolved Hide resolved

expression/builtin_info.go Outdated Show resolved Hide resolved

crazycs520 and others added 3 commits September 23, 2019 21:16

Update expression/builtin_info.go

8daf4af

Co-Authored-By: Zhang Jian <[email protected]>

address comment

b9989ad

Merge branch 'master' into encode_key

c588c44

zz-jason approved these changes Sep 24, 2019

View reviewed changes

crazycs520 merged commit 1051949 into pingcap:master Sep 24, 2019

winoros reviewed Dec 24, 2019

View reviewed changes

reafans mentioned this pull request Feb 20, 2020

Support SQL built-in func instead of the tidb-ctl cmd. #14863

Closed

12 tasks

imtbkcat mentioned this pull request Apr 1, 2020

add built-function to decode key #15951

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

expression: add builtin function `tidb_decode_key()` #12193

expression: add builtin function `tidb_decode_key()` #12193

crazycs520 commented Sep 16, 2019 •

edited

Loading

crazycs520 commented Sep 16, 2019

codecov bot commented Sep 16, 2019 •

edited

Loading

winkyao left a comment

bb7133 left a comment

zz-jason commented Sep 17, 2019

zz-jason left a comment

crazycs520 commented Sep 23, 2019

bb7133 left a comment

sre-bot commented Sep 23, 2019

sre-bot commented Sep 23, 2019

zz-jason left a comment

zz-jason commented Sep 24, 2019

winoros Dec 24, 2019

crazycs520 Jan 1, 2020

crazycs520 Jan 1, 2020

expression: add builtin function tidb_decode_key() #12193

expression: add builtin function tidb_decode_key() #12193

Conversation

crazycs520 commented Sep 16, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Other discuss

Check List

crazycs520 commented Sep 16, 2019

codecov bot commented Sep 16, 2019 • edited Loading

Codecov Report

winkyao left a comment

Choose a reason for hiding this comment

bb7133 left a comment

Choose a reason for hiding this comment

zz-jason commented Sep 17, 2019

zz-jason left a comment

Choose a reason for hiding this comment

crazycs520 commented Sep 23, 2019

bb7133 left a comment

Choose a reason for hiding this comment

sre-bot commented Sep 23, 2019

sre-bot commented Sep 23, 2019

zz-jason left a comment

Choose a reason for hiding this comment

zz-jason commented Sep 24, 2019

winoros Dec 24, 2019

Choose a reason for hiding this comment

crazycs520 Jan 1, 2020

Choose a reason for hiding this comment

crazycs520 Jan 1, 2020

Choose a reason for hiding this comment

expression: add builtin function `tidb_decode_key()` #12193

expression: add builtin function `tidb_decode_key()` #12193

crazycs520 commented Sep 16, 2019 •

edited

Loading

codecov bot commented Sep 16, 2019 •

edited

Loading