Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: implement IterReverse for tikvSnapshot and use desc scan to get latest N ddl history jobs. #10152

Merged
merged 31 commits into from
May 31, 2019
Merged

*: implement IterReverse for tikvSnapshot and use desc scan to get latest N ddl history jobs. #10152

merged 31 commits into from
May 31, 2019

Conversation

crazycs520
Copy link
Contributor

@crazycs520 crazycs520 commented Apr 15, 2019 •
edited
Loading

What problem does this PR solve?

Currently admin show ddl jobs get all ddl history jobs, but we only need the latest N ddl history jobs. So we can use desc scan to get latest N ddl history jobs.

What is changed and how it works?

  1. Implement IterReverse for tikvSnapshot.
  2. Use IterReverse to get latest ddl history jobs for admin show ddl jobs

Compare
There are 17083 ddl history jobs.

mysql root@localhost:test> admin show ddl jobs 100000
17083 rows in set
Time: 2.323s

Before spend 0.5s.

mysql root@localhost:test> admin show ddl jobs 10
+--------+---------+------------+--------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE     | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | START_TIME                        | STATE  |
+--------+---------+------------+--------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| 34165  | test23  | t2317      | create table | public       | 15586     | 34164    | 0         | 2019-04-15 19:44:45.005 +0800 CST | synced |
| 34163  | test22  | t2340      | create table | public       | 15456     | 34162    | 0         | 2019-04-15 19:44:44.955 +0800 CST | synced |
| 34161  | test20  | t2421      | create table | public       | 15238     | 34160    | 0         | 2019-04-15 19:44:44.854 +0800 CST | synced |
| 34159  | test21  | t2382      | create table | public       | 15310     | 34158    | 0         | 2019-04-15 19:44:44.754 +0800 CST | synced |
| 34157  | test23  | t2316      | create table | public       | 15586     | 34156    | 0         | 2019-04-15 19:44:44.654 +0800 CST | synced |
| 34155  | test22  | t2339      | create table | public       | 15456     | 34154    | 0         | 2019-04-15 19:44:44.458 +0800 CST | synced |
| 34153  | test20  | t2420      | create table | public       | 15238     | 34152    | 0         | 2019-04-15 19:44:43.854 +0800 CST | synced |
| 34151  | test21  | t2381      | create table | public       | 15310     | 34150    | 0         | 2019-04-15 19:44:43.706 +0800 CST | synced |
| 34149  | test23  | t2315      | create table | public       | 15586     | 34148    | 0         | 2019-04-15 19:44:43.654 +0800 CST | synced |
| 34147  | test22  | t2338      | create table | public       | 15456     | 34146    | 0         | 2019-04-15 19:44:43.506 +0800 CST | synced |
+--------+---------+------------+--------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
10 rows in set
Time: 0.517s

In This PR spend 0.01s.

mysql root@localhost:test> admin show ddl jobs
+--------+---------+------------+--------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE     | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | START_TIME                        | STATE  |
+--------+---------+------------+--------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| 34165  | test23  | t2317      | create table | public       | 15586     | 34164    | 0         | 2019-04-15 19:44:45.005 +0800 CST | synced |
| 34163  | test22  | t2340      | create table | public       | 15456     | 34162    | 0         | 2019-04-15 19:44:44.955 +0800 CST | synced |
| 34161  | test20  | t2421      | create table | public       | 15238     | 34160    | 0         | 2019-04-15 19:44:44.854 +0800 CST | synced |
| 34159  | test21  | t2382      | create table | public       | 15310     | 34158    | 0         | 2019-04-15 19:44:44.754 +0800 CST | synced |
| 34157  | test23  | t2316      | create table | public       | 15586     | 34156    | 0         | 2019-04-15 19:44:44.654 +0800 CST | synced |
| 34155  | test22  | t2339      | create table | public       | 15456     | 34154    | 0         | 2019-04-15 19:44:44.458 +0800 CST | synced |
| 34153  | test20  | t2420      | create table | public       | 15238     | 34152    | 0         | 2019-04-15 19:44:43.854 +0800 CST | synced |
| 34151  | test21  | t2381      | create table | public       | 15310     | 34150    | 0         | 2019-04-15 19:44:43.706 +0800 CST | synced |
| 34149  | test23  | t2315      | create table | public       | 15586     | 34148    | 0         | 2019-04-15 19:44:43.654 +0800 CST | synced |
| 34147  | test22  | t2338      | create table | public       | 15456     | 34146    | 0         | 2019-04-15 19:44:43.506 +0800 CST | synced |
+--------+---------+------------+--------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
10 rows in set
Time: 0.010s

Check List

Tests

  • Unit test

Code changes

Side effects

Related changes

@crazycs520 crazycs520 requested a review from MyonKeminta April 15, 2019 09:40
@crazycs520
Copy link
Contributor Author

@MyonKeminta @tiancaiamao @winkyao PTAL

@crazycs520 crazycs520 added component/DDL-need-LGT3 type/enhancement The issue or PR belongs to an enhancement. labels Apr 15, 2019
winkyao
winkyao reviewed Apr 20, 2019
View reviewed changes
meta/meta.go Outdated Show resolved Hide resolved
store/mockstore/mocktikv/rpc.go Show resolved Hide resolved
winkyao
winkyao reviewed Apr 20, 2019
View reviewed changes
store/tikv/scan.go Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented May 24, 2019 •
edited
Loading

Codecov Report

Merging #10152 into master will increase coverage by 0.0224%.
The diff coverage is 86.4077%.

@@               Coverage Diff                @@
##             master     #10152        +/-   ##
================================================
+ Coverage   78.2253%   78.2477%   +0.0224%     
================================================
  Files           413        413                
  Lines         87542      87623        +81     
================================================
+ Hits          68480      68563        +83     
+ Misses        13933      13922        -11     
- Partials       5129       5138         +9

@crazycs520
Copy link
Contributor Author

@winkyao PTAL again.

zimulala
zimulala reviewed May 29, 2019
View reviewed changes
store/mockstore/mocktikv/rpc.go Outdated Show resolved Hide resolved
store/mockstore/mocktikv/cluster.go Outdated Show resolved Hide resolved
zimulala
zimulala reviewed May 30, 2019
View reviewed changes
Copy link
Contributor

@zimulala zimulala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zimulala zimulala added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 30, 2019
@crazycs520
Copy link
Contributor Author

/run-all-tests

@crazycs520
Copy link
Contributor Author

/rebuild

@crazycs520
Copy link
Contributor Author

/run-all-tests

winkyao
winkyao approved these changes May 31, 2019
View reviewed changes
Copy link
Contributor

@winkyao winkyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@winkyao winkyao merged commit 8599fee into pingcap:master May 31, 2019
crazycs520 added a commit to crazycs520/tidb that referenced this pull request Jun 4, 2019
@crazycs520
*: implement IterReverse for tikvSnapshot and use desc scan to get la…
a3aeaaa 
…test N ddl history jobs. (pingcap#10152)
@crazycs520 crazycs520 mentioned this pull request Jun 4, 2019
Merged
zz-jason pushed a commit that referenced this pull request Jun 4, 2019
@crazycs520 @zz-jason
*: implement IterReverse for tikvSnapshot and use desc scan to get la…
5390dfa 
…test N ddl history jobs. (#10152) (#10687)
This was referenced Jul 29, 2019
Merged
Merged
Closed
crazycs520 added a commit to crazycs520/tidb that referenced this pull request Aug 20, 2019
@crazycs520
*: implement IterReverse for tikvSnapshot and use desc scan to get la…
990ea87 
…test N ddl history jobs. (pingcap#10152)
@crazycs520 crazycs520 mentioned this pull request Aug 20, 2019
Merged
winkyao pushed a commit that referenced this pull request Aug 21, 2019
@crazycs520 @winkyao
*: use desc scan to get latest N ddl history jobs. (#10152) (#11789)
98714ab
@crazycs520 crazycs520 mentioned this pull request Aug 26, 2019
Closed
This was referenced Aug 26, 2019
Closed
Closed
@you06 you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zimulala zimulala zimulala left review comments

@winkyao winkyao winkyao approved these changes

@MyonKeminta MyonKeminta Awaiting requested review from MyonKeminta

@tiancaiamao tiancaiamao Awaiting requested review from tiancaiamao

Assignees
No one assigned
Labels
sig/sql-infra SIG: SQL Infra status/LGT2 Indicates that a PR has LGTM 2. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@crazycs520 @tiancaiamao @zimulala @winkyao @you06