Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

*: fix executing shard DDL with mismatched request #336

Merged
merged 11 commits into from
Oct 28, 2019
Merged

*: fix executing shard DDL with mismatched request #336

merged 11 commits into from
Oct 28, 2019

Conversation

csuzhangxc
Copy link
Member

What problem does this PR solve?

  1. DM-worker does not verify the ExecDDLRequest from the DM-master
  2. DM-master may send duplicate ExecDDLRequest to DM-worker (in order to support re-fetching DDL info from DM-worker after DM-master restarted)

example:

  1. execute ADD INDEX idx1
  2. duplicate ADD INDEX idx1 request sent
  3. ADD COLUMN c1 synced in DM-worker
  4. duplicate request used, ADD COLUMN c1 executed, but ADD COLUMN c1 in other DM-workers not synced yet.

ref: https://internal.pingcap.net/jira/browse/ONCALL-745

What is changed and how it works?

  1. do not re-send DDL execution request in most scenarios
    1. ignore the invalid connection for ADD INDEX
    2. decrease DDL DB execution timeout, increase DDL gRPC execution request
  2. ignore duplicate DDL execution request
    • verify DDL if possible

Check List

Tests

  • Unit test
  • Integration test

Related changes

  • Need to cherry-pick to the release branch
  • Need to be included in the release note

@csuzhangxc csuzhangxc added priority/important Major change, requires approval from ≥2 primary reviewers status/WIP This PR is still work in progress type/bug-fix Bug fix priority/release-blocker This PR blocks a release. Please review it ASAP. needs-cherry-pick-release-1.0 This PR should be cherry-picked to release-1.0. Remove this label after cherry-picked to release-1.0 needs-update-release-note This PR should be added into release notes. Remove this label once the release notes are updated labels Oct 26, 2019
@csuzhangxc
Copy link
Member Author

/run-all-tests

@csuzhangxc
Copy link
Member Author

/run-compatibility-test case=all pre_commit=e63c6cdebea0edcf2ef8c91d84cff4aaa5fc2df7

@codecov
Copy link

codecov bot commented Oct 27, 2019 •
edited
Loading

Codecov Report

Merging #336 into master will increase coverage by 2.297%.
The diff coverage is 77.0833%.

@@               Coverage Diff               @@
##             master       #336       +/-   ##
===============================================
+ Coverage   57.7477%   60.0447%   +2.297%     
===============================================
  Files           158        135       -23     
  Lines         16108      15197      -911     
===============================================
- Hits           9302       9125      -177     
+ Misses         5914       5173      -741     
- Partials        892        899        +7

@csuzhangxc csuzhangxc added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/WIP This PR is still work in progress labels Oct 27, 2019
@csuzhangxc
Copy link
Member Author

@amyangfei @WangXiangUSTC PTAL

@csuzhangxc
Copy link
Member Author

/run-compatibility-test case=all pre_commit=e63c6cdebea0edcf2ef8c91d84cff4aaa5fc2df7

amyangfei
amyangfei reviewed Oct 28, 2019
View reviewed changes
syncer/error.go Outdated Show resolved Hide resolved
syncer/error.go Show resolved Hide resolved
amyangfei
amyangfei reviewed Oct 28, 2019
View reviewed changes
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amyangfei amyangfei added status/LGT1 One reviewer already commented LGTM and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Oct 28, 2019
WangXiangUSTC
WangXiangUSTC reviewed Oct 28, 2019
View reviewed changes
syncer/error.go Outdated Show resolved Hide resolved
@csuzhangxc
Copy link
Member Author

/run-all-tests

1 similar comment
@csuzhangxc
Copy link
Member Author

/run-all-tests

WangXiangUSTC
WangXiangUSTC approved these changes Oct 28, 2019
View reviewed changes
Copy link
Contributor

@WangXiangUSTC WangXiangUSTC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@csuzhangxc csuzhangxc merged commit 8b2f08e into pingcap:master Oct 28, 2019
@csuzhangxc csuzhangxc deleted the fix-shard-cache branch October 28, 2019 11:03
@sre-bot sre-bot mentioned this pull request Oct 28, 2019
Merged
@sre-bot
Copy link

sre-bot commented Oct 28, 2019

cherry pick to release-1.0 in PR #338

@sre-bot sre-bot added already-cherry-pick-1.0 The related PR is already cherry-picked to release-1.0. Add this label once the PR is cherry-picked and removed needs-cherry-pick-release-1.0 This PR should be cherry-picked to release-1.0. Remove this label after cherry-picked to release-1.0 labels Oct 28, 2019
@WangXiangUSTC WangXiangUSTC added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Oct 28, 2019
csuzhangxc pushed a commit that referenced this pull request Oct 28, 2019
@sre-bot @csuzhangxc
*: fix executing shard DDL with mismatched request (#336) (#338)
2ce7693
@csuzhangxc csuzhangxc added already-update-release-note The release note is updated. Add this label once the release note is updated and removed needs-update-release-note This PR should be added into release notes. Remove this label once the release notes are updated labels Nov 1, 2019
lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Apr 6, 2020
@csuzhangxc
*: fix executing shard DDL with mismatched request (pingcap#336)
60f2c7c
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@amyangfei amyangfei amyangfei left review comments

@WangXiangUSTC WangXiangUSTC WangXiangUSTC approved these changes

Assignees
No one assigned
Labels
already-cherry-pick-1.0 The related PR is already cherry-picked to release-1.0. Add this label once the PR is cherry-picked already-update-release-note The release note is updated. Add this label once the release note is updated priority/important Major change, requires approval from ≥2 primary reviewers priority/release-blocker This PR blocks a release. Please review it ASAP. status/LGT2 Two reviewers already commented LGTM, ready for merge type/bug-fix Bug fix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@csuzhangxc @sre-bot @amyangfei @WangXiangUSTC