Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/client: use ephemeral goroutine to handle region range request #1720

Merged
merged 8 commits into from
May 19, 2021
Merged

kv/client: use ephemeral goroutine to handle region range request #1720

merged 8 commits into from
May 19, 2021

Conversation

amyangfei
Copy link
Contributor

@amyangfei amyangfei commented Apr 28, 2021 •
edited
Loading

What problem does this PR solve?

Fix part bug found in #1599

  • The region request is processed in a block way

https://github.com/pingcap/ticdc/blob/109257825f7805bfc0f87b68d9ef7e969aa19161/cdc/kv/client.go#L578-L591

  • The divideAndSendEventFeedToRegions function scans range into several regions, and dispatch each region via scheduleRegionRequest. There could be stale regions, and these stale regions will be resent to s.requestRangeCh in L646. If the s.requestRangeCh is full, deadlock happens.

https://github.com/pingcap/ticdc/blob/109257825f7805bfc0f87b68d9ef7e969aa19161/cdc/kv/client.go#L627-L648

https://github.com/pingcap/ticdc/blob/109257825f7805bfc0f87b68d9ef7e969aa19161/cdc/kv/client.go#L616-L623

What is changed and how it works?

In order to consume region range request from requestRangeCh as soon as possible, we create a new goroutine to handle it. The sequence of region range we process is not matter, the region lock keeps the region access sequence. Besides the count or frequency of range request is limitted, we use ephemeral goroutine instead of permanent gourotine.

Check List

Tests

  • Unit test
  • Integration test

Release note

  • Fix a deadlock bug in the kv client that deadlock happens when large stale regions exist.

@amyangfei amyangfei added status/wip type/bugfix This PR fixes a bug. component/kv-client TiKV kv log client component. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. labels Apr 28, 2021
@amyangfei amyangfei added this to the v5.0.2 milestone Apr 28, 2021
@ti-chi-bot ti-chi-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Apr 28, 2021
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-integration-tests

@codecov-commenter
Copy link

codecov-commenter commented Apr 28, 2021 •
edited
Loading

Codecov Report

Merging #1720 (2100639) into master (796c966) will increase coverage by 0.1642%.
The diff coverage is 100.0000%.

@@               Coverage Diff                @@
##             master      #1720        +/-   ##
================================================
+ Coverage   52.8727%   53.0369%   +0.1642%     
================================================
  Files           153        153                
  Lines         15891      15904        +13     
================================================
+ Hits           8402       8435        +33     
+ Misses         6584       6563        -21     
- Partials        905        906         +1

@amyangfei amyangfei force-pushed the concurrent-region-range-process branch from 85812ff to 3129458 Compare April 29, 2021 08:26
@ti-chi-bot ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 29, 2021
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-unit-tests

@amyangfei amyangfei added status/ptal Could you please take a look? and removed status/wip labels Apr 29, 2021
@amyangfei
Copy link
Contributor Author

/run-all-tests

1 similar comment
@amyangfei
Copy link
Contributor Author

/run-all-tests

overvenus
overvenus reviewed May 7, 2021
View reviewed changes
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 7, 2021
@amyangfei
Copy link
Contributor Author

ptal @liuzix

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • leoppro
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 19, 2021
@amyangfei
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: cd5b837

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 19, 2021
@ben1009
Copy link
Contributor

ben1009 commented May 19, 2021

/lgtm, though i am not in the list

@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label May 19, 2021
@amyangfei
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 6d320fe

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 19, 2021
@amyangfei
Copy link
Contributor Author

/run-unit-tests

@amyangfei
Copy link
Contributor Author

/run-integration-tests

3 similar comments
@amyangfei
Copy link
Contributor Author

/run-integration-tests

@amyangfei
Copy link
Contributor Author

/run-integration-tests

@amyangfei
Copy link
Contributor Author

/run-integration-tests

@ti-chi-bot ti-chi-bot merged commit 55c4b85 into pingcap:master May 19, 2021
@ti-chi-bot ti-chi-bot mentioned this pull request May 19, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #1800.

@ti-chi-bot ti-chi-bot mentioned this pull request May 19, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #1801.

@amyangfei amyangfei deleted the concurrent-region-range-process branch May 19, 2021 09:58
ti-chi-bot added a commit that referenced this pull request May 19, 2021
@ti-chi-bot
kv/client: use ephemeral goroutine to handle region range request (#1720
f0f47ae 
) (#1800)
@amyangfei amyangfei mentioned this pull request May 20, 2021
Closed
12 tasks
ti-chi-bot added a commit that referenced this pull request May 27, 2021
@ti-chi-bot
kv/client: use ephemeral goroutine to handle region range request (#1720
a05e8cf 
) (#1801)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@overvenus overvenus overvenus left review comments

@zier-one zier-one zier-one approved these changes

@ben1009 ben1009 Awaiting requested review from ben1009

Assignees
No one assigned
Labels
component/kv-client TiKV kv log client component. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. status/ptal Could you please take a look? type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
v5.0.2
Development

Successfully merging this pull request may close these issues.
6 participants
@amyangfei @codecov-commenter @ti-chi-bot @ben1009 @overvenus @zier-one