kv/client: use region based rate limiter and decouple it from event handler (#3118)

[ti-chi-bot]

This is an automated cherry-pick of #3118

What problem does this PR solve?

To fix part of #3110

If blocking happens in region worker because of rate limit, the receiving buffer of gRPC connection could be full and cause congestion in a gRPC channel, which would lead to back pressure in TiKV and slow incremental scan.

What is changed and how it works?

• Use a standalone rate limiter for each region, put the rate limiter in kv client error handler, instead of event handler. With this optimization, we can reduce blocking time in core event handling path to accelerate consuming data from gRPC buffer.
• Use channel mechanism when waiting all events are processed instead of polling mechanism to decrease latency.
• Reset some metrics counter after region worker exits

After this PR

Before this PR

Check List

Tests

• Unit test
• Integration test

Existing unit tests have covered the functional of change, the performance benchmark is launched manuall.

Release note

Optimize the rate limit control when TiKV reloads and fix the congestion in gPRC, which may cause slow initialization phase.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[amyangfei]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 28b1779

[amyangfei]

/run-integration-tests

[amyangfei]

/run-integration-tests

[amyangfei]

/run-integration-tests

[amyangfei]

/run-integration-tests

[codecov-commenter]

Codecov Report

Merging #3131 (28b1779) into release-4.0 (753073a) will increase coverage by 3.5641%.
The diff coverage is 67.2092%.

@@                 Coverage Diff                 @@
##           release-4.0      #3131        +/-   ##
===================================================
+ Coverage      53.7661%   57.3302%   +3.5641%     
===================================================
  Files              153        164        +11     
  Lines            15958      19515      +3557     
===================================================
+ Hits              8580      11188      +2608     
- Misses            6475       7265       +790     
- Partials           903       1062       +159