reload region cache when store is resolved from invalid status #843
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: you06 <[email protected]>
you06
changed the title
Signed-off-by: you06 <[email protected]>
Signed-off-by: you06 <[email protected]>
internal/locate/region_request.go
Outdated
| @@ -573,18 +573,22 @@ func (state *accessFollower) next(bo *retry.Backoffer, selector *replicaSelector | |||
| if state.option.preferLeader { | |||
| state.lastIdx = state.leaderIdx | |||
| } | |||
| offset := rand.Intn(replicaSize) | |||
There was a problem hiding this comment.
Under PreferLeader mode, offset should be set to 0 to make the selection starting from leaderIdx.
There was a problem hiding this comment.
Yeah, I changed the code and not to use offset in the first selection.
Signed-off-by: you06 <[email protected]>
Signed-off-by: you06 <[email protected]>
|
/cc @crazycs520 |
| // as candidates to serve the Read request. Meanwhile, we should make the choice of next() meet Uniform Distribution. | ||
| for cnt := 0; cnt < replicaSize && !state.isCandidate(idx, selector.replicas[idx]); cnt++ { | ||
| idx = AccessIndex((int(idx) + rand.Intn(replicaSize)) % replicaSize) | ||
| var idx AccessIndex |
There was a problem hiding this comment.
Still recommend to revert to the original one, current implementation is bit more complex and hard to read.
Signed-off-by: you06 <[email protected]>
cfzjywxk
reviewed
Jun 19, 2023
internal/locate/region_cache.go
Outdated
| // async reload triggered by other thread. | ||
| return | ||
| } | ||
| go func() { |
There was a problem hiding this comment.
It can be risky to spawn a reload task for each region, especially if many regions become invalid due to the related store being marked as such. To mitigate this risk, we should implement a refreshing strategy in the region cache-related workers and consider incorporating the store's "health checker."
@crazycs520 is now considering refactoring the region cache component, he could give some advice about it.
There was a problem hiding this comment.
For the v6.5.x release version, a quick fix may still be needed.
| @@ -573,18 +573,36 @@ func (state *accessFollower) next(bo *retry.Backoffer, selector *replicaSelector | |||
| if state.option.preferLeader { | |||
| state.lastIdx = state.leaderIdx | |||
| } | |||
| var offset int | |||
There was a problem hiding this comment.
Peer selection can become quite complex when the accessFollower is used for all follower read and stale read processing, with various configurations. To prevent any potential issues, it may be beneficial to separate them into different categories.
ReplicaSelector itself is quite complex too..
There was a problem hiding this comment.
After discussing with @you06, we could do some refactoring on the master branch like
match access_type {
leade_only =>
follower_read_random =>
follower_read_replica_only =>
follower_read_closet =>
stale_read_random =>
stale_read_closest =>
stale_read_retry =>
}
while keeping the change on release-6.5 as simple as possible to reduce the risk.
There was a problem hiding this comment.
Maybe the refactor can be in another PR?
There was a problem hiding this comment.
Ok let's merge this first
Signed-off-by: you06 <[email protected]>
Signed-off-by: you06 <[email protected]>
…master-stale-patch
cfzjywxk
pushed a commit
that referenced
this pull request
Jun 28, 2023
#846) * reload region cache when store is resolved from invalid Signed-off-by: you06 <[email protected]> * reload region once Signed-off-by: you06 <[email protected]> * build in go1.18 Signed-off-by: you06 <[email protected]> * build in go1.18 Signed-off-by: you06 <[email protected]> * handle region reload in resolve goroutine Signed-off-by: you06 <[email protected]> * retest Signed-off-by: you06 <[email protected]> * fix data race (#736) Signed-off-by: Smityz <[email protected]> Co-authored-by: disksing <[email protected]> Signed-off-by: you06 <[email protected]> * build with go 1.18 Signed-off-by: you06 <[email protected]> * fix integration test (#673) Signed-off-by: disksing <[email protected]> Signed-off-by: you06 <[email protected]> * Update internal/locate/region_cache.go Co-authored-by: crazycs <[email protected]> Signed-off-by: you06 <[email protected]> * address comment Signed-off-by: you06 <[email protected]> * address comment Signed-off-by: you06 <[email protected]> --------- Signed-off-by: you06 <[email protected]> Signed-off-by: Smityz <[email protected]> Signed-off-by: disksing <[email protected]> Co-authored-by: Smilencer <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: crazycs <[email protected]>
cfzjywxk
reviewed
Jun 29, 2023
Signed-off-by: you06 <[email protected]>
cfzjywxk
approved these changes
Jun 29, 2023
crazycs520
approved these changes
Jul 7, 2023
iosmanthus
added a commit
that referenced
this pull request
Aug 11, 2023
* client-go: add some key range info to error when PD returned no region (#862) Signed-off-by: Chao Wang <[email protected]> * *: refine non-global stale-read request retry logic (#863) Signed-off-by: crazycs520 <[email protected]> * Fix the issue that primary pessimistic lock may be left not cleared after GC (#866) * Fix the issue that primary pessimistic lock may be left not cleared after GC Signed-off-by: MyonKeminta <[email protected]> * Fix mysteriously shown up thing that makes compilation failed Signed-off-by: MyonKeminta <[email protected]> * Fix test effectiveness (forgot to set txn2 to pessimistic txn); add more strict checks Signed-off-by: MyonKeminta <[email protected]> * Address comments Signed-off-by: MyonKeminta <[email protected]> --------- Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> * add explicit request source type to label the external request like lightning/br (#868) Signed-off-by: nolouch <[email protected]> * use '%d' instead of '%q' for some int values in error message (#875) Signed-off-by: Chao Wang <[email protected]> * format key in error message in method `scanRegions` (#876) Signed-off-by: Chao Wang <[email protected]> * make cop request timeout a config paramter (#865) * update Signed-off-by: Spade A <[email protected]> * update Signed-off-by: Spade A <[email protected]> * update Signed-off-by: Spade A <[email protected]> * update Signed-off-by: Spade A <[email protected]> --------- Signed-off-by: Spade A <[email protected]> * region_cache: support check pending tiflash peer (#821) Signed-off-by: guo-shaoge <[email protected]> Co-authored-by: disksing <[email protected]> * *: add `SnapshotIterReverse` and make `iterReverse` supports `lowerBound` (#883) Signed-off-by: Jason Mo <[email protected]> * *: fix stale read ops metric (#878) (#889) Signed-off-by: crazycs520 <[email protected]> Co-authored-by: disksing <[email protected]> * add gc options (#828) Signed-off-by: weedge <[email protected]> Co-authored-by: disksing <[email protected]> * reload region cache when store is resolved from invalid status (#843) Signed-off-by: you06 <[email protected]> Co-authored-by: disksing <[email protected]> * ci: update setup-go action (#904) Signed-off-by: disksing <[email protected]> * fix unexpected slow query during GC running after stop 1 tikv-server (#899) (#909) * fix unexpected slow query during GC running after stop 1 tikv-server Signed-off-by: crazycs520 <[email protected]> * fix test Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * resource_manager: ignore ru metrics for background request (#872) Signed-off-by: husharp <[email protected]> Co-authored-by: disksing <[email protected]> * add more log for diagnose (#915) * add more log for diagnose Signed-off-by: crazycs520 <[email protected]> * fix Signed-off-by: crazycs520 <[email protected]> * add more log for diagnose Signed-off-by: crazycs520 <[email protected]> * add more log Signed-off-by: crazycs520 <[email protected]> * address comment Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * use context logger as much as possible (#908) * use context logger as much as possible Signed-off-by: crazycs520 <[email protected]> * refine Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * Resume max retry time check for stale read retry with leader option(#903) (#911) * Resume max retry time check for stale read retry with leader option Signed-off-by: cfzjywxk <[email protected]> * add cancel Signed-off-by: cfzjywxk <[email protected]> --------- Signed-off-by: cfzjywxk <[email protected]> * request_source: remove default label (#890) * request_source: remove default label Signed-off-by: nolouch <[email protected]> * add a function to set request source task type (#925) * add a function to set request source task type Signed-off-by: glorv <[email protected]> * ci: update go version (#936) * ci: update go version Signed-off-by: crazycs520 <[email protected]> * fix test Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * use tidb_kv_read_timeout as first kv request timeout (#919) * support tidb_kv_read_timeout as first round kv request timeout Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * fix ci Signed-off-by: crazycs520 <[email protected]> * update comment Signed-off-by: crazycs520 <[email protected]> * refine test Signed-off-by: crazycs520 <[email protected]> --------- Signed-off-by: crazycs520 <[email protected]> * [pick] resource_control: bypass some internal urgent request (#938) * resource_control: bypass some internal urgent request (#884) Signed-off-by: nolouch <[email protected]> * resourcecontrol: fix nil pointer (#900) Signed-off-by: nolouch <[email protected]> --------- Signed-off-by: nolouch <[email protected]> --------- Signed-off-by: Chao Wang <[email protected]> Signed-off-by: crazycs520 <[email protected]> Signed-off-by: MyonKeminta <[email protected]> Signed-off-by: nolouch <[email protected]> Signed-off-by: Spade A <[email protected]> Signed-off-by: guo-shaoge <[email protected]> Signed-off-by: Jason Mo <[email protected]> Signed-off-by: weedge <[email protected]> Signed-off-by: you06 <[email protected]> Signed-off-by: disksing <[email protected]> Signed-off-by: husharp <[email protected]> Signed-off-by: cfzjywxk <[email protected]> Signed-off-by: glorv <[email protected]> Signed-off-by: iosmanthus <[email protected]> Co-authored-by: 王超 <[email protected]> Co-authored-by: crazycs <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: ShuNing <[email protected]> Co-authored-by: Spade A <[email protected]> Co-authored-by: guo-shaoge <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Hangjie Mo <[email protected]> Co-authored-by: weedge <[email protected]> Co-authored-by: you06 <[email protected]> Co-authored-by: Hu# <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: glorv <[email protected]>
you06
added a commit
to you06/client-go
that referenced
this pull request
Aug 11, 2023
) Signed-off-by: you06 <[email protected]> Co-authored-by: disksing <[email protected]>
you06
added a commit
to you06/client-go
that referenced
this pull request
Aug 15, 2023
) Signed-off-by: you06 <[email protected]> Co-authored-by: disksing <[email protected]>
you06
added a commit
to you06/client-go
that referenced
this pull request
Aug 15, 2023
) Signed-off-by: you06 <[email protected]> Co-authored-by: disksing <[email protected]>
cfzjywxk
added a commit
that referenced
this pull request
Sep 26, 2023
* reload region cache when store is resolved from invalid status (#843) Signed-off-by: you06 <[email protected]> Co-authored-by: disksing <[email protected]> * fallback to follower when leader is busy (#916) (#923) * fallback to follower when leader is busy Signed-off-by: you06 <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: cfzjywxk <[email protected]> * Resume max retry time check for stale read retry with leader option(#903) (#911) * Resume max retry time check for stale read retry with leader option Signed-off-by: cfzjywxk <[email protected]> * add cancel Signed-off-by: cfzjywxk <[email protected]> --------- Signed-off-by: cfzjywxk <[email protected]> * add region cache state test & fix some issues of replica selector (#910) Signed-off-by: you06 <[email protected]> remove duplicate code Signed-off-by: you06 <[email protected]> * enable workflow for tidb-7.1 Signed-off-by: you06 <[email protected]> * update Signed-off-by: you06 <[email protected]> update Signed-off-by: you06 <[email protected]> fix test Signed-off-by: you06 <[email protected]> fix test Signed-off-by: you06 <[email protected]> * lint Signed-off-by: you06 <[email protected]> * lint Signed-off-by: you06 <[email protected]> * fix flaky test Signed-off-by: you06 <[email protected]> --------- Signed-off-by: you06 <[email protected]> Signed-off-by: cfzjywxk <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: cfzjywxk <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
close #841
Before this PR:
With this PR:
