Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scratch installed v6.4.0 tidb cluster, flashback can't execute for "flashback timestamp to too close to present time" #39130

Closed
seiya-annie opened this issue Nov 14, 2022 · 5 comments · Fixed by tikv/tikv#16038
Closed

scratch installed v6.4.0 tidb cluster, flashback can't execute for "flashback timestamp to too close to present time" #39130

seiya-annie opened this issue Nov 14, 2022 · 5 comments · Fixed by tikv/tikv#16038
Assignees
@overvenus
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-6.6 affects-7.0 affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/tikv severity/major type/bug The issue is confirmed as a bug.

Comments

@seiya-annie
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. new install a tidb cluster with v6.4.0 version
  2. init tpcc
  3. run tpcc
  4. during tpcc workload running, execute flashback

2. What did you expect to see? (Required)

flashback successfully

3. What did you see instead (Required)

MySQL [(none)]> flashback cluster to timestamp '2022-11-13 11:55';
ERROR 8055 (HY000): snapshot is older than GC safe point 2022-11-14 11:42:40 +0800 CST
MySQL [(none)]> flashback cluster to timestamp '2022-11-14 11:42';
ERROR 1105 (HY000): cannot set flashback timestamp to too close to present time
MySQL [(none)]>

4. What is your TiDB version? (Required)

[root@localhost ~]# /data1/tidb-deploy/tidb-4000/bin/tidb-server -V
Release Version: v6.4.0
Edition: Community
Git Commit Hash: cf36a9c
Git Branch: heads/refs/tags/v6.4.0
UTC Build Time: 2022-11-13 05:25:30
@seiya-annie seiya-annie added the type/bug The issue is confirmed as a bug. label Nov 14, 2022
@seiya-annie
Copy link
Author

resolved ts doesn't move on in one tikv node

@seiya-annie
Copy link
Author

seiya-annie commented Nov 14, 2022 •
edited by Defined2014
Loading 

[2022/11/14 13:18:17.360 +08:00] [INFO] [kv.go:515] ["Get safe TS"] [ok=true] [storeID=5] [safeTS="2022-11-14 13:18:14.989 +0800 CST"]
[2022/11/14 13:18:17.360 +08:00] [INFO] [kv.go:515] ["Get safe TS"] [ok=true] [storeID=4] [safeTS="2022-11-14 13:18:14.989 +0800 CST"]
[2022/11/14 13:18:17.360 +08:00] [INFO] [kv.go:515] ["Get safe TS"] [ok=true] [storeID=1] [safeTS="2022-11-14 11:35:56.839 +0800 CST"]
[2022/11/14 13:18:17.360 +08:00] [INFO] [kv.go:515] ["Get safe TS"] [ok=true] [storeID=6] [safeTS="2022-11-14 13:18:14.989 +0800 CST"]
[2022/11/14 13:18:17.360 +08:00] [INFO] [kv.go:515] ["Get safe TS"] [ok=true] [storeID=7] [safeTS="2022-11-14 13:18:14.989 +0800 CST"]

Store 1 is stuck at 11:35:56.839 for 2 hours, until the log below

[2022/11/14 13:45:57.578 +08:00] [WARN] [[peer.rs:6041](http://peer.rs:6041/)] ["leader missing longer than max_leader_missing_duration. To check with pd and other peers whether it's still valid"] [expect=2h] [peer_id=562] [region_id=558]
[2022/11/14 13:45:57.578 +08:00] [WARN] [[peer.rs:6041](http://peer.rs:6041/)] ["leader missing longer than max_leader_missing_duration. To check with pd and other peers whether it's still valid"] [expect=2h] [peer_id=547] [region_id=543]

@seiya-annie
Copy link
Author

work normal in other tidb cluster

@ti-chi-bot ti-chi-bot added may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 may-affects-6.1 may-affects-6.2 may-affects-6.3 labels Nov 14, 2022
@seiya-annie seiya-annie removed may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-6.0 may-affects-6.1 may-affects-6.2 may-affects-6.3 labels Nov 14, 2022
@Defined2014 Defined2014 added component/tikv and removed sig/sql-infra SIG: SQL Infra labels Nov 14, 2022
@Defined2014
Copy link
Contributor

Defined2014 commented Nov 14, 2022 •
edited
Loading

Seems a learner region remains in TiKV, then the min_safe_ts will not advance until the learner be cleaned.

Looks like tikv/tikv#11847, the phenomenon is same.

@VelocityLight VelocityLight added the affects-6.5 This bug affects the 6.5.x(LTS) versions. label Dec 2, 2022
@VelocityLight VelocityLight added the affects-7.1 This bug affects the 7.1.x(LTS) versions. label Apr 20, 2023
@ti-chi-bot ti-chi-bot added the affects-7.5 This bug affects the 7.5.x(LTS) versions. label Oct 23, 2023
@overvenus overvenus mentioned this issue Nov 21, 2023
Merged
@overvenus overvenus self-assigned this Nov 21, 2023
ti-chi-bot bot pushed a commit to tikv/tikv that referenced this issue Nov 22, 2023
@overvenus @tonyxuqqi
raftstore: check stale peer on leader missing (#16038)
86d4a49 
close #11847, close #15520, close pingcap/tidb#39130

Stale peers can impede TiKV store resolved ts and impact RTO for essential
functions. Default 2-hour interval for stale peer check is insufficient
for stale reads, flashbacks, and ebs backup.

To mitigate this, we speed up stale read check by allowing TiKV to check for
stale peers every 10 minutes in the event that a leader is missing.

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: tonyxuqqi <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 22, 2023
Closed
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 22, 2023
@overvenus @ti-chi-bot
This is an automated cherry-pick of tikv#16038
7ec5d0a 
close tikv#11847, close tikv#15520, close pingcap/tidb#39130

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 22, 2023
Open
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 22, 2023
@overvenus @ti-chi-bot
This is an automated cherry-pick of tikv#16038
038c590 
close tikv#11847, close tikv#15520, close pingcap/tidb#39130

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 22, 2023
Open
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 22, 2023
@overvenus @ti-chi-bot
This is an automated cherry-pick of tikv#16038
3bd5df2 
close tikv#11847, close tikv#15520, close pingcap/tidb#39130

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 22, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 22, 2023
@overvenus @ti-chi-bot
This is an automated cherry-pick of tikv#16038
5d752ac 
close tikv#11847, close tikv#15520, close pingcap/tidb#39130

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 22, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 22, 2023
@overvenus @ti-chi-bot
This is an automated cherry-pick of tikv#16038
c924536 
close tikv#11847, close tikv#15520, close pingcap/tidb#39130

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 22, 2023
Merged
@overvenus
Copy link
Member

Mitigated by tikv/tikv#16038

overvenus added a commit to ti-chi-bot/tikv that referenced this issue Nov 27, 2023
@overvenus @tonyxuqqi
raftstore: check stale peer on leader missing (tikv#16038)
6439dd2 
Stale peers can impede TiKV store resolved ts and impact RTO for essential
functions. Default 2-hour interval for stale peer check is insufficient
for stale reads, flashbacks, and ebs backup.

To mitigate this, we speed up stale read check by allowing TiKV to check for
stale peers every 10 minutes in the event that a leader is missing.

close tikv#11847, close tikv#15520, close pingcap/tidb#39130

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: tonyxuqqi <[email protected]>
ti-chi-bot bot pushed a commit to tikv/tikv that referenced this issue Nov 28, 2023
@ti-chi-bot @overvenus @tonyxuqqi
raftstore: check stale peer on leader missing (#16038) (#16044)
cb17c3f 
close #11847, close #15520, close pingcap/tidb#39130

Stale peers can impede TiKV store resolved ts and impact RTO for essential
functions. Default 2-hour interval for stale peer check is insufficient
for stale reads, flashbacks, and ebs backup.

To mitigate this, we speed up stale read check by allowing TiKV to check for
stale peers every 10 minutes in the event that a leader is missing.

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: Neil Shen <[email protected]>
Co-authored-by: tonyxuqqi <[email protected]>
overvenus added a commit to ti-chi-bot/tikv that referenced this issue Dec 8, 2023
@overvenus @tonyxuqqi
raftstore: check stale peer on leader missing (tikv#16038)
544c2a2 
close tikv#11847, close tikv#15520, close pingcap/tidb#39130

Stale peers can impede TiKV store resolved ts and impact RTO for essential
functions. Default 2-hour interval for stale peer check is insufficient
for stale reads, flashbacks, and ebs backup.

To mitigate this, we speed up stale read check by allowing TiKV to check for
stale peers every 10 minutes in the event that a leader is missing.

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: tonyxuqqi <[email protected]>
ti-chi-bot bot pushed a commit to tikv/tikv that referenced this issue Dec 8, 2023
@ti-chi-bot @overvenus @tonyxuqqi
raftstore: check stale peer on leader missing (#16038) (#16045)
17c3547 
close #11847, close #15520, close pingcap/tidb#39130

Stale peers can impede TiKV store resolved ts and impact RTO for essential
functions. Default 2-hour interval for stale peer check is insufficient
for stale reads, flashbacks, and ebs backup.

To mitigate this, we speed up stale read check by allowing TiKV to check for
stale peers every 10 minutes in the event that a leader is missing.

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: Neil Shen <[email protected]>
Co-authored-by: tonyxuqqi <[email protected]>
ti-chi-bot bot added a commit to tikv/tikv that referenced this issue Jan 31, 2024
@ti-chi-bot @overvenus @ti-chi-bot
raftstore: check stale peer on leader missing (#16038) (#16046)
aee4d99 
close #11847, close #15520, close pingcap/tidb#39130

Stale peers can impede TiKV store resolved ts and impact RTO for essential
functions. Default 2-hour interval for stale peer check is insufficient
for stale reads, flashbacks, and ebs backup.

To mitigate this, we speed up stale read check by allowing TiKV to check for
stale peers every 10 minutes in the event that a leader is missing.

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: Neil Shen <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
mittalrishabh pushed a commit to mittalrishabh/tikv that referenced this issue May 8, 2024
@ti-chi-bot @overvenus @tonyxuqqi
raftstore: check stale peer on leader missing (tikv#16038) (tikv#16044)…
8ca31cc 
… (tikv#12)

close tikv#11847, close tikv#15520, close pingcap/tidb#39130

Stale peers can impede TiKV store resolved ts and impact RTO for essential
functions. Default 2-hour interval for stale peer check is insufficient
for stale reads, flashbacks, and ebs backup.

To mitigate this, we speed up stale read check by allowing TiKV to check for
stale peers every 10 minutes in the event that a leader is missing.

Co-authored-by: Ti Chi Robot <[email protected]>
Co-authored-by: Neil Shen <[email protected]>
Co-authored-by: tonyxuqqi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@overvenus overvenus

Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-6.6 affects-7.0 affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/tikv severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

raftstore: check stale peer on leader missing overvenus/tikv
5 participants
@overvenus @VelocityLight @Defined2014 @seiya-annie @ti-chi-bot