Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stale ETCD client endpoints remain in healthyChecker #7226

Closed
iosmanthus opened this issue Oct 18, 2023 · 0 comments · Fixed by #7227
Closed

Stale ETCD client endpoints remain in healthyChecker #7226

iosmanthus opened this issue Oct 18, 2023 · 0 comments · Fixed by #7227
Labels
affects-7.5 This bug affects the 7.5.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.

Comments

@iosmanthus
Copy link
Member

Bug Report

What did you do?

I tried to upgrade a 3-node PD cluster's traffic from plaintext HTTP to HTTPS, at some stage, there will be 6 client endpoints, like:

http://10.0.0.1:2379
https://10.0.0.1:2379
http://10.0.0.2:2379
https://10.0.0.2:2379
http://10.0.0.3:2379
https://10.0.0.3:2379

When I bring those HTTP endpoints offline, the health checker still blames the HTTP endpoints are not being healthy, which might stop checking those endpoints.

What did you expect to see?

Check the health status for those endpoints reported from the ETCD cluster.

What did you see instead?

Stale endpoints are checked forever.

What version of PD are you using (pd-server -V)?

master
@iosmanthus iosmanthus added the type/bug The issue is confirmed as a bug. label Oct 18, 2023
@iosmanthus iosmanthus mentioned this issue Oct 18, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Nov 16, 2023
@iosmanthus @lhy1024
etcdutil: remove stale client endpoints for healthyChecker (#7227)
a6800a9 
close #7226

remove stale client endpoints for `healthyChecker`

Signed-off-by: iosmanthus <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 16, 2023
Closed
@niubell niubell mentioned this issue Jan 12, 2024
Merged
17 tasks
@HuSharp HuSharp mentioned this issue Feb 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-7.5 This bug affects the 7.5.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
Status: Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

etcdutil: remove stale client endpoints for healthyChecker iosmanthus/pd
3 participants
@jebter @iosmanthus @lhy1024