Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: metric and logging to pinpoint liveness leaseholder #99472

Closed
2 tasks
tbg opened this issue Mar 24, 2023 · 3 comments · Fixed by #104008
Closed
2 tasks

kvserver: metric and logging to pinpoint liveness leaseholder #99472

tbg opened this issue Mar 24, 2023 · 3 comments · Fixed by #104008
Assignees
@erikgrinaker
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs

Comments

@tbg
Copy link
Member

tbg commented Mar 24, 2023 •
edited by exalate-issue-sync bot
Loading

Is your feature request related to a problem? Please describe.

The leases.expiration metric has been useful as a tool to pinpoint the liveness leaseholder, but it's annoying because there are currently two expiration-based ranges: meta1/2 and the liveness range. Also, we're possibly going to have many more expiration-based ranges. The liveness range is very important: we need to have a perfect lease history for it.

Describe the solution you'd like

  • Add a metric leases.liveness that is 1 if and only if the store has a liveness replica that is active leaseholder.
  • log whenever a liveness lease is applied, especially if it's a non-cooperative acquisition.

x-ref https://github.com/cockroachlabs/support/issues/2182

Jira issue: CRDB-25893

Epic CRDB-27235
@tbg tbg added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs GA-blocker T-kv-replication labels Mar 24, 2023
@blathers-crl
Copy link

blathers-crl bot commented Mar 24, 2023

cc @cockroachdb/replication

@blathers-crl
Copy link

blathers-crl bot commented Mar 24, 2023

Hi @tbg, please add branch-* labels to identify which branch(es) this release-blocker affects.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@tbg tbg added branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 labels Mar 24, 2023
tbg added a commit to tbg/cockroach that referenced this issue Mar 24, 2023
@tbg
[dnm] kvserver: try to repro missing gossip on first liveness lease f…
53f840e 
…ollowing restart

See: cockroachdb#99268 (comment)

This also sneaks in the logging requested in cockroachdb#99472.

Epic: none
Release note: None
@erikgrinaker
Copy link
Contributor

Removing the GA-blocker label, this doesn't rise to that level and can wait for 23.2 or a backport.

@erikgrinaker erikgrinaker removed branch-master Failures and bugs on the master branch. GA-blocker branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 labels Mar 27, 2023
@erikgrinaker erikgrinaker mentioned this issue May 27, 2023
Merged
@craig craig bot closed this as completed in 5ee25de May 30, 2023
This was referenced May 30, 2023
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@erikgrinaker erikgrinaker

Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: improve system lease observability erikgrinaker/cockroach
2 participants
@erikgrinaker @tbg