kv: "new range lease" log entry is flooding logs #119451
Assignees
Labels
A-kv-replication
Relating to Raft, consensus, and coordination.
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
O-testcluster
Issues found or occurred on a test cluster, i.e. a long-running internal cluster
P-1
Issues/test failures with a fix SLA of 1 month
Comments
ajstorm
added
C-bug
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
cc @cockroachdb/replication |
|
I believe this was added to diagnose otherwise hard-to-track-down decommission stalls due to draining nodes picking up spurious leases. @nvanbenschoten added this back in #87885, so he may have additional context. I suppose we could add a |
|
The logs all look like: Notice that they're expiration-based leases extensions, not acquisitions. We should probably only be logging this loudly when a lease is changing hands. I'll make that change. |
craig bot
pushed a commit
that referenced
this issue
Feb 21, 2024
119474: kv: don't log on lease extension while draining r=nvanbenschoten a=nvanbenschoten Fixes #119451. This commit updates the logging in `Replica.leasePostApplyLocked` to not log on expiration-based lease extensions while the new leaseholder's store is draining. We now only log when a draining store receives a new lease. Release note: None Co-authored-by: Nathan VanBenschoten <[email protected]>
blathers-crl bot
pushed a commit
that referenced
this issue
Feb 21, 2024
Fixes #119451. This commit updates the logging in `Replica.leasePostApplyLocked` to not log on expiration-based lease extensions while the new leaseholder's store is draining. We now only log when a draining store receives a new lease. Release note: None
blathers-crl bot
pushed a commit
that referenced
this issue
Feb 21, 2024
Fixes #119451. This commit updates the logging in `Replica.leasePostApplyLocked` to not log on expiration-based lease extensions while the new leaseholder's store is draining. We now only log when a draining store receives a new lease. Release note: None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
On the DRT cluster we're seeing several log files filled up very quickly with this log entry (a drain is in progress at the time):
https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/replica_proposal.go#L377-L386
In the 15 minute period below, we're filling 130 MBs of log files with this message. This is problematic because it causes the log files to wrap faster, thus losing important information (see #119384).
Marking it as a P-1 for now, as it's impeding our ability to diagnose problems on the DRT cluster.
Jira issue: CRDB-36203
The text was updated successfully, but these errors were encountered: