Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv: remove broken attempt to reject lease acquisitions on draining nodes #87885

Merged

Conversation

nvanbenschoten
Copy link
Member

Related to #83261.

This commit removes "protection" that avoided lease acquisitions on draining nodes. This protection had already been effectively disabled by acc1ad1, which allowed Raft leaders to bypass the check. As the comment here (added in 5ffaa9e) explained, Raft followers are already unable to acquire the lease. If leaders bypass the check and follower (and candidates) don't need it, the check is useless, so we remove it.

The commit also removes TestReplicaDrainLease, which was supposed to test this behavior. We remove the test not because it started failing after the change, but because it did not. It must not have been testing anything real anymore after acc1ad1.

Release justification: low risk change related to release blocker.

Release note: None

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@aayushshah15
Copy link
Contributor

pkg/kv/kvserver/replica_range_lease.go line 824 at r1 (raw file):

			)
		}
		log.Info(ctx, "trying to take the lease while we're draining since we're the raft leader")

I thought that this log line was useful in cases where draining nodes spuriously re-acquire leases. Do you think we can still keep this log line around? Do you think there's a better way to have this observability?

@nvanbenschoten nvanbenschoten force-pushed the nvanbenschoten/leaseReqDraining2 branch from ef2d7e9 to fafb80e Compare September 20, 2022 16:49
Copy link
Member Author

@nvanbenschoten nvanbenschoten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @aayushshah15)


pkg/kv/kvserver/replica_range_lease.go line 824 at r1 (raw file):

Previously, aayushshah15 (Aayush Shah) wrote…

I thought that this log line was useful in cases where draining nodes spuriously re-acquire leases. Do you think we can still keep this log line around? Do you think there's a better way to have this observability?

Good point. I unified this with the below-Raft lease acquisition logging.

// 2. the new leaseholder is draining
// 3. verbose logging is enabled
epochLeaseChangingHands := newLease.Type() == roachpb.LeaseEpoch && leaseChangingHands
if epochLeaseChangingHands || r.store.IsDraining() || log.V(1) {
log.VEventf(ctx, 1, "new range lease %s following %s", newLease, prevLease)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a log.Infof instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. This logic was slowly adapted by c3dd472 and c3dd472. I think the change I made strikes the desired balance of verbosity, but PTAL.

Related to cockroachdb#83261.

This commit removes "protection" that avoided lease acquisitions on
draining nodes. This protection had already been effectively disabled by
acc1ad1, which allowed Raft leaders to bypass the check. As the comment
here (added in 5ffaa9e) explained, Raft followers are already unable to
acquire the lease. If leaders bypass the check and follower (and
candidates) don't need it, the check is useless, so we remove it.

The commit also removes `TestReplicaDrainLease`, which was supposed to
test this behavior. We remove the test not because it started failing
after the change, but because it did not. It must not have been testing
anything real anymore after acc1ad1.

Release justification: low risk change related to release blocker.

Release note: None
@nvanbenschoten nvanbenschoten force-pushed the nvanbenschoten/leaseReqDraining2 branch from fafb80e to d89a7e4 Compare September 20, 2022 17:16
@nvanbenschoten
Copy link
Member Author

TFTR!

bors r+

@craig
Copy link
Contributor

craig bot commented Sep 20, 2022

Build failed (retrying...):

@craig
Copy link
Contributor

craig bot commented Sep 21, 2022

Build succeeded:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants