Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: transfer-leases/drain failed #90022

Closed
cockroach-teamcity opened this issue Oct 15, 2022 · 0 comments · Fixed by #90106
Closed

roachtest: transfer-leases/drain failed #90022

cockroach-teamcity opened this issue Oct 15, 2022 · 0 comments · Fixed by #90106
Assignees
@nvanbenschoten
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 15, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.transfer-leases/drain failed with artifacts on release-22.2 @ cffe9bc440988894abe9a598ea6b2f15e1b7df93:

test artifacts and logs in: /artifacts/transfer-leases/drain/run_1
	quit.go:72,quit.go:323,soon.go:69,retry.go:208,soon.go:75,soon.go:48,quit.go:228,quit.go:95,quit.go:154,context.go:91,quit.go:153,quit.go:95,quit.go:54,quit.go:359,test_runner.go:930: (1) ranges with no lease outside of node 1: []string{"43"}

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-20550
@cockroach-teamcity cockroach-teamcity added branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 15, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Oct 15, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Oct 15, 2022
@nvanbenschoten nvanbenschoten self-assigned this Oct 17, 2022
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Oct 17, 2022
@nvanbenschoten
kv: reacquire proscribed leases on drain, then transfer
52087f5 
Fixes cockroachdb#83372.
Fixes cockroachdb#90022.
Fixes cockroachdb#89963.
Fixes cockroachdb#89962.

This commit instructs stores to reacquire proscribed leases when draining in
order to subsequently transfer them away. This addresses a source of flakiness
in `transfer-lease` roachtests where some lease would not be transferred away
before the drain completed. This could result in range unavailable for up to 9
seconds while other replicas waited out the lease'S expiration. This is because
only the previous leaseholder knows that a proscribed lease is invalid. All
other replicas still consider the lease to be valid.

This failure mode was always present if a lease transfer failed during a drain.
However, it became more likely with 034611b. With that change, we began
rejecting lease transfers that were deemed to be "unsafe" more frequently.
034611b introduced a best-effort, graceful version of this check and an
airtight, ungraceful version of the check. The former performs the check before
revoking the outgoing leaseholder's lease while the latter performs the check
after revoking the outgoing leaseholder's lease. In rare cases, it was possible
to hit the airtight, ungraceful check and cause the lease to be proscribed. See
cockroachdb#83261 (comment)
for more details on how this led to test flakiness in the `transfer-lease`
roachtest suite.

Release notes: None.

Release justification: Avoids GA-blocking roachtest failures.
@nvanbenschoten nvanbenschoten mentioned this issue Oct 17, 2022
Merged
craig bot pushed a commit that referenced this issue Oct 18, 2022
@nvanbenschoten @knz @mgartner
Merge #90007 #90106 #90107 #90165 #90166
91b7b94 
90007: execbuilder: enforce_home_region should only apply to DML r=rytaft a=msirek

Fixes #89875
Fixes #88789

This fixes a problem where the enforce_home_region session flag might
cause non-DML statements to error out, such as SHOW CREATE, if those
statements utilize scans or joins of multiregion tables.

This also fixes issues with proper erroring out of mutation DML like
UPDATE AND DELETE. For example, the following previously did not error:
```
CREATE TABLE messages_rbr (
    account_id INT NOT NULL,
    message_id   UUID DEFAULT gen_random_uuid(),
    message    STRING NOT NULL,
    PRIMARY KEY (account_id),
    INDEX msg_idx(message)
)
LOCALITY REGIONAL BY ROW;

SET enforce_home_region = true;

DELETE FROM messages_rbr WHERE message = 'Hello World!'
ERROR: Query has no home region. Try adding a filter on messages_rbr.crdb_region and/or on key column (messages_rbr.account_id).
SQLSTATE: XCHR2
```

Release note (bug fix): This patch fixes an issue with the
enforce_home_region session setting which may cause SHOW CREATE TABLE or
other non-DML statements to error out if the optimizer plan for the
statement involves accessing a multiregion table.


90106: kv: reacquire proscribed leases on drain, then transfer r=shralex a=nvanbenschoten

Fixes #83372.
Fixes #90022.
Fixes #89963.
Fixes #89962.

This commit instructs stores to reacquire proscribed leases when draining in order to subsequently transfer them away. This addresses a source of flakiness in `transfer-lease` roachtests where some lease would not be transferred away before the drain completed. This could result in range unavailable for up to 9 seconds while other replicas waited out the lease'S expiration. This is because only the previous leaseholder knows that a proscribed lease is invalid. All other replicas still consider the lease to be valid.

This failure mode was always present if a lease transfer failed during a drain. However, it became more likely with 034611b. With that change, we began rejecting lease transfers that were deemed to be "unsafe" more frequently. 034611b introduced a best-effort, graceful version of this check and an airtight, ungraceful version of the check. The former performs the check before revoking the outgoing leaseholder's lease while the latter performs the check after revoking the outgoing leaseholder's lease. In rare cases, it was possible to hit the airtight, ungraceful check and cause the lease to be proscribed. See #83261 (comment) for more details on how this led to test flakiness in the `transfer-lease` roachtest suite.

Release notes: None.

Release justification: Avoids GA-blocking roachtest failures.

90107: execbuilder: fix enforce_home_region erroring of input table to LOJ r=rytaft a=msirek

Fixes #88788

This fixes erroring out of locality-optimized join when the input table's home region does not match the gateway region and session flag `enforce_home_region` is true.

Release note (bug fix): This patch fixes detection and erroring out of queries using locality-optimized join when session setting enforce_home_region is true and the input table to the join has no home region or its home region does not match the gateway region.

90165: sql,server: increase severity of upgraded-related logging r=ajwerner a=knz

Informs #90148.

This increases the severity from INFO in the following cases:

- in the case when `SET CLUSTER SETTING version` is issued from a SQL client (WARNING in case of failure).
- in the case when the server spontaneously decides to upgrade in the background (ERROR in case of failure).

Release note: None

90166: sql/rowenc: remove leftover log in test r=mgartner a=mgartner

Epic: None
Release note: None

Co-authored-by: Mark Sirek <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
Co-authored-by: Raphael 'kena' Poss <[email protected]>
Co-authored-by: Marcus Gartner <[email protected]>
@craig craig bot closed this as completed in fd3b0b9 Oct 18, 2022
@blathers-crl blathers-crl bot mentioned this issue Oct 18, 2022
Merged
blathers-crl bot pushed a commit that referenced this issue Oct 18, 2022
@nvanbenschoten
kv: reacquire proscribed leases on drain, then transfer
80719bc 
Fixes #83372.
Fixes #90022.
Fixes #89963.
Fixes #89962.

This commit instructs stores to reacquire proscribed leases when draining in
order to subsequently transfer them away. This addresses a source of flakiness
in `transfer-lease` roachtests where some lease would not be transferred away
before the drain completed. This could result in range unavailable for up to 9
seconds while other replicas waited out the lease'S expiration. This is because
only the previous leaseholder knows that a proscribed lease is invalid. All
other replicas still consider the lease to be valid.

This failure mode was always present if a lease transfer failed during a drain.
However, it became more likely with 034611b. With that change, we began
rejecting lease transfers that were deemed to be "unsafe" more frequently.
034611b introduced a best-effort, graceful version of this check and an
airtight, ungraceful version of the check. The former performs the check before
revoking the outgoing leaseholder's lease while the latter performs the check
after revoking the outgoing leaseholder's lease. In rare cases, it was possible
to hit the airtight, ungraceful check and cause the lease to be proscribed. See
#83261 (comment)
for more details on how this led to test flakiness in the `transfer-lease`
roachtest suite.

Release notes: None.

Release justification: Avoids GA-blocking roachtest failures.
blathers-crl bot pushed a commit that referenced this issue Oct 18, 2022
@nvanbenschoten
kv: reacquire proscribed leases on drain, then transfer
06bbac9 
Fixes #83372.
Fixes #90022.
Fixes #89963.
Fixes #89962.

This commit instructs stores to reacquire proscribed leases when draining in
order to subsequently transfer them away. This addresses a source of flakiness
in `transfer-lease` roachtests where some lease would not be transferred away
before the drain completed. This could result in range unavailable for up to 9
seconds while other replicas waited out the lease'S expiration. This is because
only the previous leaseholder knows that a proscribed lease is invalid. All
other replicas still consider the lease to be valid.

This failure mode was always present if a lease transfer failed during a drain.
However, it became more likely with 034611b. With that change, we began
rejecting lease transfers that were deemed to be "unsafe" more frequently.
034611b introduced a best-effort, graceful version of this check and an
airtight, ungraceful version of the check. The former performs the check before
revoking the outgoing leaseholder's lease while the latter performs the check
after revoking the outgoing leaseholder's lease. In rare cases, it was possible
to hit the airtight, ungraceful check and cause the lease to be proscribed. See
#83261 (comment)
for more details on how this led to test flakiness in the `transfer-lease`
roachtest suite.

Release notes: None.

Release justification: Avoids GA-blocking roachtest failures.
@blathers-crl blathers-crl bot mentioned this issue Oct 18, 2022
Merged
@lunevalex lunevalex added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. X-duplicate Closed as a duplicate of another issue. and removed C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Dec 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Projects
None yet
Milestone
22.2
Development

Successfully merging a pull request may close this issue.

kv: reacquire proscribed leases on drain, then transfer nvanbenschoten/cockroach
4 participants
@celiala @nvanbenschoten @cockroach-teamcity @lunevalex