backupccl: TestBackupRestoreJobTagAndLabel fails under race when running in a tenant #107336
Assignees
Labels
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
T-multitenant
Issues owned by the multi-tenant virtual team
Comments
stevendanna
added
C-bug
stevendanna
added a commit
to stevendanna/cockroach
that referenced
this issue
Jul 21, 2023
This test fails under race when running in a tenant occasionally. Requires further investigation. Informs: cockroachdb#107336 Release note: None
|
@irfansharif I've heard a rumor you might already know what is up here and have a fix? |
craig bot
pushed a commit
that referenced
this issue
Jul 21, 2023
107334: backupccl: possibly deflake TestBackupRestoreAppend r=adityamaru a=stevendanna
In CI we've seen this test fail with:
backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq:
restart transaction: TransactionRetryWithProtoRefreshError:
TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed
preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1
pri=0.03587869 epo=0 ts=1689916873.568949604,1
min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING
rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0
The `RETURNING` clauses on these two `UPDATE` statements prevent automatic transaction retries.
Here we wrap the queries in an explicit transaction with a retry loop which should prevent the test failure test, assuming that the update isn't so contended it won't ever complete.
I am unable to stress this enough locally to reproduce this error.
Probably Fixes #107330
Epic: none
Release note: None
107338: backupccl: skip TestBackupRestoreJobTagAndLabel under race r=dt a=stevendanna
This test fails under race when running in a tenant occasionally. Requires further investigation.
Informs: #107336
Release note: None
Co-authored-by: Steven Danna <[email protected]>
craig bot
pushed a commit
that referenced
this issue
Jul 24, 2023
107412: kvserver: fix 'observed raft log position' assertion r=irfansharif a=irfansharif Fixes #107336. Fixes #106123. Fixes #107156. Fixes #106589. It's possible to hit this assertion under --stress --race when the proposing replica is starved enough for raft ticks that it loses leadership right when it steps proposals through raft. We're relying on undocumented API semantics in the etcd raft library whereby it mutates stepped entries with the term+index its to end up in. But that's only applicable if stepping through entries as a leader. Simply relax this assertion instead. Release note: None Co-authored-by: irfan sharif <[email protected]>
THardy98
pushed a commit
to THardy98/cockroach
that referenced
this issue
Jul 24, 2023
This test fails under race when running in a tenant occasionally. Requires further investigation. Informs: cockroachdb#107336 Release note: None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the problem
This test was recently enable for tenant testing. It has since occasionally failed under race with an assertion failure:
Jira issue: CRDB-29995
The text was updated successfully, but these errors were encountered: