Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backupccl: possibly deflake TestBackupRestoreAppend #107334

Merged
merged 1 commit into from
Jul 21, 2023

Conversation

stevendanna
Copy link
Collaborator

In CI we've seen this test fail with:

backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq:
restart transaction: TransactionRetryWithProtoRefreshError:
TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed
preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1
pri=0.03587869 epo=0 ts=1689916873.568949604,1
min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING
rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0

The RETURNING clauses on these two UPDATE statements prevent automatic transaction retries.

Here we wrap the queries in an explicit transaction with a retry loop which should prevent the test failure test, assuming that the update isn't so contended it won't ever complete.

I am unable to stress this enough locally to reproduce this error.

Probably Fixes #107330

Epic: none

Release note: None

In CI we've seen this test fail with:

    backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq:
    restart transaction: TransactionRetryWithProtoRefreshError:
    TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed
    preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1
    pri=0.03587869 epo=0 ts=1689916873.568949604,1
    min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING
    rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0

The `RETURNING` clauses on these two `UPDATE` statements prevent
automatic transaction retries.

Here we wrap the queries in an explicit transaction with a retry loop
which should prevent the test failure test, assuming that the update
isn't so contended it won't ever complete.

I am unable to stress this enough locally to reproduce this error.

Probably Fixes cockroachdb#107330

Epic: none

Release note: None
@stevendanna stevendanna added the backport-23.1.x Flags PRs that need to be backported to 23.1 label Jul 21, 2023
@stevendanna stevendanna requested a review from a team as a code owner July 21, 2023 09:22
@stevendanna stevendanna requested review from lidorcarmel and removed request for a team July 21, 2023 09:22
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@stevendanna
Copy link
Collaborator Author

TFTR!

bors r=adityamaru

Failure is TestBackupRestoreJobTagAndLabel which I've skipped under race in another PR.

@craig
Copy link
Contributor

craig bot commented Jul 21, 2023

Build succeeded:

@craig craig bot merged commit cdf12cf into cockroachdb:master Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-23.1.x Flags PRs that need to be backported to 23.1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ccl/backupccl: TestBackupRestoreAppend failed
3 participants