Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/backupccl: TestBackupRestoreAppend failed #107330

Closed
cockroach-teamcity opened this issue Jul 21, 2023 · 0 comments · Fixed by #107334
Closed

ccl/backupccl: TestBackupRestoreAppend failed #107330

cockroach-teamcity opened this issue Jul 21, 2023 · 0 comments · Fixed by #107334
Assignees
@stevendanna
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jul 21, 2023 •
edited by cockroach-jira-scripts
Loading

ccl/backupccl.TestBackupRestoreAppend failed with artifacts on release-23.1 @ 043024b889af86e0b8f23f09619d8d3c2c9acd13:

=== RUN   TestBackupRestoreAppend
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/95e138e66d69292427dfb9528cf06d04/logTestBackupRestoreAppend4161060006
    test_log_scope.go:79: use -show-logs to present logs inline
    testutils.go:199: no Invalid Descriptors
    backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1 pri=0.03587869 epo=0 ts=1689916873.568949604,1 min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0
    testutils.go:199: no Invalid Descriptors
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/95e138e66d69292427dfb9528cf06d04/logTestBackupRestoreAppend4161060006
--- FAIL: TestBackupRestoreAppend (53.64s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-29991
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery labels Jul 21, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Jul 21, 2023
@stevendanna stevendanna self-assigned this Jul 21, 2023
@stevendanna stevendanna mentioned this issue Jul 21, 2023
Merged
craig bot pushed a commit that referenced this issue Jul 21, 2023
@stevendanna
Merge #107334 #107338
cdf12cf 
107334: backupccl: possibly deflake TestBackupRestoreAppend r=adityamaru a=stevendanna

In CI we've seen this test fail with:

    backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq:
    restart transaction: TransactionRetryWithProtoRefreshError:
    TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed
    preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1
    pri=0.03587869 epo=0 ts=1689916873.568949604,1
    min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING
    rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0

The `RETURNING` clauses on these two `UPDATE` statements prevent automatic transaction retries.

Here we wrap the queries in an explicit transaction with a retry loop which should prevent the test failure test, assuming that the update isn't so contended it won't ever complete.

I am unable to stress this enough locally to reproduce this error.

Probably Fixes #107330

Epic: none

Release note: None

107338: backupccl: skip TestBackupRestoreJobTagAndLabel under race r=dt a=stevendanna

This test fails under race when running in a tenant occasionally. Requires further investigation.

Informs: #107336

Release note: None

Co-authored-by: Steven Danna <[email protected]>
@craig craig bot closed this as completed in 245ed7e Jul 21, 2023
@blathers-crl blathers-crl bot mentioned this issue Jul 21, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Jul 21, 2023
@stevendanna
backupccl: possibly deflake TestBackupRestoreAppend
6fa2741 
In CI we've seen this test fail with:

    backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq:
    restart transaction: TransactionRetryWithProtoRefreshError:
    TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed
    preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1
    pri=0.03587869 epo=0 ts=1689916873.568949604,1
    min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING
    rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0

The `RETURNING` clauses on these two `UPDATE` statements prevent
automatic transaction retries.

Here we wrap the queries in an explicit transaction with a retry loop
which should prevent the test failure test, assuming that the update
isn't so contended it won't ever complete.

I am unable to stress this enough locally to reproduce this error.

Probably Fixes #107330

Epic: none

Release note: None
THardy98 pushed a commit to THardy98/cockroach that referenced this issue Jul 24, 2023
@stevendanna
backupccl: possibly deflake TestBackupRestoreAppend
612f595 
In CI we've seen this test fail with:

    backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq:
    restart transaction: TransactionRetryWithProtoRefreshError:
    TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed
    preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1
    pri=0.03587869 epo=0 ts=1689916873.568949604,1
    min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING
    rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0

The `RETURNING` clauses on these two `UPDATE` statements prevent
automatic transaction retries.

Here we wrap the queries in an explicit transaction with a retry loop
which should prevent the test failure test, assuming that the update
isn't so contended it won't ever complete.

I am unable to stress this enough locally to reproduce this error.

Probably Fixes cockroachdb#107330

Epic: none

Release note: None
stevendanna added a commit that referenced this issue Aug 17, 2023
@stevendanna
backupccl: possibly deflake TestBackupRestoreAppend
10d8618 
In CI we've seen this test fail with:

    backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq:
    restart transaction: TransactionRetryWithProtoRefreshError:
    TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed
    preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1
    pri=0.03587869 epo=0 ts=1689916873.568949604,1
    min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING
    rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0

The `RETURNING` clauses on these two `UPDATE` statements prevent
automatic transaction retries.

Here we wrap the queries in an explicit transaction with a retry loop
which should prevent the test failure test, assuming that the update
isn't so contended it won't ever complete.

I am unable to stress this enough locally to reproduce this error.

Probably Fixes #107330

Epic: none

Release note: None
stevendanna added a commit to stevendanna/cockroach that referenced this issue Aug 18, 2023
@stevendanna
backupccl: possibly deflake TestBackupRestoreAppend
c9e5d6e 
In CI we've seen this test fail with:

    backup_test.go:670: error scanning '&{<nil> 0xc019c5b580}': pq:
    restart transaction: TransactionRetryWithProtoRefreshError:
    TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed
    preemptive refresh): "sql txn" meta={id=77054b51 key=/Table/121/1
    pri=0.03587869 epo=0 ts=1689916873.568949604,1
    min=1689916873.436580640,0 seq=1000} lock=true stat=PENDING
    rts=1689916873.436580640,0 wto=false gul=1689916873.936580640,0

The `RETURNING` clauses on these two `UPDATE` statements prevent
automatic transaction retries.

Here we wrap the queries in an explicit transaction with a retry loop
which should prevent the test failure test, assuming that the update
isn't so contended it won't ever complete.

I am unable to stress this enough locally to reproduce this error.

Probably Fixes cockroachdb#107330

Epic: none

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stevendanna stevendanna

Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

backupccl: possibly deflake TestBackupRestoreAppend stevendanna/cockroach
2 participants
@stevendanna @cockroach-teamcity