Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: backup-restore/mixed-version failed #115956

Closed
cockroach-teamcity opened this issue Dec 10, 2023 · 1 comment · Fixed by #116001
Closed

roachtest: backup-restore/mixed-version failed #115956

cockroach-teamcity opened this issue Dec 10, 2023 · 1 comment · Fixed by #116001
Assignees
@msbutler
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-disaster-recovery
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 10, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.backup-restore/mixed-version failed with artifacts on master @ e747f6e6857a19d6048cb184b0f55c52cb8a6390:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "verify all backups"): v22.2.17: backup 2_v22.2.17-to-v23.1.12_all-planned-on-v23.1.12-executed-on-v22.2.17_database-tpcc: error loading restored contents: error when running query [SELECT index_name, fingerprint FROM [SHOW EXPERIMENTAL_FINGERPRINTS FROM TABLE restore_2_v22_2_17_to_v23_1_12_all_planned_on_v23_1_12_executed_on_v22_2_17_database_tpcc_2.order_line] ORDER BY index_name]: pq: query execution canceled due to statement timeout (artifacts collected in restore_failure_1)
test artifacts and logs in: /artifacts/backup-restore/mixed-version/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-34327
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels Dec 10, 2023
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Dec 10, 2023
@msbutler msbutler removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Dec 11, 2023
@msbutler msbutler self-assigned this Dec 11, 2023
@msbutler msbutler mentioned this issue Dec 11, 2023
Merged
@msbutler msbutler added the P-1 Issues/test failures with a fix SLA of 1 month label Dec 11, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.backup-restore/mixed-version failed with artifacts on master @ e747f6e6857a19d6048cb184b0f55c52cb8a6390:

(mixedversion.go:539).Run: mixed-version test failure while running step 38 (run "plan and run backups"): error computing contents for backup 9_v23.1.12-to-master_all-planned-and-executed-on-random-node_database-tpcc: error when running query [SELECT index_name, fingerprint FROM [SHOW EXPERIMENTAL_FINGERPRINTS FROM TABLE tpcc.order_line] AS OF SYSTEM TIME '1702298265003157945.0000000000' ORDER BY index_name]: pq: query execution canceled due to statement timeout
test artifacts and logs in: /artifacts/backup-restore/mixed-version/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Dec 11, 2023
@dt @msbutler
Merge #115519 #115944 #116001
71df30a 
115519: pkg/cloud: bump MaxIdleConnsPerHost to 64 r=dt a=dt



115944: backupccl: notify sql stats after downloads r=dt a=dt

Release note: none.
Epic: none.

116001: roachtest: use context timeout to bound descriptor validation check r=stevendanna a=msbutler

Previously, the CheckInvalidDescriptors utility would set the crdb `statement_timeout` session variable to 1 minute, which would inadvertently cause subsequent long running queries to fail. This patch runs the validation check with a context timeout instead.

Fixes #115852
Fixes #115956
Fixes #115862

Release note: none

Co-authored-by: David Taylor <[email protected]>
Co-authored-by: Michael Butler <[email protected]>
@craig craig bot closed this as completed in 8790fa0 Dec 11, 2023
blathers-crl bot pushed a commit that referenced this issue Dec 11, 2023
@msbutler
roachtest: use context timeout to bound descriptor validation check
622ea54 
Previously, the CheckInvalidDescriptors utility would set the crdb
`statement_timeout` session variable to 1 minute, which would inadvertently
cause subsequent long running queries to fail. This patch runs the validation
check with a context timeout instead.

Fixes #115852
Fixes #115956
Fixes #115862

Release note: none
@blathers-crl blathers-crl bot mentioned this issue Dec 11, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Dec 11, 2023
@msbutler
roachtest: use context timeout to bound descriptor validation check
2c685f0 
Previously, the CheckInvalidDescriptors utility would set the crdb
`statement_timeout` session variable to 1 minute, which would inadvertently
cause subsequent long running queries to fail. This patch runs the validation
check with a context timeout instead.

Fixes #115852
Fixes #115956
Fixes #115862

Release note: none
@blathers-crl blathers-crl bot mentioned this issue Dec 11, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

roachtest: use context timeout to bound descriptor validation check msbutler/cockroach
2 participants
@cockroach-teamcity @msbutler