Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgrade/upgrades: TestBackfillJobsInfoTable failed #103046

Closed
cockroach-teamcity opened this issue May 10, 2023 · 6 comments · Fixed by #104879
Closed

upgrade/upgrades: TestBackfillJobsInfoTable failed #103046

cockroach-teamcity opened this issue May 10, 2023 · 6 comments · Fixed by #104879
Assignees
@adityamaru
Labels
A-jobs branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-jobs
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 10, 2023 •
edited by cockroach-jira-scripts
Loading

upgrade/upgrades.TestBackfillJobsInfoTable failed with artifacts on release-23.1 @ a0a3b078882aa1f1a73f6683fd5fef0137626fed:

=== RUN   TestBackfillJobsInfoTable
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable3025739298
    test_log_scope.go:79: use -show-logs to present logs inline
    backfill_jobs_info_table_migration_test.go:107: query '
        SELECT count(*) FROM system.jobs AS j, system.job_info AS i
        WHERE j.id = i.job_id AND (j.payload = i.value OR j.progress = i.value) AND (j.id >= 1 AND j.id <= 5)
        ': expected:
        14
        
        got:
        12
        
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable3025739298
--- FAIL: TestBackfillJobsInfoTable (32.91s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-27810
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 10, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone May 10, 2023
@blathers-crl blathers-crl bot added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label May 10, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue May 10, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

upgrade/upgrades.TestBackfillJobsInfoTable failed with artifacts on release-23.1 @ 43347361e631585acba789669c886d5343276ac0:

=== RUN   TestBackfillJobsInfoTable
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable309814320
    test_log_scope.go:79: use -show-logs to present logs inline
    backfill_jobs_info_table_migration_test.go:107: query '
        SELECT count(*) FROM system.jobs AS j, system.job_info AS i
        WHERE j.id = i.job_id AND (j.payload = i.value OR j.progress = i.value) AND (j.id >= 1 AND j.id <= 5)
        ': expected:
        14
        
        got:
        10
        
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable309814320
--- FAIL: TestBackfillJobsInfoTable (35.15s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss rafiss removed the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label May 11, 2023
@rafiss rafiss added the T-jobs label May 11, 2023
@blathers-crl blathers-crl bot added the A-jobs label May 11, 2023
@cockroach-teamcity
Copy link
Member Author

upgrade/upgrades.TestBackfillJobsInfoTable failed with artifacts on release-23.1 @ 6cf937278a19b59b50abdf78f0e257807367600d:

=== RUN   TestBackfillJobsInfoTable
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable2878705094
    test_log_scope.go:79: use -show-logs to present logs inline
    backfill_jobs_info_table_migration_test.go:107: query '
        SELECT count(*) FROM system.jobs AS j, system.job_info AS i
        WHERE j.id = i.job_id AND (j.payload = i.value OR j.progress = i.value) AND (j.id >= 1 AND j.id <= 5)
        ': expected:
        14
        
        got:
        10
        
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable2878705094
--- FAIL: TestBackfillJobsInfoTable (32.02s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

upgrade/upgrades.TestBackfillJobsInfoTable failed with artifacts on release-23.1 @ f50bdad0bccba79b5017cec446849b9831085457:

=== RUN   TestBackfillJobsInfoTable
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable1503229787
    test_log_scope.go:79: use -show-logs to present logs inline
    backfill_jobs_info_table_migration_test.go:107: query '
        SELECT count(*) FROM system.jobs AS j, system.job_info AS i
        WHERE j.id = i.job_id AND (j.payload = i.value OR j.progress = i.value) AND (j.id >= 1 AND j.id <= 5)
        ': expected:
        14
        
        got:
        12
        
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable1503229787
--- FAIL: TestBackfillJobsInfoTable (30.49s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

upgrade/upgrades.TestBackfillJobsInfoTable failed with artifacts on release-23.1 @ 029955855b3243eecfbb7fcc0507ec663f7ddf6f:

=== RUN   TestBackfillJobsInfoTable
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable3254874492
    test_log_scope.go:79: use -show-logs to present logs inline
    backfill_jobs_info_table_migration_test.go:107: query '
        SELECT count(*) FROM system.jobs AS j, system.job_info AS i
        WHERE j.id = i.job_id AND (j.payload = i.value OR j.progress = i.value) AND (j.id >= 1 AND j.id <= 5)
        ': expected:
        14
        
        got:
        12
        
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable3254874492
--- FAIL: TestBackfillJobsInfoTable (31.02s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

upgrade/upgrades.TestBackfillJobsInfoTable failed with artifacts on release-23.1 @ 8cfaf3b4731273cedaa9e81600a7b1778700b702:

=== RUN   TestBackfillJobsInfoTable
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable3319567719
    test_log_scope.go:79: use -show-logs to present logs inline
    backfill_jobs_info_table_migration_test.go:107: query '
        SELECT count(*) FROM system.jobs AS j, system.job_info AS i
        WHERE j.id = i.job_id AND (j.payload = i.value OR j.progress = i.value) AND (j.id >= 1 AND j.id <= 5)
        ': expected:
        14
        
        got:
        9
        
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable3319567719
--- FAIL: TestBackfillJobsInfoTable (32.52s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

upgrade/upgrades.TestBackfillJobsInfoTable failed with artifacts on release-23.1 @ 7cf4908e4d88fa1cc4127bbc4c4e327e2aed8b6d:

=== RUN   TestBackfillJobsInfoTable
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable2558230056
    test_log_scope.go:79: use -show-logs to present logs inline
    backfill_jobs_info_table_migration_test.go:107: query '
        SELECT count(*) FROM system.jobs AS j, system.job_info AS i
        WHERE j.id = i.job_id AND (j.payload = i.value OR j.progress = i.value) AND (j.id >= 1 AND j.id <= 5)
        ': expected:
        14
        
        got:
        10
        
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/194b1b11adb2bb86a3c51c4924595952/logTestBackfillJobsInfoTable2558230056
--- FAIL: TestBackfillJobsInfoTable (27.57s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@adityamaru adityamaru self-assigned this Jun 14, 2023
@adityamaru adityamaru mentioned this issue Jun 14, 2023
Merged
craig bot pushed a commit that referenced this issue Jun 14, 2023
@jeffswenson @renatolabs @adityamaru
Merge #104821 #104876 #104879
3441110 
104821: upgrade: checkpoint fence version for multi-tenant clusters r=JeffSwenson a=JeffSwenson

Previously, the tenant upgrade interlock checkpointed the version attached to the upgrade, when it should have checkpointed the fence version. This made it possible for an upgrade to skip versions. The conditions needed for this are rare, but could be triggered by crashes in the upgrade system or by the autoscaler downscaling a serverless cluster with multiple nodes.

Release Note: None

Fixes: #104884

104876: logic: keep cockroach-go logs in case of failure r=rafiss a=renatolabs

This commit makes use of the new `CockroachLogsDirOpt` option introduced in [1] to pass a custom directory where cockroach logs will be stored. This directory is relative to TMPDIR, which is part of the artifacts path when the test runs in CI. With this change, when a logic test using the cockroach-go config fails, we'll be able to see both the test logs as well as the cockroach logs themselves. The latter will be in the `cockroach.stderr` file.

[1] cockroachdb/cockroach-go#170

Epic: none

Release note: None

104879: upgrades: deflake TestBackfillJobsInfoTable r=stevendanna a=adityamaru

Previously, this test created jobs that could be adopted during test execution causing the number of payload/progress updates to be variable. With this change we create startable jobs so that they are not adopted and the number of payload/progress remain constant.

Fixes: #103046
Release note: None

Co-authored-by: Jeff <[email protected]>
Co-authored-by: Renato Costa <[email protected]>
Co-authored-by: adityamaru <[email protected]>
@craig craig bot closed this as completed in f523899 Jun 14, 2023
@adityamaru adityamaru mentioned this issue Jun 15, 2023
Merged
adityamaru added a commit to adityamaru/cockroach that referenced this issue Jun 15, 2023
@adityamaru
upgrades: deflake TestBackfillJobsInfoTable
9ffbba6 
Previously, this test created jobs that could be adopted
during test execution causing the number of payload/progress
updates to be variable. With this change we create startable
jobs so that they are not adopted and the number of payload/progress
remain constant.

Fixes: cockroachdb#103046
Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adityamaru adityamaru

Labels
A-jobs branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-jobs
Projects
None yet
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

upgrades: deflake TestBackfillJobsInfoTable adityamaru/cockroach
3 participants
@rafiss @adityamaru @cockroach-teamcity