Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: autoupgrade failed #99459

Closed
cockroach-teamcity opened this issue Mar 24, 2023 · 12 comments
Closed

roachtest: autoupgrade failed #99459

cockroach-teamcity opened this issue Mar 24, 2023 · 12 comments
Assignees
@ajwerner
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-duplicate Closed as a duplicate of another issue.
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 24, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.autoupgrade failed with artifacts on release-23.1 @ 52e55d2ef172b7cfec14e8a0a954f8864b2be779:

test artifacts and logs in: /artifacts/autoupgrade/run_1
(autoupgrade.go:233).func1: determining cluster version: pq: operation "show cluster setting version" timed out after 2m0s (given timeout 2m0s): value differs between local setting ([18 8 8 22 16 2 24 0 32 45]) and KV ([18 8 8 22 16 2 24 0 32 44]); try again later (<nil> after 1m59.98358863s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-25884
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 24, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 24, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Mar 24, 2023
@tbg tbg mentioned this issue Mar 24, 2023
Closed
@tbg
Copy link
Member

tbg commented Mar 24, 2023

This looks like a duplicate of #99115. We need to apply the same fix to this test, I'll send a PR.

I asked here what recent change to SHOW CLUSTER SETTINGS is responsible.

@tbg tbg mentioned this issue Mar 24, 2023
Closed
tbg added a commit to tbg/cockroach that referenced this issue Mar 24, 2023
@tbg
roachtest: deflake autoupgrade
4fd268d 
Similar to: https://github.com/cockroachdb/cockroach/pull/99162/files#r1147643803

Fixes cockroachdb#99459.

Epic: none
Release note: None
@tbg tbg added the X-duplicate Closed as a duplicate of another issue. label Mar 24, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 25, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.autoupgrade failed with artifacts on release-23.1 @ f351747ed97862fc037717cadec23f18073fb6be:

test artifacts and logs in: /artifacts/autoupgrade/run_1
(autoupgrade.go:233).func1: determining cluster version: pq: operation "show cluster setting version" timed out after 2m0s (given timeout 2m0s): value differs between local setting ([18 8 8 22 16 2 24 0 32 53]) and KV ([18 8 8 22 16 2 24 0 32 52]); try again later (<nil> after 1m59.986290257s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.autoupgrade failed with artifacts on release-23.1 @ b995ddff53ee95a30537a366c6cd8e0e13fcee79:

test artifacts and logs in: /artifacts/autoupgrade/run_1
(autoupgrade.go:233).func1: determining cluster version: pq: operation "show cluster setting version" timed out after 2m0s (given timeout 2m0s): value differs between local setting ([18 8 8 22 16 2 24 0 32 83]) and KV ([18 8 8 22 16 2 24 0 32 82]); try again later (<nil> after 1m58.414245109s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.autoupgrade failed with artifacts on release-23.1 @ 186218edb86b5974501b2976cfd0704d4d6d1beb:

test artifacts and logs in: /artifacts/autoupgrade/run_1
(autoupgrade.go:233).func1: determining cluster version: pq: operation "show cluster setting version" timed out after 2m0s (given timeout 2m0s): value differs between local setting ([18 8 8 22 16 2 24 0 32 7]) and KV ([18 8 8 22 16 2 24 0 32 6]); try again later (<nil> after 1m58.274632189s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@nvanbenschoten nvanbenschoten removed the T-kv KV Team label Mar 27, 2023
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Mar 27, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.autoupgrade failed with artifacts on release-23.1 @ 3551dc089ac3efcb3772df2109164dcb01e79d8f:

test artifacts and logs in: /artifacts/autoupgrade/run_1
(autoupgrade.go:233).func1: determining cluster version: pq: operation "show cluster setting version" timed out after 2m0.001s (given timeout 2m0s): value differs between local setting ([18 8 8 22 16 2 24 0 32 53]) and KV ([18 8 8 22 16 2 24 0 32 52]); try again later (<nil> after 1m58.558490566s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.autoupgrade failed with artifacts on release-23.1 @ aec78f33d45a8376a0ecec885688bae60dbfb85c:

test artifacts and logs in: /artifacts/autoupgrade/run_1
(autoupgrade.go:233).func1: determining cluster version: pq: operation "show cluster setting version" timed out after 2m0s (given timeout 2m0s): value differs between local setting ([18 8 8 22 16 2 24 0 32 91]) and KV ([18 8 8 22 16 2 24 0 32 90]); try again later (<nil> after 1m58.78890457s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@tbg
Copy link
Member

tbg commented Mar 30, 2023

This looks same as #99115 (comment) which was (according to @ajwerner) fixed in #99665 and backported on the 28th: #99706 (comment)

So I'm not sure why this is still failing, am I missing something @ajwerner?

@ajwerner
Copy link
Contributor

I'm not really sure. We see the migrations seemingly being held up by the code which is trying to read the cluster setting and then the test fails because it fails to read the cluster setting. I suppose that sort of makes sense because there are some privilege checks or what not. I have a hunch that #99967 will fix it because it'll make SHOW CLUSTER SETTING not take so darn long. I'm worried I don't have much time to debug this given other goals

@ajwerner
Copy link
Contributor

It is definitely the case that the backported issue does not fix it.

@tbg tbg mentioned this issue Mar 31, 2023
Merged
@tbg
Copy link
Member

tbg commented Mar 31, 2023

Thanks @ajwerner.
Will be closed by backport of your PR, #100259, see corresponding master issue1

Footnotes

  1. https://github.com/cockroachdb/cockroach/issues/99558#issuecomment-1491388443 ↩

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 31, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.autoupgrade failed with artifacts on release-23.1 @ 7e72aae900c3ff4b44f1643c2d7ba55fbb2cbe23:

test artifacts and logs in: /artifacts/autoupgrade/run_1
(autoupgrade.go:233).func1: determining cluster version: pq: operation "show cluster setting version" timed out after 2m0s (given timeout 2m0s): value differs between local setting ([18 8 8 22 16 2 24 0 32 67]) and KV ([18 8 8 22 16 2 24 0 32 66]); try again later (<nil> after 1m59.335351596s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@ajwerner
Copy link
Contributor

Fixed by #100259

@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajwerner ajwerner

Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-duplicate Closed as a duplicate of another issue.
Projects
None yet
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

roachtest: deflake autoupgrade tbg/cockroach
5 participants
@ajwerner @celiala @tbg @nvanbenschoten @cockroach-teamcity