Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/mixed-versions failed [enum removal not yet in 23.2 release] #121411

Closed
cockroach-teamcity opened this issue Mar 30, 2024 · 14 comments
Closed

roachtest: schemachange/mixed-versions failed [enum removal not yet in 23.2 release] #121411

cockroach-teamcity opened this issue Mar 30, 2024 · 14 comments
Assignees
@rafiss
Labels
branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 30, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

(test_runner.go:1185).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-37233
@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Mar 30, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Mar 30, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ 5d952f80b3e1efe2e9aaed73f1fd68433880fcb7:

(mixedversion.go:592).Run: mixed-version test failure while running step 13 (run "run schemachange workload and validation in mixed version"): full command output in run_060642.381141873_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@rafiss
Copy link
Collaborator

rafiss commented Apr 1, 2024 

{
 "workerId": 1,
 "clientTimestamp": "06:07:16.420578",
 "ops": [
  "BEGIN",
  {
   "sql": "ALTER TYPE schema_w2_104.enum_w2_164 DROP VALUE 'ffda'"
  },
  {
   "sql": "COMMENT ON SCHEMA schema_w2_95 IS 'comment from the RSW'"
  },
  {
   "sql": "CREATE FUNCTION schema_w3_243.udf_w1_300(enum_0 public.enum_w0_4, enum_1 public._enum_w0_4, enum_2 schema_w0_19.enum_w1_56, enum_3 schema_w0_19._enum_w1_56, enum_4 schema_w0_18.enum_w1_123, enum_5 schema_w0_18._enum_w1_123, enum_6 schema_w2_43.enum_w2_122, enum_7 schema_w2_43._enum_w2_122, enum_8 schema_w4_97.enum_w0_159, enum_9 schema_w4_97._enum_w0_159, enum_10 schema_w2_104.enum_w2_164, enum_11 schema_w2_104._enum_w2_164, enum_12 schema_w4_129.enum_w3_203, enum_13 schema_w4_129._enum_w3_203, enum_14 schema_w3_143.enum_w2_244, enum_15 schema_w3_143._enum_w2_244, enum_16 schema_w4_229.enum_w0_277, enum_17 schema_w4_229._enum_w0_277, enum_18 schema_w3_99.enum_w1_287)\n\tRETURNS schema_w0_19.enum_w1_56\n\tLANGUAGE SQL\n\tAS $$ SELECT NULL WHERE TRUE $$"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (42P01): could not validate enum value removal for \"ffda\": relation \"[526]\" does not exist (SQLSTATE 42P01)",

This should have been addressed by #121237 - but maybe that commit wasn't picked up in this test run.

@rafiss rafiss added GA-blocker P-1 Issues/test failures with a fix SLA of 1 month and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 1, 2024
@rafiss rafiss self-assigned this Apr 1, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 2, 2024
Closed
@rafiss rafiss removed the GA-blocker label Apr 2, 2024
@rafiss
Copy link
Collaborator

rafiss commented Apr 3, 2024 •
edited
Loading

this is testing an upgrade from v23.2.1 to the tip of the release-24.1 branch. from mixed-version-test.log:

[mixed-version-test/11_restart-node-2-with-binary-version-v23.2.1] 06:05:56 runner.go:389: ---------- STARTING (11): restart node 2 with binary version v23.2.1 ----------
[mixed-version-test/11_restart-node-2-with-binary-version-v23.2.1] 06:05:56 runner.go:412: current cluster configuration:
                            n1                                  n2                                  n3                                  n4
released versions           release-24.1                        release-24.1                        release-24.1                        release-24.1
logical binary versions     23.2-upgrading-to-24.1-step-024     23.2-upgrading-to-24.1-step-024     23.2-upgrading-to-24.1-step-024     23.2-upgrading-to-24.1-step-024
cluster versions            23.2                                23.2                                23.2                                23.2
[mixed-version-test/11_restart-node-2-with-binary-version-v23.2.1] 06:05:56 clusterupgrade.go:357: restarting node 2 into version v23.2.1
teamcity-14635191-1711862978-16-n4cpu4: stopping and waiting
teamcity-14635191-1711862978-16-n4cpu4: stopping
[mixed-version-test/11_restart-node-2-with-binary-version-v23.2.1] 06:06:17 cockroach.go:441: teamcity-14635191-1711862978-16-n4cpu4 (system): starting cockroach processes
[mixed-version-test/11_restart-node-2-with-binary-version-v23.2.1] 06:06:23 runner.go:389: ---------- FINISHED [26.24087357s] (11): restart node 2 with binary version v23.2.1 ----------
[mixed-version-test/12_restart-node-1-with-binary-version-v23.2.1] 06:06:23 runner.go:389: ---------- STARTING (12): restart node 1 with binary version v23.2.1 ----------
[mixed-version-test/12_restart-node-1-with-binary-version-v23.2.1] 06:06:23 runner.go:412: current cluster configuration:
                            n1                                  n2          n3                                  n4
released versions           release-24.1                        v23.2.1     release-24.1                        release-24.1
logical binary versions     23.2-upgrading-to-24.1-step-024     23.2        23.2-upgrading-to-24.1-step-024     23.2-upgrading-to-24.1-step-024
cluster versions            23.2                                23.2        23.2                                23.2
[mixed-version-test/12_restart-node-1-with-binary-version-v23.2.1] 06:06:23 clusterupgrade.go:357: restarting node 1 into version v23.2.1
teamcity-14635191-1711862978-16-n4cpu4: stopping and waiting
teamcity-14635191-1711862978-16-n4cpu4: stopping
[mixed-version-test/12_restart-node-1-with-binary-version-v23.2.1] 06:06:35 cluster_synced.go:1814: teamcity-14635191-1711862978-16-n4cpu4: checking certs.tar
[mixed-version-test/12_restart-node-1-with-binary-version-v23.2.1] 06:06:36 cockroach.go:441: teamcity-14635191-1711862978-16-n4cpu4 (system): starting cockroach processes
[mixed-version-test/12_restart-node-1-with-binary-version-v23.2.1] 06:06:41 runner.go:389: ---------- FINISHED [18.424780622s] (12): restart node 1 with binary version v23.2.1 ----------

but v23.2.1 does not have the changes from #121237, so we still hit the could not validate enum value removal error. we likely should disable this mixed version test on the 24.1 branch until #121237 is in a 23.2 release.

@rafiss rafiss added P-2 Issues/test failures with a fix SLA of 3 months and removed P-1 Issues/test failures with a fix SLA of 1 month labels Apr 3, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 3, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ 369e94ddad5fc4d33038a9a6b5303cef1b19df99:

(test_runner.go:1185).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs
Copy link
Contributor

we likely should disable this mixed version test on the 24.1 branch until #121237 is in a 23.2 release

This also happens on master as 23.2 is also master's predecessor for now (i.e., this can lead to CI and bors flakes).

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ ab15ba5ecd31719682e19805ebc03cd291dccf94:

(mixedversion.go:592).Run: mixed-version test failure while running step 14 (run "run schemachange workload and validation in mixed version"): full command output in run_060807.753048413_n4_COCKROACHRANDOMSEED2.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 5, 2024
Closed
@exalate-issue-sync exalate-issue-sync bot added P-1 Issues/test failures with a fix SLA of 1 month and removed P-2 Issues/test failures with a fix SLA of 3 months labels Apr 5, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ d141d2c700eb1e323f97e15831c3b9915aba9248:

(mixedversion.go:592).Run: mixed-version test failure while running step 6 (run "run schemachange workload and validation in mixed version"): full command output in run_060336.989566585_n4_COCKROACHRANDOMSEED6.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 7, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ 02b2c2dca13de3c9aac2c12d75ac41d22091ece2:

(mixedversion.go:592).Run: mixed-version test failure while running step 8 (run "run schemachange workload and validation in mixed version"): full command output in run_060226.601895424_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@fqazi
Copy link
Collaborator

fqazi commented Apr 9, 2024

The latest failures are related to: #121821

@fqazi fqazi assigned fqazi and unassigned rafiss Apr 9, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ 87740bd080658e93b40637556c9dfa27a91f8cf5:

(mixedversion.go:592).Run: mixed-version test failure while running step 23 (run "run schemachange workload and validation after upgrade has finalized"): full command output in run_063522.360364260_n4_cockroach-debug-doct.log: COMMAND_PROBLEM: exit status 125
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ 592a66827e1864303791b4f3940fb52331bc949d:

(mixedversion.go:592).Run: mixed-version test failure while running step 14 (run "run schemachange workload and validation in mixed version"): full command output in run_063053.335777124_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Apr 12, 2024
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ 53fcac10e573f8b3f0f6729fb7840f5ffba400b3:

(mixedversion.go:592).Run: mixed-version test failure while running step 19 (run "run schemachange workload and validation in mixed version"): full command output in run_063231.459590805_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 18, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.1 @ 3111a51bdb2cbf889c92f4440bf88b0eb682cb96:

(mixedversion.go:592).Run: mixed-version test failure while running step 11 (run "run schemachange workload and validation in mixed version"): full command output in run_062827.690359596_n4_COCKROACHRANDOMSEED2.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@exalate-issue-sync exalate-issue-sync bot assigned rafiss and unassigned fqazi Apr 22, 2024
@rafiss rafiss changed the title roachtest: schemachange/mixed-versions failed roachtest: schemachange/mixed-versions failed [enum removal not yet in 23.2 release] Apr 22, 2024
This was referenced Apr 24, 2024
Closed
Closed
Closed
@rafiss rafiss mentioned this issue Apr 30, 2024
Merged
@cockroach-teamcity cockroach-teamcity mentioned this issue May 1, 2024
Closed
craig bot pushed a commit that referenced this issue May 1, 2024
@rafiss
Merge #123326
91d5a5f 
123326: roachtest: use latest predecessor in schemachange/mixed-versions r=rafiss a=rafiss

informs: #121411
Release note: None

Co-authored-by: Rafi Shamim <[email protected]>
This was referenced May 1, 2024
Merged
Merged
@rafiss
Copy link
Collaborator

rafiss commented May 7, 2024

The latest failure is

  | Error: ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: user defined functions cannot reference other user defined functions (SQLSTATE 0A000)

this is fixed by #121990

closing since it hasn't failed in 2 weeks.

@rafiss rafiss closed this as completed May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rafiss rafiss

Labels
branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
No open projects
SQL Foundations
Status: Done
Milestone
24.1
Development

No branches or pull requests
4 participants
@rafiss @fqazi @cockroach-teamcity @renatolabs