roachtest: failover/chaos/read-only failed #119454

cockroach-teamcity · 2024-02-21T14:24:00Z

roachtest.failover/chaos/read-only failed with artifacts on master @ e50b0ec4d3a53f81e26f3776ae3f3be55d435a9a:

(test_runner.go:1153).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!

Jira issue: CRDB-36206}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2024-02-22T14:24:51Z

roachtest.failover/chaos/read-only failed with artifacts on master @ 04f0416d526a43741d22fd03966758dcccdeb79f:

VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14107820-1708583418-77-n10cpu2-0006

**Other Failure**
(test_runner.go:1153).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-02-26T13:56:52Z

roachtest.failover/chaos/read-only failed with artifacts on master @ c9c3cc5f3c3a4a6ab556f4b9d5b6ec0381901bdb:

(failover.go:1774).sleepFor: sleep failed: context canceled
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
(cluster.go:2298).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

Previously, the multiple failures were started and finished independently. This caused a problem if the ability to recover from one failure depended on a different failure recovering first. To mitigate this and to add a little more chaos, start and recover each failure in a seperate goroutine. This will allow the "most important" failure to recover first so that the others can recover if they depend on each other. Note that this is more important today while we don't support all the failure modes that the chaos implements. Specifically we don't handle partial partitions handling yet. Epic: none Fixes: cockroachdb#119085 Fixes: cockroachdb#119347 Fixes: cockroachdb#119361 Fixes: cockroachdb#119454 Release note: None

cockroach-teamcity · 2024-02-29T14:37:00Z

roachtest.failover/chaos/read-only failed with artifacts on master @ 9a494f4e4846a67ceea18c4b4993cefbeb19c6d9:

(test_runner.go:1161).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-03-05T15:06:27Z

roachtest.failover/chaos/read-only failed with artifacts on master @ bf013ea0a5311726e65d37e8f047ce39ea2d5f10:

(test_runner.go:1161).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-03-06T14:33:25Z

roachtest.failover/chaos/read-only failed with artifacts on master @ ce5f34ea97475f45fa354e58aacf424779d0de49:

(test_runner.go:1161).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-03-12T13:15:32Z

roachtest.failover/chaos/read-only failed with artifacts on master @ 431d56840cacdc86638ccb48e9f3bdc16897d341:

(test_runner.go:1161).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-03-26T13:29:13Z

roachtest.failover/chaos/read-only failed with artifacts on master @ 6df7436b6388eb7823b9f63a42aa75e1fbc89a85:

(test_runner.go:1185).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: failover/chaos/read-only failed #120833 roachtest: failover/chaos/read-only failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-23.2.4-rc]
roachtest: failover/chaos/read-only failed #120408 roachtest: failover/chaos/read-only failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-23.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-03-27T13:39:25Z

roachtest.failover/chaos/read-only failed with artifacts on master @ ec9444f8287663777c5ebceff61bcd280491ba08:

(test_runner.go:1185).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: failover/chaos/read-only failed #120833 roachtest: failover/chaos/read-only failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-23.2.4-rc]
roachtest: failover/chaos/read-only failed #120408 roachtest: failover/chaos/read-only failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-23.2]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-04-02T14:51:44Z

roachtest.failover/chaos/read-only failed with artifacts on master @ c43f54cdde5b7578f4a0ca61de41463f0d690993:

(test_runner.go:1185).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

roachtest: failover/chaos/read-only failed #120833 roachtest: failover/chaos/read-only failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.2.4-rc]
roachtest: failover/chaos/read-only failed #120408 roachtest: failover/chaos/read-only failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-23.2]

_{This test on roachdash | Improve this report!}

119650: roachtest: make failure recovery independent r=nvanbenschoten a=andrewbaptist Previously, the multiple failures were started and finished independently. This caused a problem if the ability to recover from one failure depended on a different failure recovering first. To mitigate this, recover each failure in a separate goroutine. This will allow the "most important" failure to recover first so that the others can recover if they depend on each other. This is more important today while we don't recover from all the failure modes that chaos implements. Specifically we don't handle partial partitions fully with epoch leases. Epic: none Fixes: #119085 Fixes: #119347 Fixes: #119361 Fixes: #119454 Release note: None 122283: server: don't log for missing locality r=yuzefovich a=andrewbaptist Previously we would always log a message that the locality was unknown for requests from sql gateways. We should remove unnecessary logs from traces. Epic: none Release note: None Co-authored-by: Andrew Baptist <[email protected]>

Previously, the multiple failures were started and finished independently. This caused a problem if the ability to recover from one failure depended on a different failure recovering first. To mitigate this, recover each failure in a separate goroutine. This will allow the "most important" failure to recover first so that the others can recover if they depend on each other. This is more important today while we don't recover from all the failure modes that chaos implements. Specifically we don't handle partial partitions fully with epoch leases. Epic: none Fixes: #119085 Fixes: #119347 Fixes: #119361 Fixes: #119454 Release note: None

cockroach-teamcity added this to the 24.1 milestone Feb 21, 2024

andrewbaptist added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Feb 26, 2024

andrewbaptist mentioned this issue Feb 26, 2024

roachtest: make failure recovery independent #119650

Merged

nvanbenschoten added the P-2 Issues/test failures with a fix SLA of 3 months label Mar 12, 2024

cockroach-teamcity mentioned this issue Mar 13, 2024

roachtest: failover/chaos/read-only failed #120408

Closed

cockroach-teamcity mentioned this issue Mar 21, 2024

roachtest: failover/chaos/read-only failed #120833

Closed

arulajmani mentioned this issue Mar 31, 2024

roachtest: failover/chaos/read-write failed #121433

Closed

kvoli mentioned this issue Apr 1, 2024

roachtest: failover/chaos/read-write failed #121454

Closed

kvoli assigned andrewbaptist Apr 1, 2024

kvoli mentioned this issue Apr 1, 2024

roachtest: failover/chaos/read-write/lease=expiration failed #121457

Closed

cockroach-teamcity mentioned this issue Apr 10, 2024

roachtest: failover/chaos/read-only failed #122106

Closed

craig bot closed this as completed in 31a39cf Apr 12, 2024

blathers-crl bot mentioned this issue Apr 12, 2024

release-24.1: roachtest: make failure recovery independent #122308

Merged

blathers-crl bot mentioned this issue May 23, 2024

release-23.2: roachtest: make failure recovery independent #124608

Merged

github-project-automation bot added this to KV Aug 28, 2024

github-project-automation bot moved this to roachtest/unit test backlog in KV Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: failover/chaos/read-only failed #119454

roachtest: failover/chaos/read-only failed #119454

cockroach-teamcity commented Feb 21, 2024 •

edited by jlinder

Loading

cockroach-teamcity commented Feb 22, 2024

cockroach-teamcity commented Feb 26, 2024

cockroach-teamcity commented Feb 29, 2024

cockroach-teamcity commented Mar 5, 2024

cockroach-teamcity commented Mar 6, 2024

cockroach-teamcity commented Mar 12, 2024

cockroach-teamcity commented Mar 26, 2024

cockroach-teamcity commented Mar 27, 2024

cockroach-teamcity commented Apr 2, 2024

roachtest: failover/chaos/read-only failed #119454

roachtest: failover/chaos/read-only failed #119454

Comments

cockroach-teamcity commented Feb 21, 2024 • edited by jlinder Loading

cockroach-teamcity commented Feb 22, 2024

cockroach-teamcity commented Feb 26, 2024

cockroach-teamcity commented Feb 29, 2024

cockroach-teamcity commented Mar 5, 2024

cockroach-teamcity commented Mar 6, 2024

cockroach-teamcity commented Mar 12, 2024

cockroach-teamcity commented Mar 26, 2024

cockroach-teamcity commented Mar 27, 2024

cockroach-teamcity commented Apr 2, 2024

cockroach-teamcity commented Feb 21, 2024 •

edited by jlinder

Loading