roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100804

cockroach-teamcity · 2023-04-06T13:18:13Z

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ b432e8c20339de5cfa7c811a9ee6f5dc98d15a1e:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(test_runner.go:1010).runTest: test timed out (5h0m0s)
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_081748.107819932_n1_cockroach-sql-insecu: ./cockroach sql --insecure -e "RESTORE  FROM LATEST IN 's3://cockroach-fixtures/backups/tpc-e/customers=500000/v22.2.1/inc-count=48?AUTH=implicit' AS OF SYSTEM TIME '2023-01-05T17:30:00Z' " returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_081748.118180757_n1_cockroach-sql-insecu.log: exit status 137

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100341 roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed [A-kv-replication C-bug C-test-failure O-roachtest O-robot T-disaster-recovery T-kv-replication branch-master]

/cc @cockroachdb/disaster-recovery _{This test on roachdash | Improve this report!

Jira issue: CRDB-26634}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2023-04-08T09:27:23Z

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ de239a7438f44d382c9aefceb65d9c39911dabd2:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 137)

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100341 roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed [A-kv-replication C-bug C-test-failure O-roachtest O-robot T-disaster-recovery T-kv-replication branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-10T08:19:00Z

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ 2f96695f75b07c872ec5f146acc1fa198135768f:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 7: dead (exit status 137)

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100341 roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed [A-kv-replication C-bug C-test-failure O-roachtest O-robot T-disaster-recovery T-kv-replication branch-master]

_{This test on roachdash | Improve this report!}

msbutler · 2023-04-10T12:09:08Z

Latest roachtest failed on cluster teamcity-9516100-1681103961-27-n10cpu8 according to test.log, which ran c5.2xlarge machines according to cluster spec. This confirms that #100286 is behaving as expected, by removing the d in the machine type.

cockroach-teamcity · 2023-04-10T12:42:33Z

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ 2f96695f75b07c872ec5f146acc1fa198135768f:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(test_runner.go:1010).runTest: test timed out (5h0m0s)
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_074205.134359961_n1_cockroach-sql-insecu: ./cockroach sql --insecure -e "RESTORE  FROM LATEST IN 's3://cockroach-fixtures/backups/tpc-e/customers=500000/v22.2.1/inc-count=48?AUTH=implicit' AS OF SYSTEM TIME '2023-01-05T17:30:00Z' " returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_074205.147800821_n1_cockroach-sql-insecu.log: exit status 137

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100341 roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed [A-kv-replication C-bug C-test-failure O-roachtest O-robot T-disaster-recovery T-kv-replication branch-master]

_{This test on roachdash | Improve this report!}

msbutler · 2023-04-12T12:50:07Z

On the failure two days ago: before node 7 died, raft was consuming 9 GB of memory:

And 7.dmesg.txt confirms the oom killer was in the building:

 [Mon Apr 10 08:17:51 2023] cockroach invoked oom-killer: gfp_mask=0x101cca(GFP_HIGHUSER_MOVABLE|__GFP_WRITE), order=0, oom_score_adj=0

msbutler · 2023-04-12T13:07:31Z

And last night, the restore test timed out after 5 hours: throughout restore, ranges were under replicated according to test.log. In fact, the system.replication_stats.txt file in the debug zip indicated that there were unavailable ranges, but i couldn't find anything in the cockroach logs to confirm this.

cockroach-teamcity · 2023-04-12T13:26:41Z

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(test_runner.go:1010).runTest: test timed out (5h0m0s)
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_082619.980953729_n1_cockroach-sql-insecu: ./cockroach sql --insecure -e "RESTORE  FROM LATEST IN 's3://cockroach-fixtures/backups/tpc-e/customers=500000/v22.2.1/inc-count=48?AUTH=implicit' AS OF SYSTEM TIME '2023-01-05T17:30:00Z' " returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_082619.998953143_n1_cockroach-sql-insecu.log: exit status 137

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

erikgrinaker · 2023-04-13T23:20:54Z

Keeping this open until the backport lands in #101507.

cockroach-teamcity · 2023-04-14T08:23:17Z

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ ad16885ca3b4567ed5eb34646fe8281fd2d740e3:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 137)

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #101426 roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed [C-test-failure GA-blocker O-roachtest O-robot T-kv-replication branch-release-23.1.0]

_{This test on roachdash | Improve this report!}

blathers-crl · 2023-04-14T09:29:09Z

cc @cockroachdb/replication

cockroach-teamcity · 2023-04-17T12:52:22Z

roachtest.restore/tpce/8TB/aws/nodes=10/cpus=8 failed with artifacts on release-23.1 @ 6cce6c746150307a9ecf3b529dfd633a6985c110:

test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1
(test_runner.go:1010).runTest: test timed out (5h0m0s)
(monitor.go:127).Wait: monitor failure: monitor task failed: output in run_075204.139392467_n1_cockroach-sql-insecu: ./cockroach sql --insecure -e "RESTORE  FROM LATEST IN 's3://cockroach-fixtures/backups/tpc-e/customers=500000/v22.2.1/inc-count=48?AUTH=implicit' AS OF SYSTEM TIME '2023-01-05T17:30:00Z' " returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_075204.160753301_n1_cockroach-sql-insecu.log: exit status 137

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #101426 roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed [C-bug C-test-failure GA-blocker O-roachtest O-robot T-kv-replication branch-release-23.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity added this to the 23.1 milestone Apr 6, 2023

blathers-crl bot added the T-disaster-recovery label Apr 6, 2023

cockroach-teamcity mentioned this issue Apr 7, 2023

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100341

Closed

msbutler mentioned this issue Apr 10, 2023

roachtest: prevent aws roachtest panic #100286

Merged

msbutler added GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 12, 2023

cockroach-teamcity mentioned this issue Apr 13, 2023

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #101426

Closed

erikgrinaker mentioned this issue Apr 13, 2023

base: reduce RaftMaxInflightBytes to 32 MB #101437

Merged

livlobo assigned erikgrinaker Apr 13, 2023

craig bot closed this as completed in 16cb080 Apr 13, 2023

This was referenced Apr 13, 2023

release-23.1: base: reduce RaftMaxInflightBytes to 32 MB #101507

Merged

release-23.1.0: base: reduce RaftMaxInflightBytes to 32 MB #101508

Merged

erikgrinaker reopened this Apr 13, 2023

erikgrinaker added the T-kv-replication label Apr 14, 2023

erikgrinaker removed the T-disaster-recovery label Apr 14, 2023

aliher1911 added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Apr 14, 2023

erikgrinaker closed this as completed Apr 18, 2023

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100804

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100804

cockroach-teamcity commented Apr 6, 2023 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Apr 8, 2023

cockroach-teamcity commented Apr 10, 2023

msbutler commented Apr 10, 2023 •

edited

Loading

cockroach-teamcity commented Apr 10, 2023

msbutler commented Apr 12, 2023 •

edited

Loading

msbutler commented Apr 12, 2023

cockroach-teamcity commented Apr 12, 2023

erikgrinaker commented Apr 13, 2023

cockroach-teamcity commented Apr 14, 2023

blathers-crl bot commented Apr 14, 2023

cockroach-teamcity commented Apr 17, 2023

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100804

roachtest: restore/tpce/8TB/aws/nodes=10/cpus=8 failed #100804

Comments

cockroach-teamcity commented Apr 6, 2023 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Apr 8, 2023

cockroach-teamcity commented Apr 10, 2023

msbutler commented Apr 10, 2023 • edited Loading

cockroach-teamcity commented Apr 10, 2023

msbutler commented Apr 12, 2023 • edited Loading

msbutler commented Apr 12, 2023

cockroach-teamcity commented Apr 12, 2023

erikgrinaker commented Apr 13, 2023

cockroach-teamcity commented Apr 14, 2023

blathers-crl bot commented Apr 14, 2023

cockroach-teamcity commented Apr 17, 2023

cockroach-teamcity commented Apr 6, 2023 •

edited by cockroach-jira-scripts

Loading

msbutler commented Apr 10, 2023 •

edited

Loading

msbutler commented Apr 12, 2023 •

edited

Loading