Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpccbench/nodes=3/cpu=4 failed [perm denied #72635] #72597

Closed
cockroach-teamcity opened this issue Nov 10, 2021 · 7 comments · Fixed by #72803
Closed

roachtest: tpccbench/nodes=3/cpu=4 failed [perm denied #72635] #72597

cockroach-teamcity opened this issue Nov 10, 2021 · 7 comments · Fixed by #72803
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

Comments

@cockroach-teamcity
Copy link
Member

roachtest.tpccbench/nodes=3/cpu=4 failed with artifacts on master @ 2de17e7fbe66e14039fc7969a76139625761438f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=4/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3705715-1636529870-06-n4cpu4:1-3 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3705715-1636529870-06-n4cpu4:1-3 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1a19940, 0xc000608760, 0x1, 0x2, 0xc00035fa10, 0xc00035fa38)
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1a19940, 0xc000608760, 0x1, 0x2)
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1a19940, 0xc000608740, 0x2, 0x2, 0x1a19940, 0xc000608740)
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1a196c0, 0x0, 0x0, 0xc0005388c0)
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 08:22:14.238357 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

|

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Nov 10, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=3/cpu=4 failed with artifacts on master @ 4236daf8ac1494feab9193058517278c73bbdf27:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=4/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3712075-1636615811-08-n4cpu4:1-3 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3712075-1636615811-08-n4cpu4:1-3 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1a1a940, 0xc0003ffec0, 0x1, 0x2, 0xc0006bfa10, 0xc0006bfa38)
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1a1a940, 0xc0003ffec0, 0x1, 0x2)
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1a1a940, 0xc0003ffea0, 0x2, 0x2, 0x1a1a940, 0xc0003ffea0)
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1a1a6c0, 0x0, 0x0, 0xc00046c700)
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 08:11:57.096461 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

|

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

@tbg tbg changed the title roachtest: tpccbench/nodes=3/cpu=4 failed roachtest: tpccbench/nodes=3/cpu=4 failed [perm denied #72635] Nov 11, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=3/cpu=4 failed with artifacts on master @ d16a755cfa43e10a85e4c9aa9400b5a147b65e69:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=4/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3718971-1636702330-05-n4cpu4:1-3 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3718971-1636702330-05-n4cpu4:1-3 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc0005ca4c0, 0x1, 0x2, 0xc00063fa10, 0xc00063fa38)
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc0005ca4c0, 0x1, 0x2)
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc0005ca4a0, 0x2, 0x2, 0x1acda60, 0xc0005ca4a0)
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc0000e8700)
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 08:14:33.027211 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=3/cpu=4 failed with artifacts on master @ d16a755cfa43e10a85e4c9aa9400b5a147b65e69:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=4/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3721766-1636735213-10-n4cpu4:1-3 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3721766-1636735213-10-n4cpu4:1-3 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc0005e36c0, 0x1, 0x2, 0xc0006bfa10, 0xc0006bfa38)
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc0005e36c0, 0x1, 0x2)
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc0005e36a0, 0x2, 0x2, 0x1acda60, 0xc0005e36a0)
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc0004fe700)
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 17:24:32.388871 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=3/cpu=4 failed with artifacts on master @ f82ff534856738d5385073167d048feafb0b4f3e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=4/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3724941-1636787793-15-n4cpu4:1-3 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3724941-1636787793-15-n4cpu4:1-3 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc0005ddb00, 0x1, 0x2, 0xc000647a10, 0xc000647a38)
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc0005ddb00, 0x1, 0x2)
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc0005ddae0, 0x2, 0x2, 0x1acda60, 0xc0005ddae0)
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc000574700)
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 07:59:09.463756 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=3/cpu=4 failed with artifacts on master @ 13669f9c9bd92a4c3b0378a558d7735f122c4e72:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=4/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3727706-1636874060-04-n4cpu4:1-3 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3727706-1636874060-04-n4cpu4:1-3 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc0005e6340, 0x1, 0x2, 0xc00035fa10, 0xc00035fa38)
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc0005e6340, 0x1, 0x2)
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc0005e6320, 0x2, 0x2, 0x1acda60, 0xc0005e6320)
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc0004fe700)
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 07:56:11.336870 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=3/cpu=4 failed with artifacts on master @ 3aeb3756887fcb35dcd19c0cee6894a143228727:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=4/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3730576-1636963510-14-n4cpu4:1-3 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3730576-1636963510-14-n4cpu4:1-3 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc0005da320, 0x1, 0x2, 0xc00035fa10, 0xc00035fa38)
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc0005da320, 0x1, 0x2)
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc0005da300, 0x2, 0x2, 0x1acda60, 0xc0005da300)
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc0005388c0)
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 08:49:03.017698 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=3/cpu=4 failed with artifacts on master @ e921036c0640d833548363cd8f3fea78ae534bd1:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=4/run_1
	cluster.go:1857,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3738783-1637046944-11-n4cpu4:1-3 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false teamcity-3738783-1637046944-11-n4cpu4:1-3 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acca60, 0xc0005fd0a0, 0x1, 0x2, 0xc0006bfa10, 0xc0006bfa38)
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:461 +0xdf
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !main.wrap.func1(0x1acca60, 0xc0005fd0a0, 0x1, 0x2)
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acca60, 0xc0005fd080, 0x2, 0x2, 0x1acca60, 0xc0005fd080)
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acc7e0, 0x0, 0x0, 0xc00046e700)
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !main.main()
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1174 +0x26a5
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !****************************************************************************
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !process is terminating.
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !improve CockroachDB based on your report.
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !    [email protected]
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 07:57:57.779950 1 roachprod/install/cluster_synced.go:1779  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@AlexTalks AlexTalks linked a pull request Nov 16, 2021 that will close this issue
roachprod,roachtest: fix mounting on GCP and disable RAID0 in roachtests #72803
Merged
@craig craig bot closed this as completed in #72803 Nov 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachprod,roachtest: fix mounting on GCP and disable RAID0 in roachtests AlexTalks/cockroach
1 participant
@cockroach-teamcity