Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed [perm denied #72635] #72631

Closed
cockroach-teamcity opened this issue Nov 11, 2021 · 7 comments · Fixed by #72803
Closed

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed [perm denied #72635] #72631

cockroach-teamcity opened this issue Nov 11, 2021 · 7 comments · Fixed by #72803
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

roachtest.tpccbench/nodes=9/cpu=4/chaos/partition failed with artifacts on master @ 2de17e7fbe66e14039fc7969a76139625761438f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3705715-1636529870-84-n10cpu4:1-9 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3705715-1636529870-84-n10cpu4:1-9 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1a19940, 0xc000404150, 0x1, 0x3, 0xc000647a10, 0xc000647a38)
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1a19940, 0xc000404150, 0x1, 0x3)
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1a19940, 0xc000404120, 0x3, 0x3, 0x1a19940, 0xc000404120)
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1a196c0, 0x0, 0x0, 0xc00046c700)
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211110 14:09:10.924052 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

|

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Nov 11, 2021
@tbg tbg changed the title roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed [perm denied error] Nov 11, 2021
@tbg tbg changed the title roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed [perm denied error] roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed [perm denied #72635] Nov 11, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=9/cpu=4/chaos/partition failed with artifacts on master @ 4236daf8ac1494feab9193058517278c73bbdf27:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3712075-1636615811-87-n10cpu4:1-9 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3712075-1636615811-87-n10cpu4:1-9 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1a1a940, 0xc0000aa120, 0x1, 0x3, 0xc0006bfa10, 0xc0006bfa38)
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1a1a940, 0xc0000aa120, 0x1, 0x3)
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1a1a940, 0xc0000aa0f0, 0x3, 0x3, 0x1a1a940, 0xc0000aa0f0)
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1a1a6c0, 0x0, 0x0, 0xc0000e8700)
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211111 13:32:18.644066 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

|

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=9/cpu=4/chaos/partition failed with artifacts on master @ d16a755cfa43e10a85e4c9aa9400b5a147b65e69:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3718971-1636702330-85-n10cpu4:1-9 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3718971-1636702330-85-n10cpu4:1-9 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc0004bd7a0, 0x1, 0x3, 0xc00035fa10, 0xc00035fa38)
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc0004bd7a0, 0x1, 0x3)
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc0004bd770, 0x3, 0x3, 0x1acda60, 0xc0004bd770)
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc0004d6700)
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 13:36:43.118721 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=9/cpu=4/chaos/partition failed with artifacts on master @ d16a755cfa43e10a85e4c9aa9400b5a147b65e69:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3721766-1636735213-79-n10cpu4:1-9 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3721766-1636735213-79-n10cpu4:1-9 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc00053d7a0, 0x1, 0x3, 0xc00035fa10, 0xc00035fa38)
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc00053d7a0, 0x1, 0x3)
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc00053d770, 0x3, 0x3, 0x1acda60, 0xc00053d770)
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc000556700)
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211112 23:22:39.736533 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=9/cpu=4/chaos/partition failed with artifacts on master @ f82ff534856738d5385073167d048feafb0b4f3e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3724941-1636787793-22-n10cpu4:1-9 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3724941-1636787793-22-n10cpu4:1-9 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc000404150, 0x1, 0x3, 0xc0006bfa10, 0xc0006bfa38)
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc000404150, 0x1, 0x3)
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc000404120, 0x3, 0x3, 0x1acda60, 0xc000404120)
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc000584700)
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211113 10:29:01.179570 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=9/cpu=4/chaos/partition failed with artifacts on master @ 13669f9c9bd92a4c3b0378a558d7735f122c4e72:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3727706-1636874060-22-n10cpu4:1-9 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3727706-1636874060-22-n10cpu4:1-9 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc000129770, 0x1, 0x3, 0xc00035fa10, 0xc00035fa38)
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc000129770, 0x1, 0x3)
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc000129740, 0x3, 0x3, 0x1acda60, 0xc000129740)
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc00017c700)
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211114 10:43:17.786071 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=9/cpu=4/chaos/partition failed with artifacts on master @ 3aeb3756887fcb35dcd19c0cee6894a143228727:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1856,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3730576-1636963510-22-n10cpu4:1-9 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3730576-1636963510-22-n10cpu4:1-9 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acda60, 0xc000120120, 0x1, 0x3, 0xc0006bfa10, 0xc0006bfa38)
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:457 +0xdf
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.wrap.func1(0x1acda60, 0xc000120120, 0x1, 0x3)
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acda60, 0xc0001200f0, 0x3, 0x3, 0x1acda60, 0xc0001200f0)
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acd7e0, 0x0, 0x0, 0xc000174700)
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !main.main()
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170 +0x26a5
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !****************************************************************************
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !process is terminating.
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !improve CockroachDB based on your report.
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !    [email protected]
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !
		  | F211115 11:15:39.933116 1 roachprod/install/cluster_synced.go:1777  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpccbench/nodes=9/cpu=4/chaos/partition failed with artifacts on master @ e921036c0640d833548363cd8f3fea78ae534bd1:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1857,tpcc.go:1125,tpcc.go:1135,search.go:43,search.go:173,tpcc.go:1131,tpcc.go:905,test_runner.go:777: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3738783-1637046944-22-n10cpu4:1-9 returned: exit status 7
		(1) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --encrypt=false --racks=3 teamcity-3738783-1637046944-22-n10cpu4:1-9 returned
		  | stderr:
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | nc9(0x1acca60, 0xc00053b7a0, 0x1, 0x3, 0xc00035fa10, 0xc00035fa38)
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:461 +0xdf
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !main.wrap.func1(0x1acca60, 0xc00053b7a0, 0x1, 0x3)
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123 +0x6b
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !github.com/spf13/cobra.(*Command).execute(0x1acca60, 0xc00053b770, 0x3, 0x3, 0x1acca60, 0xc00053b770)
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !github.com/spf13/cobra.(*Command).ExecuteC(0x1acc7e0, 0x0, 0x0, 0xc000554700)
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !github.com/spf13/cobra.(*Command).Execute(...)
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !main.main()
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1174 +0x26a5
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !****************************************************************************
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !This node experienced a fatal error (printed above), and as a result the
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !process is terminating.
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !problem in CockroachDB. With your help, the support team at Cockroach Labs
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !will try to determine the root cause, recommend next steps, and we can
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !improve CockroachDB based on your report.
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !Please submit a crash report by following the instructions here:
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !    https://github.com/cockroachdb/cockroach/issues/new/choose
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !If you would rather not post publicly, please contact us directly at:
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !    [email protected]
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !
		  | F211116 10:29:10.590711 1 roachprod/install/cluster_synced.go:1779  [-] 2 !The Cockroach Labs team appreciates your feedback.
		Wraps: (2) exit status 7
		Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 17, 2021
Closed
@AlexTalks AlexTalks removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 17, 2021
@AlexTalks AlexTalks linked a pull request Nov 17, 2021 that will close this issue
roachprod,roachtest: fix mounting on GCP and disable RAID0 in roachtests #72803
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachprod,roachtest: fix mounting on GCP and disable RAID0 in roachtests AlexTalks/cockroach
2 participants
@AlexTalks @cockroach-teamcity