Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #49167

Closed
cockroach-teamcity opened this issue May 17, 2020 · 4 comments
Closed

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #49167

cockroach-teamcity opened this issue May 17, 2020 · 4 comments
Assignees
@itsbilal
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone
20.1

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@2cbf620cd229a55622fe17fb15d20ada1dbcccd3:

		  |   | *
		  |   | *
		  |   | * ERROR: ERROR: connection lost.
		  |   | * 
		  |   | * cockroach server exited with error: failed to create engines: unexpected EOF
		  |   | *
		  |   | ERROR: connection lost.
		  |   |
		  |   | cockroach server exited with error: failed to create engines: unexpected EOF
		  |   | Failed running "start"
		  |   | E200517 12:27:52.551925 1 cli/error.go:373  ERROR: exit status 1
		  |   | ERROR: exit status 1
		  |   | Failed running "start"
		  | Wraps: (4) exit status 1
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *exec.ExitError: 
		  | I200517 12:27:52.573465 1 cluster_synced.go:1749  command failed
		Wraps: (9) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1948994-1589701075-30-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		7: dead
		3: 15314
		5: 14606
		8: 13980
		4: 13400
		6: 14502
		9: 13766
		1: 14421
		2: 13487
		Error: UNCLASSIFIED_PROBLEM: 7: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 7: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errors.errorString
More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels May 17, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone May 17, 2020
@andreimatei
Copy link
Contributor

@itsbilal mind taking a look at this one? The test claimed it can't restart node 7, and node 7 says

E200517 12:27:52.542162 1 cli/error.go:373  ERROR: connection lost.

cockroach server exited with error: failed to create engines: unexpected EOF

@andreimatei andreimatei assigned itsbilal and unassigned andreimatei May 18, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@1520ad2ba7c926f8043de8b6e044ab35c2f67b13:

		  |   | *
		  |   | *
		  |   | * ERROR: ERROR: connection lost.
		  |   | * 
		  |   | * cockroach server exited with error: failed to create engines: unexpected EOF
		  |   | *
		  |   | ERROR: connection lost.
		  |   |
		  |   | cockroach server exited with error: failed to create engines: unexpected EOF
		  |   | Failed running "start"
		  |   | E200521 13:00:57.965157 1 cli/error.go:373  ERROR: exit status 1
		  |   | ERROR: exit status 1
		  |   | Failed running "start"
		  | Wraps: (4) exit status 1
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *exec.ExitError: 
		  | I200521 13:00:57.987826 1 cluster_synced.go:1749  command failed
		Wraps: (9) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1957823-1590046508-30-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		4: dead
		1: 20135
		3: 19005
		2: 17862
		8: 18825
		9: 17540
		6: 17815
		5: 17641
		7: 16795
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1799
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 4: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errors.errorString
More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

This was referenced May 22, 2020
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@a51c5e3d03497b84a74ef61c7658326c31615a93:

		  |   | *
		  |   | * ERROR: ERROR: connection lost.
		  |   | * 
		  |   | * cockroach server exited with error: failed to create engines: unexpected EOF
		  |   | *
		  |   | ERROR: connection lost.
		  |   |
		  |   | cockroach server exited with error: failed to create engines: unexpected EOF
		  |   | Failed running "start"
		  |   | E200603 12:18:05.827086 1 cli/error.go:374  ERROR: exit status 1
		  |   | ERROR: exit status 1
		  |   | Failed running "start"
		  | Wraps: (4) exit status 1
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *exec.ExitError: 
		  | I200603 12:18:05.850048 1 cluster_synced.go:1749  command failed
		Wraps: (9) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1512,context.go:135,cluster.go:1501,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1986174-1591168901-29-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		4: dead
		6: 22108
		7: 20122
		2: 20738
		1: 20938
		9: 21017
		5: 21177
		3: 20840
		8: 21150
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 4: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString
More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@knz
Copy link
Contributor

knz commented Jun 4, 2020

dup #49747 see my comment in #49747 (comment)

@knz knz closed this as completed Jun 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itsbilal itsbilal

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
20.1
Development

No branches or pull requests
4 participants
@itsbilal @andreimatei @knz @cockroach-teamcity