Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: acceptance/bank/cluster-recovery failed [skipped] #53829

Closed
tbg opened this issue Sep 2, 2020 · 3 comments
Closed

roachtest: acceptance/bank/cluster-recovery failed [skipped] #53829

tbg opened this issue Sep 2, 2020 · 3 comments
Assignees
@knz
Labels
A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered).

Comments

@tbg
Copy link
Member

tbg commented Sep 2, 2020 

[Run local roachtests] local: stopping and waiting06:25:41 test.go:337: test failure: 	bank.go:354,bank.go:548,acceptance.go:91,test_runner.go:754: pq: cannot up-replicate to s2; missing gossiped StoreDescriptor
[06:25:41]
[Run local roachtests] 06:25:41 bank.go:163: client 3 shutting down
[06:25:41]
[Run local roachtests] 06:25:41 bank.go:163: client 1 shutting down
[06:25:41]
[Run local roachtests] 06:25:41 test.go:337: [not the first failure] test failure: 
[06:25:41]
[Run local roachtests] 	cluster.go:2163,cluster.go:2094,bank.go:240: /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3 returned: context canceled
[06:25:41]
[Run local roachtests] 		(1) /go/src/github.com/cockroachdb/cockroach/bin/roachprod stop local:3 returned
[06:25:41]
[Run local roachtests] 		  | stderr:
[06:25:41]
[Run local roachtests] 		  |
[06:25:41]
[Run local roachtests] 		  | stdout:
[06:25:41]
[Run local roachtests] 		  | local: stopping and waiting
[06:25:41]
[Run local roachtests] 		Wraps: (2) secondary error attachment
[06:25:41]
[Run local roachtests] 		  | signal: killed
[06:25:41]
[Run local roachtests] 		  | (1) signal: killed
[06:25:41]
[Run local roachtests] 		  | Error types: (1) *exec.ExitError
[06:25:41]
[Run local roachtests] 		Wraps: (3) context canceled
[06:25:41]
[Run local roachtests] 		Error types: (1) *main.withCommandDetails (2) *secondary.withSecondaryError (3) *errors.errorString
[06:25:41]
[Run local roachtests] 06:25:41 bank.go:168: client 2 shutting down
[06:25:41]
[Run local roachtests] panic: t.Fatal() was called
[06:25:41]
[Run local roachtests] 
[06:25:41]
[Run local roachtests] goroutine 23199 [running]:
[06:25:41]
[Run local roachtests] main.(*test).fatalfInner(0xc0003387e0, 0x0, 0x0, 0xc000529510, 0x1, 0x1)
[06:25:41]
[Run local roachtests] 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test.go:291 +0xcd
[06:25:41]
[Run local roachtests] main.(*test).Fatal(0xc0003387e0, 0xc000529510, 0x1, 0x1)
[06:25:41]
[Run local roachtests] 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test.go:266 +0x51
[06:25:41]
[Run local roachtests] main.(*cluster).Stop(0xc000c8c1e0, 0x526cb20, 0xc00137c420, 0xc00120fd90, 0x1, 0x1)
[06:25:41]
[Run local roachtests] 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2163 +0x105
[06:25:41]
[Run local roachtests] main.(*cluster).Restart(0xc000c8c1e0, 0x526cb20, 0xc00137c420, 0xc0003387e0, 0xc00050ffc8, 0x1, 0x1)
[06:25:41]
[Run local roachtests] 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2094 +0xed
[06:25:41]
[Run local roachtests] main.(*bankState).startChaosMonkey.func1(0xc00137e3c0, 0xc000c8c1e0, 0x526caa0, 0xc000b22040, 0xc0003387e0, 0xc000902fa0)
[06:25:41]
[Run local roachtests] 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/bank.go:240 +0x4b6
[06:25:41]
[Run local roachtests] created by main.(*bankState).startChaosMonkey
[06:25:41]
[Run local roachtests] 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/bank.go:210 +0x92
[06:25:41]
[Run local roachtests]
@tbg tbg added A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). labels Sep 2, 2020
@tbg tbg self-assigned this Sep 2, 2020
@tbg tbg changed the title roachtest: acceptance/zerosum-restart failed roachtest: acceptance/bank/cluster-recovery failed Sep 2, 2020
@tbg tbg mentioned this issue Sep 2, 2020
Merged
@tbg
Copy link
Member Author

tbg commented Sep 2, 2020

PS I believe this affects all acceptance/bank tests potentially.

@tbg tbg assigned knz and unassigned tbg Sep 2, 2020
craig bot pushed a commit that referenced this issue Sep 3, 2020
@tbg
Merge #53833
91c7ae2 
53833: roachtest: re-skip two bank tests r=tbg a=tbg

- Revert "roachtest: re-enable acceptance/bank/zerosum-splits"
- Revert "roachtest: re-enable acceptance/bank/zerosum-restart"

They hit flakes: #53829

In particular, there seems to be a fatal on an untracked goroutine, which causes roachtest to terminate prematurely.

Release justification: testing
Release note: None

Co-authored-by: Tobias Grieger <[email protected]>
@tbg tbg changed the title roachtest: acceptance/bank/cluster-recovery failed roachtest: acceptance/bank/cluster-recovery failed [skipped] Sep 8, 2020
@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 21, 2020
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 2, 2020
Closed
@sumeerbhola
Copy link
Collaborator

Failed again https://teamcity.cockroachdb.com/viewLog.html?buildId=2525179&tab=buildResultsDiv&buildTypeId=Cockroach_UnitTests_Roachtest

@tbg
Copy link
Member Author

tbg commented Jan 11, 2021

This last one looks more like #57798 (comment). Tracking in #57342

@tbg tbg closed this as completed Jan 11, 2021
@andreimatei andreimatei mentioned this issue Mar 29, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@knz knz

Labels
A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered).
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@knz @tbg @sumeerbhola