Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server: TestBootstrapNewStore failed #64262

Closed
cockroach-teamcity opened this issue Apr 27, 2021 · 3 comments · Fixed by #76825
Closed

server: TestBootstrapNewStore failed #64262

cockroach-teamcity opened this issue Apr 27, 2021 · 3 comments · Fixed by #76825
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
21.1

Comments

@cockroach-teamcity
Copy link
Member

server.TestBootstrapNewStore failed with artifacts on master @ 588d9d7ad7618784ee23a5e5f440a9d9d36bb0d5:

Fatal error:

panic: unexpected Stopped processor

Stack:

goroutine 127270 [running]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).registerWithRangefeedRaftMuLocked(0xc00709a400, 0x52d1ae0, 0xc008940d50, 0xc0049e6cd0, 0x1, 0x8, 0xc0049e6cd8, 0x1, 0x1, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:385 +0x745
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(0xc00709a400, 0xc0070c4180, 0x532b040, 0xc00165c2a0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:232 +0x5b7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc00430d000, 0xc0070c4180, 0x532b040, 0xc00165c2a0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:2502 +0x11e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc0009930a0, 0xc0070c4180, 0x532b040, 0xc00165c2a0, 0xc00161cf68)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:216 +0xce
github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0xc0033f1080, 0xc0070c4180, 0x532b040, 0xc00165c2a0, 0xc00161cf68, 0x1a8973a)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:1011 +0x54
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1(0xc006afbf30, 0x53177e0, 0xc0033f1080, 0xc0070c4180, 0x52d1a20, 0xc00311b4c0, 0xc00220ade0, 0xc00220ae40)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:573 +0xc5
created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:571 +0x11e
Log preceding fatal error 

=== RUN   TestBootstrapNewStore
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBootstrapNewStore537493561
    test_log_scope.go:74: use -show-logs to present logs inline

Reproduce

To reproduce, try: 

make stressrace TESTS=TestBootstrapNewStore PKG=./pkg/server TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-parallel=4

/cc @spencerkimball

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 27, 2021
@cockroach-teamcity cockroach-teamcity added this to the 21.1 milestone Apr 27, 2021
@cockroach-teamcity
Copy link
Member Author

server.TestBootstrapNewStore failed with artifacts on master @ 6ff6c77e7aebec2edc947c1ed99a1f7cb67e1a3c:

Fatal error:

panic: unexpected Stopped processor

Stack:

goroutine 13226195 [running]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).registerWithRangefeedRaftMuLocked(0xc005bf6a80, {0x4e0d198, 0xc018ca1b30}, {{0xc001a9e818, 0x1, 0x8}, {0xc001a9e820, 0x1, 0x8}}, {0x16c89d4855b9d441, ...}, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:414 +0x798
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc005bf6a80, 0x10, 0xc013914c60, {0x4e789a0, 0xc01721bae0})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:237 +0x635
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:145
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc0139c5200, 0xc013914c60, {0x4e789a0, 0xc01721bae0})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:2979 +0xdc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0x0, 0xc013914c60, {0x4e789a0, 0xc01721bae0})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:216 +0xe8
github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x4e789a0, 0x0, {0x4e789a0, 0xc01721bae0})
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:1157 +0x38
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:694 +0xbb
created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:692 +0x18f
Log preceding fatal error 

=== RUN   TestBootstrapNewStore
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBootstrapNewStore889239033
    test_log_scope.go:80: use -show-logs to present logs inline

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=deadlock

  • GOFLAGS=-parallel=4

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

server.TestBootstrapNewStore failed with artifacts on master @ a5158c4c93e9dca858b38ef7e94321f0a0a2b5c5:

Fatal error:

panic: close of closed channel [recovered]
	panic: close of closed channel [recovered]
	panic: close of closed channel

Stack:

goroutine 15654111 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00e301d40, {0x5032c58, 0xc017893650})
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:251 +0x94
panic({0x39a9260, 0x4f62390})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00e301d40, {0x5032c58, 0xc017893980})
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:251 +0x94
panic({0x39a9260, 0x4f62390})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/cockroachdb/pebble/sstable.(*writeQueue).finish(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/sstable/write_queue.go:108
github.com/cockroachdb/pebble/sstable.(*Writer).Close(0xc00370e000)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/sstable/writer.go:1047 +0xc7
github.com/cockroachdb/cockroach/pkg/storage.(*SSTWriter).Close(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/sst_writer.go:296
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*multiSSTWriter).Close(0xc005c0b028)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_snapshot.go:221 +0x2a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, {_, _}, {{0x15, 0x4, 0xc01062a540, 0xc013c58960, 0xc012002560, ...}, ...})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_snapshot.go:272 +0x6c5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc00bc60000, {0x5032c58, 0xc017893980}, 0xc00db01680, {0x7f2c83996598, 0xc00bc50090})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_snapshot.go:679 +0x5c5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x5032c58, 0xc017893980})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:82 +0x145
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00e301d40, {0x5032c58, 0xc017893980}, {0x3d59900, 0xc017893680}, 0xc003754c48)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:344 +0xdd
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc00bc60000, {0x5032c58, 0xc017893650}, 0xc00db01680, {0x7f2c82ede968, 0xc00bc50090})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:72 +0xcc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot.func1.1({0x5095ab0, 0xc00bc50090}, 0xc0166dec80, {0x5032c58, 0xc017893650})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/raft_transport.go:434 +0x13a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot.func1({0x5032c58, 0xc017893650})
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/raft_transport.go:435 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494 +0x16f
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:483 +0x445
Log preceding fatal error 

=== RUN   TestBootstrapNewStore
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBootstrapNewStore868387766
    test_log_scope.go:80: use -show-logs to present logs inline

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=deadlock

  • GOFLAGS=-parallel=4

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

server.TestBootstrapNewStore failed with artifacts on master @ 8d0a1773a6b55f06c04ea77e6a39426e32ba9a63:

=== RUN   TestBootstrapNewStore
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/ab5462d2cb3989e2a33018db970f23de/logTestBootstrapNewStore210669690
    test_log_scope.go:80: use -show-logs to present logs inline
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

This test on roachdash | Improve this report!

@erikgrinaker erikgrinaker mentioned this issue Feb 20, 2022
Merged
@craig craig bot closed this as completed in 4f3e5e0 Feb 20, 2022
This was referenced Feb 20, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
21.1
Development

Successfully merging a pull request may close this issue.

rangefeed: fix panic due to rangefeed stopper race erikgrinaker/cockroach
1 participant
@cockroach-teamcity