Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestApplier failed #53183

Closed
cockroach-teamcity opened this issue Aug 21, 2020 · 5 comments · Fixed by #53295
Closed

kv/kvnemesis: TestApplier failed #53183

cockroach-teamcity opened this issue Aug 21, 2020 · 5 comments · Fixed by #53295
Assignees
@ajwerner
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
20.2

Comments

@cockroach-teamcity
Copy link
Member

(kv/kvnemesis).TestApplier failed on master@991a05367d38ca32c1f6584e56143ac65270afc8:

  github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:986 +0x78
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:595 +0xf8

Previous write at 0x00c0015341d0 by goroutine 235:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:214 +0x207
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).maybeRunCatchupScan()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:286 +0xa1
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:235 +0x53
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:271 +0x122
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:240 +0x7e
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 295 (running) created at:
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:593 +0x134
  github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).RangeFeed()
      <autogenerated>:1 +0xd9
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:257 +0x5b1
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:150 +0x3d6
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:69 +0x7c
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Goroutine 235 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:246 +0x15f6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:177 +0x67
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162
==================
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestApplier657779867
--- FAIL: TestApplier (1.15s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestApplier657779867
    test_log_scope.go:63: use -show-logs to present logs inline
    testing.go:853: race detected during execution of test
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestApplier PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 21, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Aug 21, 2020
@nvanbenschoten
Copy link
Member

This looks like fallout from 5cbf775#diff-48566323d33f658e71fbcf1d3fb00c2dR217. @ajwerner do you mind taking a look when you get a chance?

@nvanbenschoten nvanbenschoten assigned ajwerner and unassigned danhhz Aug 22, 2020
@nvanbenschoten
Copy link
Member

Sorry, meant to add the race output:

[06:34:27]	[Step 2/2] WARNING: DATA RACE
[06:34:27]	[Step 2/2] Read at 0x00c0015341d0 by goroutine 295:
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed.func1()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:177 +0x38
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:228 +0x6ec
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:2434 +0x4f3
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:202 +0x20e
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:986 +0x78
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:595 +0xf8
[06:34:27]	[Step 2/2] 
[06:34:27]	[Step 2/2] Previous write at 0x00c0015341d0 by goroutine 235:
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed.func2()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:214 +0x207
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).maybeRunCatchupScan()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:286 +0xa1
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:235 +0x53
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:271 +0x122
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:240 +0x7e
[06:34:27]	[Step 2/2]   github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
[06:34:27]	[Step 2/2]       /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestApplier failed on master@7425e857e62fe4280f614f9076f310322cc78649:

  github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:986 +0x78
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:595 +0xf8

Previous write at 0x00c00040c7c8 by goroutine 267:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:214 +0x207
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).maybeRunCatchupScan()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:286 +0xa1
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:235 +0x53
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:271 +0x122
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:240 +0x7e
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 80 (running) created at:
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:593 +0x134
  github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).RangeFeed()
      <autogenerated>:1 +0xd9
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:257 +0x5b1
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:150 +0x3d6
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:69 +0x7c
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Goroutine 267 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:246 +0x15f6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:177 +0x67
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162
==================
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestApplier408198732
--- FAIL: TestApplier (1.16s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestApplier408198732
    test_log_scope.go:63: use -show-logs to present logs inline
    testing.go:853: race detected during execution of test
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestApplier PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@ajwerner
Copy link
Contributor

Looking now

@ajwerner ajwerner mentioned this issue Aug 23, 2020
Merged
ajwerner added a commit to ajwerner/cockroach that referenced this issue Aug 23, 2020
@ajwerner
kvserver: fix race in rangefeed processor setup
9f64083 
Fixes cockroachdb#53183

Release note: None
ajwerner added a commit to ajwerner/cockroach that referenced this issue Aug 24, 2020
@ajwerner
kvserver: fix race in rangefeed processor setup
4ba37cc 
Fixes cockroachdb#53183

Release note: None
@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestApplier failed on master@b0ded83998c7b3b71d433dd93ad35a0fed561d3b:

  github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:986 +0x78
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:595 +0xf8

Previous write at 0x00c000ef2040 by goroutine 102:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:214 +0x207
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).maybeRunCatchupScan()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:286 +0xa1
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:235 +0x53
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:271 +0x122
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:240 +0x7e
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162

Goroutine 221 (running) created at:
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:593 +0x134
  github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).RangeFeed()
      <autogenerated>:1 +0xd9
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:257 +0x5b1
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:150 +0x3d6
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:69 +0x7c
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Goroutine 102 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14a
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:246 +0x15f6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:177 +0x67
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x162
==================
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestApplier312852580
--- FAIL: TestApplier (0.94s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestApplier312852580
    test_log_scope.go:63: use -show-logs to present logs inline
    testing.go:853: race detected during execution of test
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestApplier PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

craig bot pushed a commit that referenced this issue Aug 24, 2020
@knz @ajwerner
Merge #53263 #53295
620f0a4 
53263: util/logs: request redactable logs by default  r=irfansharif a=knz

First two commits from  #53313

This changes the default value for the `--redactable-logs` flag to
`true`. This makes server produce redactable logs by default.

Note that `--redactable-logs` only triggers redactable logs if the
caller calls `SetupRedactionAndStderrRedirects()`. This is done by
e.g. the server code. This means that even with this patch,
*unit tests* continue to produce non-redactable logs.

This commit also adds telemetry for the enablement of redactable
markers, as `server.logging.redactable_logs.{enabled,disabled}`.

Fixes #51834.


Release note (cli change): The `cockroach start`, `start-single-node`
and `demo` command now enable `--redactable-logs` by default.
This causes log files to become redactable, so that 
`cockroach debug merge-log --redact` or `cockroach debug zip --redact`
can remove sensitive information out of log files.
(Reminder: `cockroach debug zip --redact` only affects *log files*;
other items collected by the command can still contain sensitive information)

53295: kvserver: fix race in rangefeed processor setup r=nvanbenschoten a=ajwerner

Fixes #53183

Release note: None

Co-authored-by: Raphael 'kena' Poss <[email protected]>
Co-authored-by: Andrew Werner <[email protected]>
@craig craig bot closed this as completed in 8898880 Aug 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajwerner ajwerner

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
20.2
Development

Successfully merging a pull request may close this issue.

kvserver: fix race in rangefeed processor setup ajwerner/cockroach
4 participants
@danhhz @ajwerner @nvanbenschoten @cockroach-teamcity