ccl/changefeedccl: TestChangefeedNemeses failed #48459

cockroach-teamcity · 2020-05-06T00:04:15Z

(ccl/changefeedccl).TestChangefeedNemeses failed on master@19ab7176c362177c03756424136a4c8d056be3b4:

=== RUN   TestChangefeedNemeses
--- FAIL: TestChangefeedNemeses (14.61s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestChangefeedNemeses137369527
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestChangefeedNemeses/enterprise
    --- FAIL: TestChangefeedNemeses/enterprise (5.11s)
        nemeses_test.go:38: could not pause job
            github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest.(*jobFeed).Pause.func1
            	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest/testfeed.go:282
            github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts
            	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:159
            github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest.(*jobFeed).Pause
            	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest/testfeed.go:276
            github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest.pause
            	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest/nemeses.go:654
            github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest.logEvent.func1
            	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest/nemeses.go:499
            github.com/cockroachdb/cockroach/pkg/util/fsm.Transitions.apply
            	/go/src/github.com/cockroachdb/cockroach/pkg/util/fsm/fsm.go:106
            github.com/cockroachdb/cockroach/pkg/util/fsm.(*Machine).ApplyWithPayload
            	/go/src/github.com/cockroachdb/cockroach/pkg/util/fsm/fsm.go:135
            github.com/cockroachdb/cockroach/pkg/util/fsm.(*Machine).Apply
            	/go/src/github.com/cockroachdb/cockroach/pkg/util/fsm/fsm.go:129
            github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest.RunNemesis
            	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest/nemeses.go:215
            github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestChangefeedNemeses.func2
            	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/nemeses_test.go:36
            github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.enterpriseTestWithServerArgs.func1
            	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:288
            testing.tRunner
            	/usr/local/go/src/testing/testing.go:909
            runtime.goexit
            	/usr/local/go/src/runtime/asm_amd64.s:1357

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestChangefeedNemeses PKG=./pkg/ccl/changefeedccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

ccl/changefeedccl: TestChangefeedNemeses failed #47153 ccl/changefeedccl: TestChangefeedNemeses failed C-test-failure O-robot branch-release-20.1
ccl/changefeedccl: TestChangefeedNemeses failed under stress #46706 ccl/changefeedccl: TestChangefeedNemeses failed under stress C-test-failure O-robot

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-05-09T06:06:55Z

(ccl/changefeedccl).TestChangefeedNemeses failed on master@1d6db7e386e3ab21bca53f38637014f61153dc90:

      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/schemafeed/schema_feed.go:180 +0xd5
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/schemafeed.(*SchemaFeed).Run-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/schemafeed/schema_feed.go:156 +0x55
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Previous read at 0x00c006983690 by goroutine 147:
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.copyFromSourceToSinkUntilTableEvent.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:329 +0x160
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.copyFromSourceToSinkUntilTableEvent()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:379 +0x37f
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).runUntilTableEvent.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:268 +0x1ba
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Goroutine 379 (running) created at:
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:72 +0x608
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).startKVFeed.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:229 +0x15d
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x162

Goroutine 147 (running) created at:
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).runUntilTableEvent()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:267 +0x367
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:181 +0x12b
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).run-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:172 +0x55
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71
==================
    --- FAIL: TestChangefeedNemeses/enterprise (92.27s)
        testing.go:853: race detected during execution of test

More

Parameters:

TAGS=
GOFLAGS=-race -parallel=2

make stressrace TESTS=TestChangefeedNemeses PKG=./pkg/ccl/changefeedccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

ccl/changefeedccl: TestChangefeedNemeses failed #47153 ccl/changefeedccl: TestChangefeedNemeses failed C-test-failure O-robot branch-release-20.1
ccl/changefeedccl: TestChangefeedNemeses failed under stress #46706 ccl/changefeedccl: TestChangefeedNemeses failed under stress C-test-failure O-robot

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-05-27T06:57:05Z

(ccl/changefeedccl).TestChangefeedNemeses failed on master@d3602e591d2bd079cd2622f24037707f9e8e950e:

  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/schemafeed.(*SchemaFeed).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/schemafeed/schema_feed.go:180 +0xd5
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/schemafeed.(*SchemaFeed).Run-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/schemafeed/schema_feed.go:156 +0x55
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Previous read at 0x00c006354820 by goroutine 323:
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.copyFromSourceToSinkUntilTableEvent.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:329 +0x160
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.copyFromSourceToSinkUntilTableEvent()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:379 +0x37f
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).runUntilTableEvent.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:268 +0x1ba
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71

Goroutine 268 (running) created at:
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:72 +0x608
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).startKVFeed.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:229 +0x15d
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x162

Goroutine 323 (running) created at:
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0xda
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).runUntilTableEvent()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:267 +0x367
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:181 +0x12b
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed.(*kvFeed).run-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed/kv_feed.go:172 +0x55
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4e
  github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x71
==================
FAIL	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl	672.217s

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestChangefeedNemeses PKG=./pkg/ccl/changefeedccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

ccl/changefeedccl: TestChangefeedNemeses failed #47153 ccl/changefeedccl: TestChangefeedNemeses failed C-test-failure O-robot branch-release-20.1
ccl/changefeedccl: TestChangefeedNemeses failed under stress #46706 ccl/changefeedccl: TestChangefeedNemeses failed under stress C-test-failure O-robot

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-22T00:25:51Z

(ccl/changefeedccl).TestChangefeedNemeses failed on master@f8738910e9e95a386f554a75a421cfb60defaab4:

Fatal error:

panic: semaphore release without acquire [recovered]
	panic: semaphore release without acquire

Stack:

goroutine 248228 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc0048ea240, 0x5c22820, 0xc00654dac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:207 +0x11f
panic(0x44cbc60, 0x5b2e440)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/marusama/semaphore.(*semaphore).Release(0xc00538e450, 0x1, 0xc00287f340)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/marusama/semaphore/semaphore.go:170 +0x11d
github.com/cockroachdb/cockroach/pkg/util/limit.(*ConcurrentRequestLimiter).Finish(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/limit/limiter.go:54
github.com/cockroachdb/cockroach/pkg/kv/kvserver.iteratorWithCloser.Close(0x5c790a0, 0xc00287f340, 0xc004c20b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_rangefeed.go:120 +0x3b
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).maybeRunCatchupScan.func1(0x5c79a60, 0xc007e69520, 0xc007167b20, 0x173ff508, 0xed6d256cf, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:290 +0x31
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).maybeRunCatchupScan(0xc007167b20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:411 +0xe22
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc007167b20, 0x5c22820, 0xc00654dc00, 0xc00654dc00, 0xc004c20c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:235 +0x43
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc007167b20, 0x5c22820, 0xc00654dc00)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:271 +0xc2
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1(0x5c22820, 0xc00654dac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:240 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0048ea240, 0x5c22820, 0xc00654dac0, 0xc007ab2c80, 0x1e, 0x0, 0x0, 0xc007e69500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x131

Log preceding fatal error

=== RUN   TestChangefeedNemeses/cloudstorage

More

Parameters:

GOFLAGS=-json

make stressrace TESTS=TestChangefeedNemeses PKG=./pkg/ccl/changefeedccl TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

pbardea · 2020-09-09T22:04:52Z

Looks like #53295 should have fixed the latest failure.

pbardea · 2020-09-15T17:26:17Z

#54273 looks like it should address the race failures.

ajwerner · 2020-09-15T17:33:05Z

I'm going to roachprod stressrace this and see if I can hit a repro in hopes of feeling better about the patch.

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 6, 2020

cockroach-teamcity added this to the 20.1 milestone May 6, 2020

cockroach-teamcity assigned danhhz May 6, 2020

This was referenced Jun 4, 2020

ccl/changefeedccl: TestChangefeedNemeses failed #47153

Closed

ccl/changefeedccl: TestChangefeedNemeses failed under stress #46706

Closed

pbardea unassigned danhhz Sep 9, 2020

pbardea mentioned this issue Sep 10, 2020

schemafeed: fix race when peeking events from schema feed #54181

Closed

pbardea self-assigned this Sep 10, 2020

ajwerner mentioned this issue Sep 11, 2020

changefeedccl/schemafeed: only sort the unsorted tail of descriptors #54273

Merged

pbardea assigned ajwerner and unassigned pbardea Sep 15, 2020

craig bot closed this as completed in 19eb974 Sep 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/changefeedccl: TestChangefeedNemeses failed #48459

ccl/changefeedccl: TestChangefeedNemeses failed #48459

cockroach-teamcity commented May 6, 2020

cockroach-teamcity commented May 9, 2020

cockroach-teamcity commented May 27, 2020

cockroach-teamcity commented Aug 22, 2020

pbardea commented Sep 9, 2020

pbardea commented Sep 15, 2020

ajwerner commented Sep 15, 2020

ccl/changefeedccl: TestChangefeedNemeses failed #48459

ccl/changefeedccl: TestChangefeedNemeses failed #48459

Comments

cockroach-teamcity commented May 6, 2020

cockroach-teamcity commented May 9, 2020

cockroach-teamcity commented May 27, 2020

cockroach-teamcity commented Aug 22, 2020

pbardea commented Sep 9, 2020

pbardea commented Sep 15, 2020

ajwerner commented Sep 15, 2020