Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

teamcity: failed tests on master: testrace/TestRaftSSTableSideloadingUpdatedReplicaID, test/TestRaftSSTableSideloadingUpdatedReplicaID #27232

Closed
cockroach-teamcity opened this issue Jul 6, 2018 · 0 comments · Fixed by #27699
Assignees
Labels
A-kv-client Relating to the KV client and the KV interface. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#762920:

--- FAIL: testrace/TestRaftSSTableSideloadingUpdatedReplicaID (0.000s)
Race detected!

------- Stdout: -------
I180706 21:01:03.982150 11752 storage/engine/rocksdb.go:566  opening rocksdb instance at "/go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID265162223"
I180706 21:01:04.059144 11752 storage/replica_sideload_test.go:1040  olddir is /go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID265162223/auxiliary/sideloading/1/1.1, newdir is /go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID265162223/auxiliary/sideloading/1/1.2
I180706 21:01:04.059356 11752 storage/engine/rocksdb.go:699  closing rocksdb instance at "/go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID265162223"
I180706 21:01:04.060610 11752 util/stop/stopper.go:471  quiescing; tasks left:
1      [async] gossip.infoStore: callback
==================
Read at 0x00c420572560 by goroutine 43:
  github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).Iterate()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:798 +0x57
  github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).examineQueue()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:421 +0x234
  github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:131 +0x3f9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14a

Previous write at 0x00c420572560 by goroutine 32:
  github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).Close()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:712 +0x297
  github.com/cockroachdb/cockroach/pkg/storage.TestRaftSSTableSideloadingUpdatedReplicaID()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_sideload_test.go:1048 +0x7ef
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:777 +0x16d

Goroutine 43 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xba
  github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:110 +0x13c
  github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1460 +0xa05
  github.com/cockroachdb/cockroach/pkg/storage.(*testContext).StartWithStoreConfig()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:208 +0x1333
  github.com/cockroachdb/cockroach/pkg/storage.(*testContext).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:158 +0x29d
  github.com/cockroachdb/cockroach/pkg/storage.TestRaftSSTableSideloadingUpdatedReplicaID()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_sideload_test.go:1004 +0x1f7
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:777 +0x16d

Goroutine 32 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:824 +0x564
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1063 +0xa4
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:777 +0x16d
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1061 +0x4e1
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:978 +0x2cd
  github.com/cockroachdb/cockroach/pkg/storage_test.TestMain()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x2a8
  main.main()
      _testmain.go:1062 +0x22a
==================
--- FAIL: test/TestRaftSSTableSideloadingUpdatedReplicaID (0.060s)

------- Stdout: -------
I180706 20:49:26.935853 12032 storage/engine/rocksdb.go:566  opening rocksdb instance at "/go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID081132010"
I180706 20:49:26.972385 12032 storage/store.go:1449  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180706 20:49:26.985020 12032 storage/replica_sideload_test.go:1040  olddir is /go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID081132010/auxiliary/sideloading/1/1.1, newdir is /go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID081132010/auxiliary/sideloading/1/1.2
I180706 20:49:26.985063 12032 storage/engine/rocksdb.go:699  closing rocksdb instance at "/go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID081132010"Race detected!

------- Stdout: -------
I180706 21:01:03.982150 11752 storage/engine/rocksdb.go:566  opening rocksdb instance at "/go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID265162223"
I180706 21:01:04.059144 11752 storage/replica_sideload_test.go:1040  olddir is /go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID265162223/auxiliary/sideloading/1/1.1, newdir is /go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID265162223/auxiliary/sideloading/1/1.2
I180706 21:01:04.059356 11752 storage/engine/rocksdb.go:699  closing rocksdb instance at "/go/src/github.com/cockroachdb/cockroach/artifacts/TestRaftSSTableSideloadingUpdatedReplicaID265162223"
I180706 21:01:04.060610 11752 util/stop/stopper.go:471  quiescing; tasks left:
1      [async] gossip.infoStore: callback
==================
Read at 0x00c420572560 by goroutine 43:
  github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).Iterate()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:798 +0x57
  github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).examineQueue()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:421 +0x234
  github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:131 +0x3f9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14a

Previous write at 0x00c420572560 by goroutine 32:
  github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).Close()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:712 +0x297
  github.com/cockroachdb/cockroach/pkg/storage.TestRaftSSTableSideloadingUpdatedReplicaID()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_sideload_test.go:1048 +0x7ef
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:777 +0x16d

Goroutine 43 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xba
  github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:110 +0x13c
  github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1460 +0xa05
  github.com/cockroachdb/cockroach/pkg/storage.(*testContext).StartWithStoreConfig()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:208 +0x1333
  github.com/cockroachdb/cockroach/pkg/storage.(*testContext).Start()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:158 +0x29d
  github.com/cockroachdb/cockroach/pkg/storage.TestRaftSSTableSideloadingUpdatedReplicaID()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_sideload_test.go:1004 +0x1f7
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:777 +0x16d

Goroutine 32 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:824 +0x564
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1063 +0xa4
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:777 +0x16d
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1061 +0x4e1
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:978 +0x2cd
  github.com/cockroachdb/cockroach/pkg/storage_test.TestMain()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x2a8
  main.main()
      _testmain.go:1062 +0x22a
==================

Please assign, take a look and update the issue accordingly.

@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone Jul 6, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jul 6, 2018
@tbg tbg added the A-kv-client Relating to the KV client and the KV interface. label Jul 10, 2018
@tbg tbg self-assigned this Jul 17, 2018
tbg added a commit to tbg/cockroach that referenced this issue Jul 18, 2018
Starting workers without a surrounding task is unfortunately often not
the right thing to do when the worker accesses other state that might
become invalidated once the stopper begins to stop. In this particular
case, the compactor might end up accessing the engine even though it
had already been closed.

I wasn't able to repro this failure in the first place, but pretty sure
this:
Fixes cockroachdb#27232.

Release note: None
craig bot pushed a commit that referenced this issue Jul 18, 2018
26362: RFC: follower reads r=bdarnell,nvanbenschoten a=tschottdorf

NB: this is extracted from #21056; please don't add new commentary on the
tech note there.

----

Follower reads are consistent reads at historical timestamps from follower
replicas. They make the non-leader replicas in a range suitable sources for
historical reads.

The key enabling technology is the propagation of **closed timestamp
heartbeats** from the range leaseholder to followers. The closed timestamp
heartbeat (CT heartbeat) is more than just a timestamp. It is a set of
conditions, that if met, guarantee that a follower replica has all state
necessary to satisfy reads at or before the CT.

Consistent historical reads are useful for analytics queries and in particular
allow such queries to be carried out more efficiently and, with appropriate
configuration, away from foreground traffic. But historical reads are also key
to a proposal for [reference-like tables](#26301) aimed at cutting
down on foreign key check latencies particularly in geo-distributed clusters;
they help recover a reasonably recent consistent snapshot of a cluster after a
loss of quorum; and they are one of the ingredients for [Change Data
Capture](#25229).

Release note: None

27699: storage: fix stopper race in compactor r=petermattis a=tschottdorf

Starting workers without a surrounding task is unfortunately often not
the right thing to do when the worker accesses other state that might
become invalidated once the stopper begins to stop. In this particular
case, the compactor might end up accessing the engine even though it
had already been closed.

I wasn't able to repro this failure in the first place, but pretty sure
this:
Fixes #27232.

Release note: None

27704: issues: fix email fallback r=petermattis a=tschottdorf

This was not my email address.

Release note: None

Co-authored-by: Tobias Schottdorf <[email protected]>
@craig craig bot closed this as completed in #27699 Jul 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-kv-client Relating to the KV client and the KV interface. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants