Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: TestSplitSnapshotRace_SnapshotWins is disabled #8416

Closed
tbg opened this issue Aug 9, 2016 · 7 comments
Closed

storage: TestSplitSnapshotRace_SnapshotWins is disabled #8416

tbg opened this issue Aug 9, 2016 · 7 comments
Assignees
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@tbg
Copy link
Member

tbg commented Aug 9, 2016

The following test appears to have failed:

#21119:

I160809 16:10:59.496234 storage/engine/rocksdb.go:398  closing in-memory rocksdb instance
I160809 16:10:59.496523 storage/engine/rocksdb.go:398  closing in-memory rocksdb instance
I160809 16:10:59.499416 storage/engine/rocksdb.go:398  closing in-memory rocksdb instance
I160809 16:10:59.499679 storage/engine/rocksdb.go:398  closing in-memory rocksdb instance
I160809 16:10:59.500355 storage/engine/rocksdb.go:398  closing in-memory rocksdb instance
--- FAIL: TestSplitSnapshotRace_SnapshotWins (16.98s)
    <autogenerated>:12: storage/client_split_test.go:1050, condition failed to evaluate within 15s: expected [0 0 0 2 25 25], got [0 0 0 25 25 25]
=== RUN   TestStoreSplitTimestampCacheReadRace
I160809 16:10:59.511950 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
W160809 16:10:59.514467 gossip/gossip.go:1015  not connected to cluster; use --join to specify a connected node
I160809 16:10:59.524527 storage/store.go:2732  system config not yet available
I160809 16:10:59.527705 storage/replica.go:2660  gossiping cluster id {"00000000-0000-0000-0000-000000000000"} from store 1, range 1
I160809 16:10:59.532568 storage/replica.go:2047  store=1:1 range=1 [/Min-/Max): applying command with forced error: storage/replica.go:1972: no-op on empty Raft entry
E160809 16:10:59.533413 storage/replica.go:2082  store=1:1 range=1 [/Min-/Max): error executing raft command: storage/replica.go:1972: no-op on empty Raft entry
I160809 16:10:59.534131 storage/replica_trigger.go:308  store=1:1 range=1 [/Min-/Max): new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 900.000123ms following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=1970-01-01 00:00:00.000000123 +0000 UTC]
I160809 16:10:59.534206 storage/replica.go:2682  gossiping sentinel from store 1, range 1
--
I160809 16:11:08.725458 stopper.go:408  quiesceing; tasks left:
18     gossip/infostore.go:290
I160809 16:11:08.725584 stopper.go:408  quiesceing; tasks left:
17     gossip/infostore.go:290
--- PASS: Example_rebalancing (0.95s)
FAIL
FAIL    github.com/cockroachdb/cockroach/storage    86.524s
=== RUN   TestBatchBasics
I160809 16:11:18.274805 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
I160809 16:11:18.277250 storage/engine/rocksdb.go:398  closing in-memory rocksdb instance
--- PASS: TestBatchBasics (0.01s)
=== RUN   TestBatchRepr
I160809 16:11:18.292548 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
I160809 16:11:18.294597 storage/engine/rocksdb.go:398  closing in-memory rocksdb instance
--- PASS: TestBatchRepr (0.01s)
=== RUN   TestBatchGet
I160809 16:11:18.302996 storage/engine/rocksdb.go:353  opening in memory rocksdb instance

Please assign, take a look and update the issue accordingly.

@tbg tbg added this to the Q2 milestone Aug 9, 2016
@tbg tbg added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 9, 2016
@tamird
Copy link
Contributor

tamird commented Aug 11, 2016

@tamird
Copy link
Contributor

tamird commented Sep 1, 2016

#8987 is the current incarnation of this.

@tamird tamird closed this as completed Sep 1, 2016
@rjnn
Copy link
Contributor

rjnn commented Sep 1, 2016

This doesn't look like the same issue as #8987. In #8987, SnapshotWins is disabled for being flaky - it's SplitWins which is causing the problem. (SnapshotWins was skipped in #8703).

@tamird
Copy link
Contributor

tamird commented Sep 1, 2016

I believe it is the same issue - this failure comes from the shared setup code used by both tests.

@tamird
Copy link
Contributor

tamird commented Sep 1, 2016

Wait, nope, you're right - this is a failure in the actual test.

@tamird tamird reopened this Sep 1, 2016
@tamird
Copy link
Contributor

tamird commented Oct 4, 2016

This test is disabled as of 63e4425.

@tamird tamird changed the title circleci: failed tests: TestSplitSnapshotRace_SnapshotWins storage: TestSplitSnapshotRace_SnapshotWins is disabled Oct 4, 2016
@rjnn rjnn assigned vivekmenezes and unassigned rjnn Nov 3, 2016
@bdarnell
Copy link
Contributor

bdarnell commented Nov 6, 2016

Assigning myself since I'm looking at this test's sibling in #9204.

@bdarnell bdarnell assigned bdarnell and unassigned vivekmenezes Nov 6, 2016
bdarnell added a commit to bdarnell/cockroach that referenced this issue Nov 7, 2016
LocalProposalData is processed only on the proposer, which is unlikely in
practice to be the node being removed (but was apparently the case in
all of our tests except in rare cases).

This deflakes the SplitSnapshotRace tests, which would occasionally end
up with the lease on a different store and miss the automatic GC (and
because the test uses a manual clock, the background replicaGCQueue scan
wouldn't GC them either).

Fixes cockroachdb#8416
Fixes cockroachdb#9204
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

No branches or pull requests

5 participants