Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stress: failed test in cockroach/storage/storage.test: TestReplicateAfterTruncation #8086

Closed
cockroach-teamcity opened this issue Jul 28, 2016 · 1 comment
Closed

stress: failed test in cockroach/storage/storage.test: TestReplicateAfterTruncation #8086

cockroach-teamcity opened this issue Jul 28, 2016 · 1 comment
Assignees
@tbg
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
Q3

Comments

@cockroach-teamcity
Copy link
Member

Binary: cockroach/static-tests.tar.gz sha: https://github.com/cockroachdb/cockroach/commits/a79e2bd677e746b73269ff7dac6c559e00166e2c

Stress build found a failed test:

=== RUN   TestReplicateAfterTruncation
I160728 05:22:02.408139 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
W160728 05:22:02.409038 gossip/gossip.go:953  not connected to cluster; use --join to specify a connected node
W160728 05:22:02.410296 gossip/gossip.go:953  not connected to cluster; use --join to specify a connected node
I160728 05:22:02.410459 storage/store.go:2781  system config not yet available
I160728 05:22:02.411088 storage/replica_command.go:1741  store=1:1 ([]=) range=1 [/Min-/Max): new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 900ms following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0 [physicalTime=1970-01-01 00:00:00 +0000 UTC]
I160728 05:22:02.411432 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
W160728 05:22:02.412230 gossip/gossip.go:953  not connected to cluster; use --join to specify a connected node
I160728 05:22:02.412786 storage/store.go:2781  system config not yet available
I160728 05:22:02.412867 gossip/client.go:74  starting client to 127.0.0.1:35625
I160728 05:22:02.414267 storage/replica_command.go:1532  store=1:1 ([]=) range=1 [/Min-/Max): attempting to truncate raft logs for another range 0. Normally this is due to a merge and can be ignored.
I160728 05:22:02.414868 storage/replica_raftstorage.go:467  generated snapshot for range 1 at index 15 in 157.465µs. encoded size=2849, 28 KV pairs, 5 log entries
I160728 05:22:02.415159 storage/replica_command.go:3204  store=1:1 ([]=) range=1 [/Min-/Max): change replicas of 1: read existing descriptor range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 
I160728 05:22:02.415546 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/""/RangeDescriptor,/Min), ConditionalPut [/Local/Range/""/RangeDescriptor,/Min), Put [/Meta2/Max,/Min), Put [/Meta1/Max,/Min), pErr=<nil>
I160728 05:22:02.415854 storage/replica.go:1419  store=1:1 ([]=) range=1 [/Min-/Max): proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:2} for range 1: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2}]
I160728 05:22:02.416171 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/Max): applied part of replica change txn: EndTransaction [/Local/Range/""/RangeDescriptor,/Min), pErr=<nil>
I160728 05:22:02.416489 storage/replica_raftstorage.go:588  store=2:2 ([]=) range=1 [/Min-/Min): with replicaID [?], applying preemptive snapshot for range 1 at index 15 (encoded size=2849, 28 KV pairs, 5 log entries)
W160728 05:22:02.417377 storage/replica.go:337  range 1: outgoing raft transport stream to {2 2 0} closed by the remote: <nil>
I160728 05:22:02.417588 storage/replica_raftstorage.go:591  store=2:2 ([]=) range=1 [/Min-/Max): with replicaID [?], applied preemptive snapshot for range 1 in 1.059719ms
--- FAIL: TestReplicateAfterTruncation (0.01s)
panic: descriptor is not initialized [recovered]
    panic: descriptor is not initialized [recovered]
    panic: descriptor is not initialized [recovered]
    panic: descriptor is not initialized

goroutine 68108 [running]:
panic(0x179f4c0, 0xc827da4d80)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
testing.tRunner.func1(0xc820430120)
    /usr/local/go/src/testing/testing.go:467 +0x192
panic(0x179f4c0, 0xc827da4d80)
    /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/cockroachdb/cockroach/util/leaktest.AfterTest.func1()
    /go/src/github.com/cockroachdb/cockroach/util/leaktest/leaktest.go:77 +0x95
panic(0x179f4c0, 0xc827da4d80)
    /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/cockroachdb/cockroach/storage.(*Store).Send.func1(0xc827b953c0, 0xc827b95438, 0x0, 0x3d, 0xc827b95430, 0xc8200ab180)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2119 +0x6e
panic(0x179f4c0, 0xc827da4d80)
    /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/cockroachdb/cockroach/roachpb.NewRangeKeyMismatchError(0xc827da4be8, 0x1, 0x8, 0x0, 0x0, 0x0, 0xc820577620, 0x0)
    /go/src/github.com/cockroachdb/cockroach/roachpb/errors.go:301 +0xb2
github.com/cockroachdb/cockroach/storage.(*Replica).checkCmdHeader(0xc8206f1800, 0xc827da4be8, 0x1, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:855 +0xfa
github.com/cockroachdb/cockroach/storage.(*Replica).executeCmd(0xc8206f1800, 0x7f8c374fc240, 0xc820783d70, 0x0, 0x0, 0x0, 0x7f8c37478258, 0xc8249e4300, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica_command.go:75 +0x138
github.com/cockroachdb/cockroach/storage.(*Replica).executeBatch(0xc8206f1800, 0x7f8c374fc240, 0xc820783d70, 0x0, 0x0, 0x7f8c37478258, 0xc8249e4300, 0x0, 0x0, 0x3c, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:2324 +0x5d4
github.com/cockroachdb/cockroach/storage.(*Replica).addReadOnlyCmd(0xc8206f1800, 0x7f8c374fc240, 0xc820783d70, 0x0, 0x3c, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1186 +0x332
github.com/cockroachdb/cockroach/storage.(*Replica).Send(0xc8206f1800, 0x7f8c374fc240, 0xc820783d70, 0x0, 0x3c, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:830 +0x52f
github.com/cockroachdb/cockroach/storage.(*Store).Send(0xc8200ab180, 0x7f8c37478000, 0xc82000c518, 0x0, 0x3c, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2214 +0xd67
github.com/cockroachdb/cockroach/internal/client.Wrap.func1(0x7f8c37478000, 0xc82000c518, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/sender.go:76 +0xb4
github.com/cockroachdb/cockroach/internal/client.SenderFunc.Send(0xc8200df620, 0x7f8c37478000, 0xc82000c518, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/sender.go:37 +0x53
github.com/cockroachdb/cockroach/internal/client.SendWrappedWith(0x7f8c374fcf88, 0xc8200df620, 0x7f8c37478000, 0xc82000c518, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/sender.go:54 +0x1a9
github.com/cockroachdb/cockroach/storage_test.TestReplicateAfterTruncation.func1(0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_raft_test.go:523 +0x2e2
github.com/cockroachdb/cockroach/util.RetryForDuration(0x37e11d600, 0xc827b95c68, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/util/testing.go:127 +0x180
github.com/cockroachdb/cockroach/util.SucceedsSoonDepth(0x1, 0x7f8c373f6a40, 0xc820430120, 0xc827b95c68)
    /go/src/github.com/cockroachdb/cockroach/util/testing.go:114 +0x37
github.com/cockroachdb/cockroach/util.SucceedsSoon(0x7f8c373f6a40, 0xc820430120, 0xc827b95c68)
    /go/src/github.com/cockroachdb/cockroach/util/testing.go:108 +0x3e
github.com/cockroachdb/cockroach/storage_test.TestReplicateAfterTruncation(0xc820430120)
    /go/src/github.com/cockroachdb/cockroach/storage/client_raft_test.go:529 +0xf21
testing.tRunner(0xc820430120, 0x2869308)
    /usr/local/go/src/testing/testing.go:473 +0x98
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:582 +0x892

goroutine 1 [chan receive]:
testing.RunTests(0x2026d28, 0x2867ca0, 0x127, 0x127, 0xc82024cc01)
    /usr/local/go/src/testing/testing.go:583 +0x8d2
testing.(*M).Run(0xc8204bdef8, 0xc820283c98)
    /usr/local/go/src/testing/testing.go:515 +0x81
github.com/cockroachdb/cockroach/storage_test.TestMain(0xc8204bdef8)
    /go/src/github.com/cockroachdb/cockroach/storage/main_test.go:57 +0x260
main.main()
    github.com/cockroachdb/cockroach/storage/_test/_testmain.go:652 +0x114

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x2b82da0)
    /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1015 +0x64
created by github.com/cockroachdb/cockroach/util/log.init.1
    /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:598 +0x8a


ERROR: exit status 2

Run Details:

0 runs so far, 0 failures, over 5s
0 runs so far, 0 failures, over 10s
0 runs so far, 0 failures, over 15s
0 runs so far, 0 failures, over 20s
0 runs so far, 0 failures, over 25s
0 runs so far, 0 failures, over 30s
0 runs so far, 0 failures, over 35s
0 runs so far, 0 failures, over 40s
0 runs so far, 0 failures, over 45s
0 runs so far, 0 failures, over 50s
0 runs so far, 0 failures, over 55s
1 runs completed, 1 failures, over 56s
FAIL

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added this to the Q3 milestone Jul 28, 2016
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jul 28, 2016
This was referenced Aug 1, 2016
Closed
Closed
Closed
@tbg tbg self-assigned this Aug 1, 2016
@tamird
Copy link
Contributor

tamird commented Aug 3, 2016

#8157.

@tamird tamird closed this as completed Aug 3, 2016
tbg added a commit to tbg/cockroach that referenced this issue Aug 3, 2016
@tbg
storage: do not use non-authoritative Desc()
2c509dd 
After ec00732, we were using a denormalized
copy of the descriptor which is not protected by the same mutex in places
where we shouldn't, resulting in subtle races. Change it back to use the
authoritative one almost everywhere, except for informational callsites
(such as stringifying a `Replica`).

Fixes cockroachdb#8086.
Fixes cockroachdb#8157.
@tbg tbg mentioned this issue Aug 3, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tbg tbg

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
Q3
Development

No branches or pull requests
3 participants
@tamird @tbg @cockroach-teamcity