stress: failed test in cockroach/security/security.test: TestUseCerts

[cockroach-teamcity]

Binary: cockroach/static-tests.tar.gz sha: https://github.com/cockroachdb/cockroach/commits/0bd1b6a408578214e62f7f3704950f1cfda9b36e

Stress build found a failed test:

=== RUN   TestUseCerts
I160824 04:50:32.527025 storage/engine/rocksdb.go:354  opening in memory rocksdb instance
W160824 04:50:32.538425 gossip/gossip.go:1039  not connected to cluster; use --join to specify a connected node
W160824 04:50:32.539205 server/status/runtime.go:109  Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I160824 04:50:32.540221 server/node.go:407  store store=0:0 not bootstrapped
I160824 04:50:32.548289 storage/replica_trigger.go:308  store=1:1 range=1 [/Min-/Max): new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 408892h50m41.547565066s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=2016-08-24 04:50:32.548193757 +0000 UTC]
I160824 04:50:32.548929 server/node.go:339  **** cluster {d66851c0-de04-4ba0-a37b-e273d4a6ac27} has been created
I160824 04:50:32.548962 server/node.go:340  **** add additional nodes by specifying --join=127.0.0.1:43956
I160824 04:50:32.549756 storage/store.go:982  store=1:1: failed initial metrics computation: store=1:1: system config not yet available
I160824 04:50:32.549792 server/node.go:420  initialized store store=1:1: {Capacity:536870912 Available:536870912 RangeCount:0}
I160824 04:50:32.549828 server/node.go:312  node ID 1 initialized
I160824 04:50:32.549885 storage/stores.go:302  read 0 node addresses from persistent storage
I160824 04:50:32.549992 server/node.go:549  connecting to gossip network to verify cluster ID...
I160824 04:50:32.550285 server/node.go:570  node connected via gossip and verified as part of cluster {"d66851c0-de04-4ba0-a37b-e273d4a6ac27"}
I160824 04:50:32.550340 server/node.go:358  node=1: started with [[]=] engine(s) and attributes []
I160824 04:50:32.550491 storage/split_queue.go:99  splitting store=1:1 range=1 [/Min-/Max) at keys [/Table/11/0 /Table/12/0 /Table/13/0 /Table/14/0]
I160824 04:50:32.550848 sql/event_log.go:109  Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:43956} Attrs:} ClusterID:{UUID:d66851c0-de04-4ba0-a37b-e273d4a6ac27} StartedAt:1472014232550291630}
I160824 04:50:32.551669 server/server.go:459  starting https server at 127.0.0.1:45899
I160824 04:50:32.552234 server/server.go:460  starting grpc/postgres server at 127.0.0.1:43956
I160824 04:50:32.552499 sql/event_log.go:109  Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:43956} Attrs:} ClusterID:{UUID:d66851c0-de04-4ba0-a37b-e273d4a6ac27} StartedAt:1472014232550291630}
I160824 04:50:32.552735 storage/replica_command.go:2213  store=1:1 range=1 [/Min-/Max): initiating a split of this range at key /Table/11
I160824 04:50:32.559997 storage/replica_command.go:2213  store=1:1 range=2 [/Table/11-/Max): initiating a split of this range at key /Table/12
I160824 04:50:32.592661 storage/replica_command.go:2213  store=1:1 range=3 [/Table/12-/Max): initiating a split of this range at key /Table/13
W160824 04:50:32.593193 storage/intent_resolver.go:344  could not GC completed transaction: end key /Local/Range/"\x93"/"rdsc\x00" must be greater than start /Local/Range/"\x93"/RangeDescriptor
I160824 04:50:32.597847 storage/replica_command.go:2213  store=1:1 range=4 [/Table/13-/Max): initiating a split of this range at key /Table/14
W160824 04:50:32.598341 storage/intent_resolver.go:344  could not GC completed transaction: end key /Local/Range/"\x94"/"rdsc\x00" must be greater than start /Local/Range/"\x94"/RangeDescriptor
I160824 04:50:32.609874 /go/src/google.golang.org/grpc/clientconn.go:460  Failed to dial 127.0.0.1:43956: connection error: desc = "transport: remote error: tls: bad record MAC"; please retry.
I160824 04:50:32.610056 /go/src/google.golang.org/grpc/server.go:369  grpc: Server.Serve failed to complete security handshake from "127.0.0.1:34606": local error: tls: bad record MAC
W160824 04:50:32.634859 storage/intent_resolver.go:344  could not GC completed transaction: end key /Local/Range/"\x95"/"rdsc\x00" must be greater than start /Local/Range/"\x95"/RangeDescriptor
I160824 04:50:32.753133 context.go:196  no certificates specified: using insecure TLS
E160824 04:50:32.753337 server.go:2315  http: TLS handshake error from 127.0.0.1:42816: EOF
--- FAIL: TestUseCerts (0.27s)
    certs_test.go:185: Expected OK, got: 500

Run Details:

199 runs so far, 0 failures, over 5s
404 runs so far, 0 failures, over 10s
608 runs so far, 0 failures, over 15s
812 runs so far, 0 failures, over 20s
1016 runs so far, 0 failures, over 25s
1221 runs so far, 0 failures, over 30s
1424 runs so far, 0 failures, over 35s
1626 runs so far, 0 failures, over 40s
1829 runs so far, 0 failures, over 45s
2034 runs so far, 0 failures, over 50s
2232 runs so far, 0 failures, over 55s
2440 runs so far, 0 failures, over 1m0s
2639 runs so far, 0 failures, over 1m5s
2844 runs so far, 0 failures, over 1m10s
3044 runs so far, 0 failures, over 1m15s
3251 runs so far, 0 failures, over 1m20s
3459 runs so far, 0 failures, over 1m25s
3662 runs so far, 0 failures, over 1m30s
3866 runs so far, 0 failures, over 1m35s
4070 runs so far, 0 failures, over 1m40s
4276 runs so far, 0 failures, over 1m45s
4479 runs so far, 0 failures, over 1m50s
4685 runs so far, 0 failures, over 1m55s
4887 runs so far, 0 failures, over 2m0s
5088 runs so far, 0 failures, over 2m5s
5291 runs so far, 0 failures, over 2m10s
5492 runs so far, 0 failures, over 2m15s
5694 runs so far, 0 failures, over 2m20s
5899 runs so far, 0 failures, over 2m25s
6104 runs so far, 0 failures, over 2m30s
6306 runs so far, 0 failures, over 2m35s
6507 runs so far, 0 failures, over 2m40s
6710 runs so far, 0 failures, over 2m45s
6914 runs so far, 0 failures, over 2m50s
7114 runs so far, 0 failures, over 2m55s
7318 runs so far, 0 failures, over 3m0s
7523 runs so far, 0 failures, over 3m5s
7727 runs so far, 0 failures, over 3m10s
7926 runs so far, 0 failures, over 3m15s
8128 runs so far, 0 failures, over 3m20s
8333 runs so far, 0 failures, over 3m25s
8354 runs completed, 1 failures, over 3m26s
FAIL

Please assign, take a look and update the issue accordingly.

[tamird]

130k runs in GCE and no repro. Closing.