Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: TestSnapshotAfterTruncation failed under stress #28003

Closed
cockroach-teamcity opened this issue Jul 28, 2018 · 5 comments
Closed

storage: TestSnapshotAfterTruncation failed under stress #28003

cockroach-teamcity opened this issue Jul 28, 2018 · 5 comments
Assignees
@jordanlewis @andreimatei
Labels
A-kv-client Relating to the KV client and the KV interface. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/a149437eec20fe632d058aa146817b5f9614095b

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=801563&tab=buildLog

=== RUN   TestSnapshotAfterTruncation
--- FAIL: TestSnapshotAfterTruncation (46.88s)
=== RUN   TestSnapshotAfterTruncation/differentTerm
I180728 06:25:34.602949 57843 storage/store.go:1493  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I180728 06:25:34.620950 57843 gossip/gossip.go:363  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:34397" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I180728 06:25:34.670523 57843 storage/store.go:1493  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
W180728 06:25:34.680286 57843 gossip/gossip.go:1325  [n2] no incoming or outgoing connections
I180728 06:25:34.680928 57843 gossip/gossip.go:363  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:33661" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I180728 06:25:34.683666 57929 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:34397
I180728 06:25:34.740291 57843 storage/store.go:1493  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
W180728 06:25:34.750274 57843 gossip/gossip.go:1325  [n3] no incoming or outgoing connections
I180728 06:25:34.751631 57843 gossip/gossip.go:363  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:40193" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I180728 06:25:34.754790 58205 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:34397
I180728 06:25:34.859739 57843 storage/replica_raftstorage.go:520  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 5c0945aa at index 17
I180728 06:25:34.860312 57843 rpc/nodedialer/nodedialer.go:90  [s1,r1/1:/M{in-ax}] connection to n2 established
I180728 06:25:34.866380 57843 storage/store_snapshot.go:605  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 48, log entries: 7, rate-limit: 2.0 MiB/sec, 5ms
I180728 06:25:34.868675 58095 storage/replica_raftstorage.go:737  [s2,r1/?:{-}] applying preemptive snapshot at index 17 (id=5c0945aa, encoded size=8279, 1 rocksdb batches, 7 log entries)
I180728 06:25:34.882711 58095 storage/replica_raftstorage.go:743  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 12ms [clear=0ms batch=0ms entries=2ms commit=9ms]
I180728 06:25:34.888681 57843 storage/replica_command.go:745  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I180728 06:25:34.915583 57843 storage/replica.go:3641  [txn=23c6d7d9,s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I180728 06:25:34.935801 57843 storage/replica_raftstorage.go:520  [s1,r1/1:/M{in-ax}] generated preemptive snapshot bc8f7591 at index 19
I180728 06:25:34.936169 57843 rpc/nodedialer/nodedialer.go:90  [s1,r1/1:/M{in-ax}] connection to n3 established
I180728 06:25:34.944192 58259 storage/replica_raftstorage.go:737  [s3,r1/?:{-}] applying preemptive snapshot at index 19 (id=bc8f7591, encoded size=9223, 1 rocksdb batches, 9 log entries)
I180728 06:25:34.948720 58259 storage/replica_raftstorage.go:743  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 4ms [clear=0ms batch=0ms entries=3ms commit=1ms]
I180728 06:25:34.950371 57843 storage/store_snapshot.go:605  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 51, log entries: 9, rate-limit: 2.0 MiB/sec, 14ms
I180728 06:25:34.952452 57843 storage/replica_command.go:745  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I180728 06:25:34.962121 58034 rpc/nodedialer/nodedialer.go:90  connection to n1 established
I180728 06:25:34.983317 57843 storage/replica.go:3641  [txn=fcca963b,s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
I180728 06:25:35.153988 57843 util/stop/stopper.go:537  quiescing; tasks left:
1      [async] kv.TxnCoordSender: heartbeat loop
W180728 06:25:35.176158 58099 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: node unavailable; try another peer
I180728 06:25:35.178674 58281 internal/client/txn.go:721  async rollback failed: node unavailable; try another peer
I180728 06:25:35.210927 58240 storage/client_test.go:1233  [hb,txn=b8bec6c3,range-lookup=/System/NodeLiveness/3,range-lookup=/Meta2/System/NodeLiveness/3/NULL] test clock advanced to: 1.800000125,0
I180728 06:25:35.362828 57843 util/stop/stopper.go:537  quiescing; tasks left:
2      [async] kv.TxnCoordSender: heartbeat loop
I180728 06:25:35.364125 57843 util/stop/stopper.go:537  quiescing; tasks left:
1      [async] kv.TxnCoordSender: heartbeat loop
I180728 06:25:35.365622 58204 storage/node_liveness.go:813  [hb] retrying liveness update after storage.errRetryLiveness: TransactionStatusError: could not commit in one phase as requested (REASON_UNKNOWN)
W180728 06:25:35.366048 58204 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context canceled
I180728 06:25:35.366950 58270 internal/client/txn.go:721  async rollback failed: node unavailable; try another peer
W180728 06:25:35.888098 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:35.892670 57755 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=d3699fee key=/System/NodeLiveness/3 rw=true pri=0.00195079 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,169 orig=1.800000125,169 max=1.800000125,188 wto=false rop=false seq=4
W180728 06:25:36.340877 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:36.345083 58225 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=81520562 key=/System/NodeLiveness/3 rw=true pri=0.03974233 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,271 orig=1.800000125,271 max=1.800000125,276 wto=false rop=false seq=4
W180728 06:25:36.793996 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:36.804045 58408 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=3407697e key=/System/NodeLiveness/3 rw=true pri=0.01200652 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,351 orig=1.800000125,351 max=1.800000125,354 wto=false rop=false seq=4
W180728 06:25:37.247046 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:37.260910 58505 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=d3d267b8 key=/System/NodeLiveness/3 rw=true pri=0.03223419 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,431 orig=1.800000125,431 max=1.800000125,432 wto=false rop=false seq=4
W180728 06:25:37.699974 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:37.703128 58274 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=5131ebfa key=/System/NodeLiveness/3 rw=true pri=0.06118179 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,512 orig=1.800000125,512 max=1.800000125,513 wto=false rop=false seq=4
W180728 06:25:38.153074 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:38.161977 58489 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=eefaf743 key=/System/NodeLiveness/3 rw=true pri=0.01048055 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,594 orig=1.800000125,594 max=1.800000125,595 wto=false rop=false seq=4
I180728 06:25:38.520614 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:38.606217 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:38.617901 58411 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=0fefe0d6 key=/System/NodeLiveness/3 rw=true pri=0.01949053 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,672 orig=1.800000125,672 max=1.800000125,675 wto=false rop=false seq=4
W180728 06:25:39.059464 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:39.065977 58412 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=0011d64a key=/System/NodeLiveness/3 rw=true pri=0.02105734 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,749 orig=1.800000125,749 max=1.800000125,754 wto=false rop=false seq=4
W180728 06:25:39.513841 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:39.520995 57843 storage/client_test.go:942  SucceedsSoon: node not live
I180728 06:25:39.525210 58510 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=dd505787 key=/System/NodeLiveness/3 rw=true pri=0.01976471 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,832 orig=1.800000125,832 max=1.800000125,833 wto=false rop=false seq=4
W180728 06:25:39.977117 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:39.982287 58556 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=1a586207 key=/System/NodeLiveness/3 rw=true pri=0.05982216 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,907 orig=1.800000125,907 max=1.800000125,908 wto=false rop=false seq=4
W180728 06:25:40.430322 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:40.433956 58583 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=27c49f1a key=/System/NodeLiveness/3 rw=true pri=0.02046986 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,988 orig=1.800000125,988 max=1.800000125,990 wto=false rop=false seq=4
I180728 06:25:40.521434 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:40.883882 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:40.888589 58577 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=3bdfdc7a key=/System/NodeLiveness/3 rw=true pri=0.03916160 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1068 orig=1.800000125,1068 max=1.800000125,1073 wto=false rop=false seq=4
W180728 06:25:41.341350 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:41.344574 58598 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=a55a5dec key=/System/NodeLiveness/3 rw=true pri=0.02382002 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1151 orig=1.800000125,1151 max=1.800000125,1152 wto=false rop=false seq=4
I180728 06:25:41.521857 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:41.798537 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:41.806837 58586 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=e7be261f key=/System/NodeLiveness/3 rw=true pri=0.06361818 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1231 orig=1.800000125,1231 max=1.800000125,1232 wto=false rop=false seq=4
W180728 06:25:42.256434 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:42.262466 58589 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=acf8d97c key=/System/NodeLiveness/3 rw=true pri=0.02428023 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1306 orig=1.800000125,1306 max=1.800000125,1307 wto=false rop=false seq=4
I180728 06:25:42.522394 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:42.709501 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:42.713049 58593 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=c384a3fb key=/System/NodeLiveness/3 rw=true pri=0.03890716 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1386 orig=1.800000125,1386 max=1.800000125,1393 wto=false rop=false seq=4
W180728 06:25:43.162528 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:43.172860 58493 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=7464f02b key=/System/NodeLiveness/3 rw=true pri=0.01688097 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1469 orig=1.800000125,1469 max=1.800000125,1470 wto=false rop=false seq=4
I180728 06:25:43.522894 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:43.616482 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:43.637838 58644 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=e30ac0b2 key=/System/NodeLiveness/3 rw=true pri=0.00778441 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1550 orig=1.800000125,1550 max=1.800000125,1551 wto=false rop=false seq=4
W180728 06:25:44.069626 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:44.073081 58608 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=64e5bf7f key=/System/NodeLiveness/3 rw=true pri=0.02647568 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1624 orig=1.800000125,1624 max=1.800000125,1625 wto=false rop=false seq=4
W180728 06:25:44.522752 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:44.524424 57843 storage/client_test.go:942  SucceedsSoon: node not live
I180728 06:25:44.534139 58676 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=8e0bcee6 key=/System/NodeLiveness/3 rw=true pri=0.02141157 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1710 orig=1.800000125,1710 max=1.800000125,1711 wto=false rop=false seq=4
W180728 06:25:44.975825 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:44.979276 58678 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=abddf2e2 key=/System/NodeLiveness/3 rw=true pri=0.00407533 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1788 orig=1.800000125,1788 max=1.800000125,1790 wto=false rop=false seq=4
W180728 06:25:45.428197 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:45.432193 58707 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=e3a69ecb key=/System/NodeLiveness/3 rw=true pri=0.01813038 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1866 orig=1.800000125,1866 max=1.800000125,1869 wto=false rop=false seq=4
I180728 06:25:45.524976 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:45.880533 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:45.883686 58709 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=06e9285d key=/System/NodeLiveness/3 rw=true pri=0.03940142 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,1943 orig=1.800000125,1943 max=1.800000125,1947 wto=false rop=false seq=4
W180728 06:25:46.333347 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:46.338078 58713 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=003b7f41 key=/System/NodeLiveness/3 rw=true pri=0.01351964 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2025 orig=1.800000125,2025 max=1.800000125,2028 wto=false rop=false seq=4
I180728 06:25:46.525379 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:46.786587 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:46.798978 58694 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=a1f80e70 key=/System/NodeLiveness/3 rw=true pri=0.02167369 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2102 orig=1.800000125,2102 max=1.800000125,2106 wto=false rop=false seq=4
W180728 06:25:47.244294 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:47.247705 58726 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=630ebf95 key=/System/NodeLiveness/3 rw=true pri=0.00066836 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2186 orig=1.800000125,2186 max=1.800000125,2187 wto=false rop=false seq=4
I180728 06:25:47.525856 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:47.697297 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:47.700760 58664 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=1159d4b5 key=/System/NodeLiveness/3 rw=true pri=0.02850415 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2267 orig=1.800000125,2267 max=1.800000125,2268 wto=false rop=false seq=4
W180728 06:25:48.150408 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:48.159030 58695 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=9db27d29 key=/System/NodeLiveness/3 rw=true pri=0.08696993 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2346 orig=1.800000125,2346 max=1.800000125,2347 wto=false rop=false seq=4
I180728 06:25:48.526268 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:48.603424 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:48.613468 58638 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=358d0615 key=/System/NodeLiveness/3 rw=true pri=0.03423519 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2426 orig=1.800000125,2426 max=1.800000125,2427 wto=false rop=false seq=4
W180728 06:25:49.062862 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:49.067189 58731 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=c34e1369 key=/System/NodeLiveness/3 rw=true pri=0.01912625 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2502 orig=1.800000125,2502 max=1.800000125,2503 wto=false rop=false seq=4
W180728 06:25:49.516900 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:49.520500 58673 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=953bc089 key=/System/NodeLiveness/3 rw=true pri=0.04013143 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2584 orig=1.800000125,2584 max=1.800000125,2588 wto=false rop=false seq=4
I180728 06:25:49.527087 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:49.970171 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:49.974062 58774 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=1d0f62db key=/System/NodeLiveness/3 rw=true pri=0.04533678 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2664 orig=1.800000125,2664 max=1.800000125,2666 wto=false rop=false seq=4
W180728 06:25:50.425535 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:50.428803 58697 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=97d5fad5 key=/System/NodeLiveness/3 rw=true pri=0.02022879 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2742 orig=1.800000125,2742 max=1.800000125,2744 wto=false rop=false seq=4
I180728 06:25:50.527916 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:50.878256 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:50.881249 58776 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=f288ac83 key=/System/NodeLiveness/3 rw=true pri=0.02752347 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2820 orig=1.800000125,2820 max=1.800000125,2825 wto=false rop=false seq=4
W180728 06:25:51.331093 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:51.338610 58751 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=38eff29d key=/System/NodeLiveness/3 rw=true pri=0.04156912 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2902 orig=1.800000125,2902 max=1.800000125,2905 wto=false rop=false seq=4
I180728 06:25:51.528430 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:51.784149 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:51.787991 58753 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=97002f45 key=/System/NodeLiveness/3 rw=true pri=0.01040376 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,2979 orig=1.800000125,2979 max=1.800000125,2983 wto=false rop=false seq=4
W180728 06:25:52.237113 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:52.240525 58795 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=666dcd15 key=/System/NodeLiveness/3 rw=true pri=0.00377917 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3064 orig=1.800000125,3064 max=1.800000125,3065 wto=false rop=false seq=4
I180728 06:25:52.528878 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:52.689832 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:52.692733 58799 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=0a1c8d5d key=/System/NodeLiveness/3 rw=true pri=0.05102898 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3142 orig=1.800000125,3142 max=1.800000125,3143 wto=false rop=false seq=4
W180728 06:25:53.142887 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:53.146404 58836 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=bf27fbd8 key=/System/NodeLiveness/3 rw=true pri=0.01269002 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3220 orig=1.800000125,3220 max=1.800000125,3224 wto=false rop=false seq=4
I180728 06:25:53.530156 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:53.595146 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:53.599222 58820 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=cde9e468 key=/System/NodeLiveness/3 rw=true pri=0.02128668 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3301 orig=1.800000125,3301 max=1.800000125,3302 wto=false rop=false seq=4
W180728 06:25:54.048092 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:54.051532 58869 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=3a8e971d key=/System/NodeLiveness/3 rw=true pri=0.10966728 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3379 orig=1.800000125,3379 max=1.800000125,3382 wto=false rop=false seq=4
W180728 06:25:54.501136 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:54.510319 58884 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=d1d5e95d key=/System/NodeLiveness/3 rw=true pri=0.03116042 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3459 orig=1.800000125,3459 max=1.800000125,3461 wto=false rop=false seq=4
I180728 06:25:54.530690 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:54.956112 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:54.960638 58888 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=782a5bcb key=/System/NodeLiveness/3 rw=true pri=0.01323288 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3537 orig=1.800000125,3537 max=1.800000125,3538 wto=false rop=false seq=4
W180728 06:25:55.408723 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:55.416140 58873 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=761ad2e2 key=/System/NodeLiveness/3 rw=true pri=0.02533635 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3616 orig=1.800000125,3616 max=1.800000125,3621 wto=false rop=false seq=4
I180728 06:25:55.531111 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:55.861373 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:55.864516 58877 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=296ad9dc key=/System/NodeLiveness/3 rw=true pri=0.01013102 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3695 orig=1.800000125,3695 max=1.800000125,3696 wto=false rop=false seq=4
W180728 06:25:56.314150 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:56.317406 58858 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=748166d7 key=/System/NodeLiveness/3 rw=true pri=0.03389618 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3780 orig=1.800000125,3780 max=1.800000125,3781 wto=false rop=false seq=4
I180728 06:25:56.531566 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:56.769399 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:56.777860 58895 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=b06387da key=/System/NodeLiveness/3 rw=true pri=0.03531720 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3855 orig=1.800000125,3855 max=1.800000125,3859 wto=false rop=false seq=4
W180728 06:25:57.222673 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:57.226359 58860 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=addf950e key=/System/NodeLiveness/3 rw=true pri=0.03634165 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,3938 orig=1.800000125,3938 max=1.800000125,3940 wto=false rop=false seq=4
I180728 06:25:57.532196 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:57.675774 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:57.689193 58902 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=dddaec18 key=/System/NodeLiveness/3 rw=true pri=0.06085369 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4015 orig=1.800000125,4015 max=1.800000125,4019 wto=false rop=false seq=4
W180728 06:25:58.128824 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:58.132526 58807 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=ac5931a1 key=/System/NodeLiveness/3 rw=true pri=0.01333556 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4095 orig=1.800000125,4095 max=1.800000125,4096 wto=false rop=false seq=4
I180728 06:25:58.532617 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:58.581849 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:58.588069 58905 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=f984d16b key=/System/NodeLiveness/3 rw=true pri=0.02446468 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4178 orig=1.800000125,4178 max=1.800000125,4179 wto=false rop=false seq=4
I180728 06:25:58.856852 57953 gossip/gossip.go:1339  [n2] node has connected to cluster via gossip
W180728 06:25:59.035055 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:59.045049 58916 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=5096cd4f key=/System/NodeLiveness/3 rw=true pri=0.02552327 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4256 orig=1.800000125,4256 max=1.800000125,4257 wto=false rop=false seq=4
W180728 06:25:59.488030 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:59.500796 58810 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=d7692298 key=/System/NodeLiveness/3 rw=true pri=0.04649065 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4336 orig=1.800000125,4336 max=1.800000125,4338 wto=false rop=false seq=4
I180728 06:25:59.533103 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:25:59.940495 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:25:59.944208 58939 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=b196be63 key=/System/NodeLiveness/3 rw=true pri=0.01874804 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4411 orig=1.800000125,4411 max=1.800000125,4412 wto=false rop=false seq=4
W180728 06:26:00.393002 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:00.396511 58952 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=4cebf6ff key=/System/NodeLiveness/3 rw=true pri=0.00335905 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4492 orig=1.800000125,4492 max=1.800000125,4496 wto=false rop=false seq=4
I180728 06:26:00.533543 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:00.845355 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:00.849202 58967 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=c557a85d key=/System/NodeLiveness/3 rw=true pri=0.00905595 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4574 orig=1.800000125,4574 max=1.800000125,4575 wto=false rop=false seq=4
I180728 06:26:01.067314 58202 gossip/gossip.go:1339  [n3] node has connected to cluster via gossip
W180728 06:26:01.298607 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:01.315274 58946 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=3402e751 key=/System/NodeLiveness/3 rw=true pri=0.01353021 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4652 orig=1.800000125,4652 max=1.800000125,4656 wto=false rop=false seq=4
I180728 06:26:01.533961 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:01.751284 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:01.764536 58928 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=b954eef2 key=/System/NodeLiveness/3 rw=true pri=0.02072048 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4729 orig=1.800000125,4729 max=1.800000125,4730 wto=false rop=false seq=4
W180728 06:26:02.204818 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:02.209361 58930 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=30a10afc key=/System/NodeLiveness/3 rw=true pri=0.09951093 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4813 orig=1.800000125,4813 max=1.800000125,4814 wto=false rop=false seq=4
I180728 06:26:02.534383 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:02.658814 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:02.669199 59014 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=c221df2c key=/System/NodeLiveness/3 rw=true pri=0.01898631 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4888 orig=1.800000125,4888 max=1.800000125,4893 wto=false rop=false seq=4
W180728 06:26:03.111531 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:03.113834 58815 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=cfcca8b3 key=/System/NodeLiveness/3 rw=true pri=0.00265897 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,4972 orig=1.800000125,4972 max=1.800000125,4976 wto=false rop=false seq=4
I180728 06:26:03.534910 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:03.564516 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:03.568197 59019 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=00832c41 key=/System/NodeLiveness/3 rw=true pri=0.02326965 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5050 orig=1.800000125,5050 max=1.800000125,5052 wto=false rop=false seq=4
W180728 06:26:04.017643 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:04.020994 58985 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=0093be9c key=/System/NodeLiveness/3 rw=true pri=0.02247710 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5134 orig=1.800000125,5134 max=1.800000125,5135 wto=false rop=false seq=4
W180728 06:26:04.470745 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:04.474249 59005 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=a6263a74 key=/System/NodeLiveness/3 rw=true pri=0.02305601 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5212 orig=1.800000125,5212 max=1.800000125,5213 wto=false rop=false seq=4
I180728 06:26:04.539235 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:04.923707 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:04.927675 59046 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=a1d99c45 key=/System/NodeLiveness/3 rw=true pri=0.03846674 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5288 orig=1.800000125,5288 max=1.800000125,5292 wto=false rop=false seq=4
W180728 06:26:05.376617 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:05.380511 59032 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=008f5b1c key=/System/NodeLiveness/3 rw=true pri=0.01011346 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5367 orig=1.800000125,5367 max=1.800000125,5369 wto=false rop=false seq=4
I180728 06:26:05.539692 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:05.829246 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:05.831561 59062 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=34d17a34 key=/System/NodeLiveness/3 rw=true pri=0.00745812 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5447 orig=1.800000125,5447 max=1.800000125,5451 wto=false rop=false seq=4
W180728 06:26:06.281596 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:06.284734 58991 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=fab51f93 key=/System/NodeLiveness/3 rw=true pri=0.02157129 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5527 orig=1.800000125,5527 max=1.800000125,5531 wto=false rop=false seq=4
I180728 06:26:06.540096 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:06.740343 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:06.750142 59093 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=615fabe4 key=/System/NodeLiveness/3 rw=true pri=0.03266712 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5607 orig=1.800000125,5607 max=1.800000125,5610 wto=false rop=false seq=4
W180728 06:26:07.193884 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:07.197196 59036 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=16dadb21 key=/System/NodeLiveness/3 rw=true pri=0.01801373 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5685 orig=1.800000125,5685 max=1.800000125,5686 wto=false rop=false seq=4
I180728 06:26:07.540526 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:07.647232 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:07.651508 59112 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=dd0b8d81 key=/System/NodeLiveness/3 rw=true pri=0.01321561 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5769 orig=1.800000125,5769 max=1.800000125,5770 wto=false rop=false seq=4
W180728 06:26:08.100567 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:08.107441 59078 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=9b9afeaa key=/System/NodeLiveness/3 rw=true pri=0.02083760 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5844 orig=1.800000125,5844 max=1.800000125,5848 wto=false rop=false seq=4
I180728 06:26:08.541890 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:08.554272 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:08.557331 59083 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=8f4a61d1 key=/System/NodeLiveness/3 rw=true pri=0.03656079 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,5924 orig=1.800000125,5924 max=1.800000125,5925 wto=false rop=false seq=4
W180728 06:26:09.006920 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:09.009461 59100 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=50bf7cc3 key=/System/NodeLiveness/3 rw=true pri=0.03207433 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6003 orig=1.800000125,6003 max=1.800000125,6007 wto=false rop=false seq=4
W180728 06:26:09.461662 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:09.464868 59102 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=de816070 key=/System/NodeLiveness/3 rw=true pri=0.01668120 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6084 orig=1.800000125,6084 max=1.800000125,6088 wto=false rop=false seq=4
I180728 06:26:09.542332 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:09.913971 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:09.916806 59115 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=5f77159d key=/System/NodeLiveness/3 rw=true pri=0.00643306 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6167 orig=1.800000125,6167 max=1.800000125,6168 wto=false rop=false seq=4
W180728 06:26:10.367749 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:10.377202 59089 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=6ac02853 key=/System/NodeLiveness/3 rw=true pri=0.01900704 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6243 orig=1.800000125,6243 max=1.800000125,6247 wto=false rop=false seq=4
I180728 06:26:10.413043 57978 gossip/gossip.go:1339  [n1] node has connected to cluster via gossip
I180728 06:26:10.542803 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:10.823462 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:10.827079 59118 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=7de07e53 key=/System/NodeLiveness/3 rw=true pri=0.04542596 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6324 orig=1.800000125,6324 max=1.800000125,6326 wto=false rop=false seq=4
W180728 06:26:11.276365 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:11.280294 59173 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=3f2eaa46 key=/System/NodeLiveness/3 rw=true pri=0.01086188 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6419 orig=1.800000125,6419 max=1.800000125,6421 wto=false rop=false seq=4
I180728 06:26:11.547495 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:11.730715 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:11.738731 59070 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=2b990b83 key=/System/NodeLiveness/3 rw=true pri=0.00458444 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6491 orig=1.800000125,6491 max=1.800000125,6495 wto=false rop=false seq=4
W180728 06:26:12.183704 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:12.195331 59179 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=ae011cac key=/System/NodeLiveness/3 rw=true pri=0.02415311 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6570 orig=1.800000125,6570 max=1.800000125,6571 wto=false rop=false seq=4
I180728 06:26:12.547956 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:12.640918 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:12.646703 59181 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=b7acc444 key=/System/NodeLiveness/3 rw=true pri=0.04926058 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6654 orig=1.800000125,6654 max=1.800000125,6656 wto=false rop=false seq=4
W180728 06:26:13.093363 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:13.106461 59161 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=43ba422c key=/System/NodeLiveness/3 rw=true pri=0.02508662 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6735 orig=1.800000125,6735 max=1.800000125,6736 wto=false rop=false seq=4
W180728 06:26:13.546288 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:13.548417 57843 storage/client_test.go:942  SucceedsSoon: node not live
I180728 06:26:13.554178 59164 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=b3d0f692 key=/System/NodeLiveness/3 rw=true pri=0.02128137 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6817 orig=1.800000125,6817 max=1.800000125,6818 wto=false rop=false seq=4
W180728 06:26:13.999382 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:14.005357 58543 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=8fe52326 key=/System/NodeLiveness/3 rw=true pri=0.09317030 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6895 orig=1.800000125,6895 max=1.800000125,6897 wto=false rop=false seq=4
W180728 06:26:14.455344 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:14.458626 59208 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=a57731e6 key=/System/NodeLiveness/3 rw=true pri=0.04551299 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,6980 orig=1.800000125,6980 max=1.800000125,6981 wto=false rop=false seq=4
I180728 06:26:14.549111 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:14.908960 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:14.917684 59212 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=c346a91c key=/System/NodeLiveness/3 rw=true pri=0.03571883 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7059 orig=1.800000125,7059 max=1.800000125,7061 wto=false rop=false seq=4
W180728 06:26:15.362859 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:15.365324 59151 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=fc55d073 key=/System/NodeLiveness/3 rw=true pri=0.04842146 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7135 orig=1.800000125,7135 max=1.800000125,7137 wto=false rop=false seq=4
I180728 06:26:15.549614 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:15.817813 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:15.818581 59267 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=34975dc4 key=/System/NodeLiveness/3 rw=true pri=0.02560955 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7218 orig=1.800000125,7218 max=1.800000125,7220 wto=false rop=false seq=4
W180728 06:26:16.270451 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:16.274353 59227 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=19a46a1e key=/System/NodeLiveness/3 rw=true pri=0.02532690 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7298 orig=1.800000125,7298 max=1.800000125,7299 wto=false rop=false seq=4
I180728 06:26:16.556720 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:16.723198 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:16.729334 59240 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=59104e5e key=/System/NodeLiveness/3 rw=true pri=0.04287530 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7374 orig=1.800000125,7374 max=1.800000125,7376 wto=false rop=false seq=4
W180728 06:26:17.175428 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:17.178208 59244 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=93dce0ba key=/System/NodeLiveness/3 rw=true pri=0.02369271 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7458 orig=1.800000125,7458 max=1.800000125,7459 wto=false rop=false seq=4
I180728 06:26:17.557189 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:17.628477 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:17.637241 59233 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=8437a35d key=/System/NodeLiveness/3 rw=true pri=0.01398027 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7537 orig=1.800000125,7537 max=1.800000125,7538 wto=false rop=false seq=4
W180728 06:26:18.080920 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:18.093822 59299 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=61e88f30 key=/System/NodeLiveness/3 rw=true pri=0.02489621 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7618 orig=1.800000125,7618 max=1.800000125,7619 wto=false rop=false seq=4
W180728 06:26:18.533163 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:18.536603 59259 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=82d31695 key=/System/NodeLiveness/3 rw=true pri=0.01463229 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7692 orig=1.800000125,7692 max=1.800000125,7694 wto=false rop=false seq=4
I180728 06:26:18.557645 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:18.985930 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:18.991966 59315 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=d666d979 key=/System/NodeLiveness/3 rw=true pri=0.05023242 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7776 orig=1.800000125,7776 max=1.800000125,7777 wto=false rop=false seq=4
W180728 06:26:19.449979 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:19.452066 59305 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=2422917b key=/System/NodeLiveness/3 rw=true pri=0.04368294 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7854 orig=1.800000125,7854 max=1.800000125,7855 wto=false rop=false seq=4
I180728 06:26:19.558373 57843 storage/client_test.go:942  SucceedsSoon: node not live
W180728 06:26:19.902675 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:19.905970 59265 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=36bbc3c3 key=/System/NodeLiveness/3 rw=true pri=0.02247174 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,7934 orig=1.800000125,7934 max=1.800000125,7936 wto=false rop=false seq=4
W180728 06:26:20.355345 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: context deadline exceeded
I180728 06:26:20.357851 59347 internal/client/txn.go:721  async rollback failed: TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND): "unnamed" id=d971d6b6 key=/System/NodeLiveness/3 rw=true pri=0.00434108 iso=SERIALIZABLE stat=PENDING epo=0 ts=1.800000125,8011 orig=1.800000125,8011 max=1.800000125,8014 wto=false rop=false seq=4
I180728 06:26:20.559406 59334 util/stop/stopper.go:537  quiescing; tasks left:
1      [async] kv.TxnCoordSender: heartbeat loop
I180728 06:26:20.560118 59332 util/stop/stopper.go:537  quiescing; tasks left:
1      [async] monitoring pusher txn
W180728 06:26:20.624211 58500 internal/client/txn.go:612  [hb] failure aborting transaction: node unavailable; try another peer; abort caused by: failed to send RPC: sending to all 3 replicas failed; last error: (err: node unavailable; try another peer) <nil>
W180728 06:26:20.624710 58500 storage/node_liveness.go:490  [hb] failed node liveness heartbeat: failed to send RPC: sending to all 3 replicas failed; last error: (err: node unavailable; try another peer) <nil>
W180728 06:26:20.644546 58262 storage/raft_transport.go:575  while processing outgoing Raft queue to node 1: EOF:
W180728 06:26:20.645534 58202 gossip/gossip.go:1325  [n3] no incoming or outgoing connections
    --- FAIL: TestSnapshotAfterTruncation/differentTerm (46.12s)
    	soon.go:49: condition failed to evaluate within 45s: node not live
    		goroutine 57843 [running]:
    		runtime/debug.Stack(0xa7a358200, 0xc4212e4f30, 0x394f8c0)
    			/usr/local/go/src/runtime/debug/stack.go:24 +0xb5
    		github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x39aab40, 0xc420229950, 0xc42043af60)
    			/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:50 +0x172
    		github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContext).restartStore(0xc42211e680, 0x2)
    			/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:942 +0x160
    		github.com/cockroachdb/cockroach/pkg/storage_test.TestSnapshotAfterTruncation.func1(0xc420229950)
    			/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:713 +0x9f9
    		testing.tRunner(0xc420229950, 0xc4202e5370)
    			/usr/local/go/src/testing/testing.go:777 +0x16e
    		created by testing.(*T).Run
    			/usr/local/go/src/testing/testing.go:824 +0x565
@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone Jul 28, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jul 28, 2018
@tbg tbg added the A-kv-client Relating to the KV client and the KV interface. label Jul 31, 2018
@andreimatei
Copy link
Contributor

Stressing...

FWIW, don't be fooled by the flurry of TransactionStatusError: txn record not found (REASON_TXN_NOT_FOUND). That's expected and benign after the server started quiescing. But I'll send a PR to remove the spam.

@andreimatei
Copy link
Contributor

andreimatei commented Aug 3, 2018 •
edited
Loading

Reproduces at the failure commit above on GCE in a few min with

make stressrace PKG='./pkg/storage' TESTS=TestSnapshotAfterTruncation STRESSFLAGS='-p 30'

    --- FAIL: TestSnapshotAfterTruncation/differentTerm (46.57s)
        soon.go:49: condition failed to evaluate within 45s: node not live
                goroutine 1238 [running]:
                runtime/debug.Stack(0xa7a358200, 0xc420ac62d0, 0x37f8140)
                        /usr/local/go/src/runtime/debug/stack.go:24 +0xb5
                github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x38533c0, 0xc42027f860, 0xc4203204c0)
                        /home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:50 +0x172
                github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContext).restartStore(0xc420922820, 0x2)
                        /home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:942 +0x160
                github.com/cockroachdb/cockroach/pkg/storage_test.TestSnapshotAfterTruncation.func1(0xc42027f860)
                        /home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:713 +0x9f9
                testing.tRunner(0xc42027f860, 0xc4203e3180)
                        /usr/local/go/src/testing/testing.go:777 +0x16e
                created by testing.(*T).Run
                        /usr/local/go/src/testing/testing.go:824 +0x565

1366 runs completed, 1 failures, over 3m52s

@andreimatei
Copy link
Contributor

Doesn't repro at master in over 1h. Repros consistently in a few minutes at the old commit. Calling it fixed.

@andreimatei
Copy link
Contributor

andreimatei commented Aug 3, 2018 •
edited
Loading

Shit, it did repro on master in 1.5h.

make stressrace PKG='./pkg/storage' TESTS=TestSnapshotAfterTruncation STRESSFLAGS='-p 20'

--- FAIL: TestSnapshotAfterTruncation (47.63s)
    --- PASS: TestSnapshotAfterTruncation/sameTerm (1.16s)
    --- FAIL: TestSnapshotAfterTruncation/differentTerm (46.46s)
        soon.go:49: condition failed to evaluate within 45s: node not live
                goroutine 918 [running]:
                runtime/debug.Stack(0xa7a358200, 0xc420784510, 0x3856180)
                        /usr/local/go/src/runtime/debug/stack.go:24 +0xb5
                github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x38b27e0, 0xc420d881e0, 0xc420369540)
                        /home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:50 +0x172
                github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContext).restartStore(0xc420d4a820, 0x2)
                        /home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:942 +0x160

@andreimatei andreimatei reopened this Aug 3, 2018
@tbg
Copy link
Member

tbg commented Sep 20, 2018

Hasn't reproduced in nightly runs in a long time, closing.

@tbg tbg closed this as completed Sep 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jordanlewis jordanlewis

@andreimatei andreimatei

Labels
A-kv-client Relating to the KV client and the KV interface. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.1
Development

No branches or pull requests
4 participants
@jordanlewis @andreimatei @tbg @cockroach-teamcity