teamcity: failed tests on master: test/TestBackupRestoreControlJob #24136

cockroach-teamcity · 2018-03-21T23:56:27Z

The following tests appear to have failed:

#581903:

--- FAIL: test/TestBackupRestoreControlJob/foreign (0.000s)
Test ended in panic.

------- Stdout: -------
I180321 23:32:41.987219 33877 storage/replica.go:3218  [n2,s2,r35/3:/Table/51/1/{890-900}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n2,s2):3] next=4
I180321 23:32:41.988184 30387 storage/store.go:3543  [n1,s1,r35/1:/Table/51/1/{890-900}] added to replica GC queue (peer suggestion)
I180321 23:32:41.988601 30968 sql/event_log.go:121  [n1,client=127.0.0.1:50132,user=root] Event: "create_database", target: 52, info: {DatabaseName:orig_fkdb Statement:CREATE DATABASE orig_fkdb User:root}
E180321 23:32:41.995330 33872 storage/queue.go:778  [replicate,n2,s2,r101/2:/Table/51/1/9{00-10}] no removable replicas from range that needs a removal: [raft progress unknown]
E180321 23:32:42.001794 33902 storage/queue.go:778  [replicate,n2,s2,r35/3:/Table/51/1/{890-900}] no removable replicas from range that needs a removal: [3*:30]
I180321 23:32:42.008264 33878 storage/store.go:2414  [replicaGC,n1,s1,r35/1:/Table/51/1/{890-900}] removing replica
I180321 23:32:42.008610 33878 storage/replica.go:836  [replicaGC,n1,s1,r35/1:/Table/51/1/{890-900}] removed 19 (10+9) keys in 0ms [clear=0ms commit=0ms]
I180321 23:32:42.011606 30768 sql/event_log.go:121  [n1,client=127.0.0.1:50120,user=root] Event: "create_database", target: 53, info: {DatabaseName:restore_fkdb Statement:CREATE DATABASE restore_fkdb User:root}
I180321 23:32:42.022549 33839 storage/replica_command.go:1784  [replicate,n2,s2,r22/3:/Table/51{-/1/20}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r22:/Table/51{-/1/20} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4]
I180321 23:32:42.024204 30968 sql/event_log.go:121  [n1,client=127.0.0.1:50132,user=root] Event: "create_table", target: 54, info: {TableName:orig_fkdb.public.fk Statement:CREATE TABLE orig_fkdb.public.fk (i INT, FOREIGN KEY (i) REFERENCES data.public.bank) User:root}
I180321 23:32:42.032363 33964 storage/replica.go:3218  [n2,s2,r22/3:/Table/51{-/1/20}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n2,s2):3 (n3,s3):2] next=4
I180321 23:32:42.036265 30047 storage/replica_proposal.go:201  [n3,s3,r22/2:/Table/51{-/1/20}] new range lease repl=(n3,s3):2 seq=4 start=1521675162.033379490,0 epo=1 pro=1521675162.033382350,0 following repl=(n2,s2):3 seq=3 start=1521675161.346301561,0 epo=1 pro=1521675161.346303927,0
I180321 23:32:42.036511 30387 storage/store.go:3543  [n1,s1,r22/1:/Table/51{-/1/20}] added to replica GC queue (peer suggestion)
I180321 23:32:42.037205 34004 storage/store.go:2414  [replicaGC,n1,s1,r22/1:/Table/51{-/1/20}] removing replica
I180321 23:32:42.041432 34004 storage/replica.go:836  [replicaGC,n1,s1,r22/1:/Table/51{-/1/20}] removed 29 (20+9) keys in 0ms [clear=0ms commit=0ms]
I180321 23:32:42.043270 30968 sql/lease.go:352  [n1,client=127.0.0.1:50132,user=root] publish: descID=51 (bank) version=2 mtime=2018-03-21 23:32:42.043089554 +0000 UTC
I180321 23:32:42.055407 30968 sql/lease.go:275  publish (count leases): descID=51 name=bank version=1 count=1
I180321 23:32:42.055546 34025 sql/lease.go:812  new lease: 51("bank") ver=2:1521675466.868641019,0, refcount=0
I180321 23:32:42.088440 30968 sql/lease.go:352  [n1,client=127.0.0.1:50132,user=root] publish: descID=54 (fk) version=2 mtime=2018-03-21 23:32:42.088281721 +0000 UTC
I180321 23:32:42.091540 30968 sql/lease.go:352  [n1,client=127.0.0.1:50132,user=root] publish: descID=54 (fk) version=3 mtime=2018-03-21 23:32:42.091421417 +0000 UTC
I180321 23:32:42.165087 33887 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r47/1:/{Table/51/1/9…-Max}] generated Raft snapshot 91053bd9 at index 10
E180321 23:32:42.165851 33887 storage/queue.go:778  [raftsnapshot,n1,s1,r47/1:/{Table/51/1/9…-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r47: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.167590 34086 storage/replica_command.go:862  [n1,s1,r47/1:/{Table/51/1/9…-Max}] initiating a split of this range at key /Table/54/1/0 [r80]
I180321 23:32:42.172456 34096 storage/replica_command.go:1784  [replicate,n1,s1,r47/1:/Table/5{1/1/970-4/1/0}] change replicas (REMOVE_REPLICA (n3,s3):2): read existing descriptor r47:/Table/5{1/1/970-4/1/0} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4]
I180321 23:32:42.176435 34106 storage/replica.go:3218  [n1,s1,r47/1:/Table/5{1/1/970-4/1/0}] proposing REMOVE_REPLICA((n3,s3):2): updated=[(n1,s1):1 (n2,s2):3] next=4
I180321 23:32:42.176785 33990 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r80/1:/{Table/54/1/0-Max}] generated Raft snapshot 9dafa113 at index 11
I180321 23:32:42.177553 33993 storage/replica_command.go:1784  [replicate,n1,s1,r47/1:/Table/5{1/1/970-4/1/0}] change replicas (REMOVE_REPLICA (n2,s2):3): read existing descriptor r47:/Table/5{1/1/970-4/1/0} [(n1,s1):1, (n2,s2):3, next=4]
I180321 23:32:42.178695 34184 storage/replica_command.go:862  [n1,s1,r80/1:/{Table/54/1/0-Max}] initiating a split of this range at key /Table/54/1/1 [r81]
E180321 23:32:42.179047 33990 storage/queue.go:778  [raftsnapshot,n1,s1,r80/1:/{Table/54/1/0-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r80: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.181053 34111 storage/replica.go:3218  [n1,s1,r47/1:/Table/5{1/1/970-4/1/0}] proposing REMOVE_REPLICA((n2,s2):3): updated=[(n1,s1):1] next=4
E180321 23:32:42.190064 34167 storage/queue.go:778  [replicate,n1,s1,r47/1:/Table/5{1/1/970-4/1/0}] no removable replicas from range that needs a removal: [1*:22]
I180321 23:32:42.194498 30409 storage/store.go:3543  [n2,s2,r47/3:/Table/5{1/1/970-4/1/0}] added to replica GC queue (peer suggestion)
I180321 23:32:42.200887 34197 storage/store.go:2414  [replicaGC,n2,s2,r47/3:/Table/5{1/1/970-4/1/0}] removing replica
I180321 23:32:42.201378 34197 storage/replica.go:836  [replicaGC,n2,s2,r47/3:/Table/5{1/1/970-4/1/0}] removed 39 (30+9) keys in 0ms [clear=0ms commit=0ms]
I180321 23:32:42.206463 34161 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r81/1:/{Table/54/1/1-Max}] generated Raft snapshot 5f6904a1 at index 10
E180321 23:32:42.207509 34161 storage/queue.go:778  [raftsnapshot,n1,s1,r81/1:/{Table/54/1/1-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r81: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.210316 34199 storage/replica_command.go:862  [n1,s1,r81/1:/{Table/54/1/1-Max}] initiating a split of this range at key /Table/54/1/2 [r82]
I180321 23:32:42.221281 33999 storage/replica_command.go:1784  [replicate,n2,s2,r36/3:/Table/51/1/1{20-40}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r36:/Table/51/1/1{20-40} [(n1,s1):1, (n2,s2):3, next=4]
I180321 23:32:42.222489 34227 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r82/1:/{Table/54/1/2-Max}] generated Raft snapshot 6085947b at index 10
E180321 23:32:42.223115 34227 storage/queue.go:778  [raftsnapshot,n1,s1,r82/1:/{Table/54/1/2-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r82: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.228463 34121 storage/replica_command.go:862  [n1,s1,r82/1:/{Table/54/1/2-Max}] initiating a split of this range at key /Table/54/1/3 [r83]
I180321 23:32:42.230523 34270 storage/replica.go:3218  [n2,s2,r36/3:/Table/51/1/1{20-40}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n2,s2):3] next=4
E180321 23:32:42.231852 34247 storage/queue.go:778  [replicate,n2,s2,r36/3:/Table/51/1/1{20-40}] no removable replicas from range that needs a removal: [3*:26]
I180321 23:32:42.234617 30387 storage/store.go:3543  [n1,s1,r36/1:/Table/51/1/1{20-40}] added to replica GC queue (peer suggestion)
I180321 23:32:42.235110 34252 storage/store.go:2414  [replicaGC,n1,s1,r36/1:/Table/51/1/1{20-40}] removing replica
I180321 23:32:42.237652 34252 storage/replica.go:836  [replicaGC,n1,s1,r36/1:/Table/51/1/1{20-40}] removed 30 (20+10) keys in 0ms [clear=0ms commit=0ms]
I180321 23:32:42.265540 34287 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r83/1:/{Table/54/1/3-Max}] generated Raft snapshot 3330f1d6 at index 13
I180321 23:32:42.267928 34327 storage/replica_command.go:862  [n1,s1,r83/1:/{Table/54/1/3-Max}] initiating a split of this range at key /Table/54/1/4 [r84]
E180321 23:32:42.268461 34287 storage/queue.go:778  [raftsnapshot,n1,s1,r83/1:/{Table/54/1/3-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r83: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.284499 34320 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r84/1:/{Table/54/1/4-Max}] generated Raft snapshot 0b20a7cf at index 11
E180321 23:32:42.285141 34320 storage/queue.go:778  [raftsnapshot,n1,s1,r84/1:/{Table/54/1/4-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r84: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.286684 34387 storage/replica_command.go:862  [n1,s1,r84/1:/{Table/54/1/4-Max}] initiating a split of this range at key /Table/54/1/5 [r85]
I180321 23:32:42.306121 34238 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r85/1:/{Table/54/1/5-Max}] generated Raft snapshot 2f40a3ab at index 11
E180321 23:32:42.306944 34238 storage/queue.go:778  [raftsnapshot,n1,s1,r85/1:/{Table/54/1/5-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r85: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.309036 34239 storage/replica_command.go:862  [n1,s1,r85/1:/{Table/54/1/5-Max}] initiating a split of this range at key /Table/54/1/6 [r86]
I180321 23:32:42.322463 34421 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r86/1:/{Table/54/1/6-Max}] generated Raft snapshot 7888ab87 at index 10
E180321 23:32:42.323185 34421 storage/queue.go:778  [raftsnapshot,n1,s1,r86/1:/{Table/54/1/6-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r86: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.331486 34350 storage/replica_command.go:862  [n1,s1,r86/1:/{Table/54/1/6-Max}] initiating a split of this range at key /Table/54/1/7 [r87]
I180321 23:32:42.358792 34444 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r87/1:/{Table/54/1/7-Max}] generated Raft snapshot 67a7afb1 at index 11
E180321 23:32:42.359685 34444 storage/queue.go:778  [raftsnapshot,n1,s1,r87/1:/{Table/54/1/7-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r87: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.360799 34365 storage/replica_command.go:862  [n1,s1,r87/1:/{Table/54/1/7-Max}] initiating a split of this range at key /Table/54/1/8 [r88]
I180321 23:32:42.371522 34378 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r88/1:/{Table/54/1/8-Max}] generated Raft snapshot 0796e9c1 at index 10
E180321 23:32:42.372047 34378 storage/queue.go:778  [raftsnapshot,n1,s1,r88/1:/{Table/54/1/8-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r88: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.375394 34534 storage/replica_command.go:862  [n1,s1,r88/1:/{Table/54/1/8-Max}] initiating a split of this range at key /Table/54/1/9 [r89]
I180321 23:32:42.392811 34585 ccl/storageccl/export.go:118  [n1,s1,r89/1:/{Table/54/1/9-Max}] export [/Table/54/2,/Table/54/3)
I180321 23:32:42.393361 34549 ccl/storageccl/export.go:118  [n1,s1,r86/1:/Table/54/1/{6-7}] export [/Table/54/1/6,/Table/54/1/7)
I180321 23:32:42.394802 34566 ccl/storageccl/export.go:118  [n1,s1,r88/1:/Table/54/1/{8-9}] export [/Table/54/1/8,/Table/54/1/9)
I180321 23:32:42.393557 34563 ccl/storageccl/export.go:118  [n1,s1,r84/1:/Table/54/1/{4-5}] export [/Table/54/1/4,/Table/54/1/5)
I180321 23:32:42.393620 34426 ccl/storageccl/export.go:118  [n1,s1,r85/1:/Table/54/1/{5-6}] export [/Table/54/1/5,/Table/54/1/6)
I180321 23:32:42.393681 34564 ccl/storageccl/export.go:118  [n1,s1,r80/1:/Table/54/1/{0-1}] export [/Table/54/1/0,/Table/54/1/1)
I180321 23:32:42.395040 34567 ccl/storageccl/export.go:118  [n1,s1,r87/1:/Table/54/1/{7-8}] export [/Table/54/1/7,/Table/54/1/8)
I180321 23:32:42.395043 34429 ccl/storageccl/export.go:118  [n1,s1,r89/1:/{Table/54/1/9-Max}] export [/Table/54/1/9,/Table/54/2)
I180321 23:32:42.395171 34565 ccl/storageccl/export.go:118  [n1,s1,r47/1:/Table/5{1/1/970-4/1/0}] export [/Table/54/1,/Table/54/1/0)
I180321 23:32:42.395350 34428 ccl/storageccl/export.go:118  [n1,s1,r82/1:/Table/54/1/{2-3}] export [/Table/54/1/2,/Table/54/1/3)
I180321 23:32:42.395536 34427 ccl/storageccl/export.go:118  [n1,s1,r81/1:/Table/54/1/{1-2}] export [/Table/54/1/1,/Table/54/1/2)
I180321 23:32:42.397518 34504 ccl/storageccl/export.go:118  [n1,s1,r83/1:/Table/54/1/{3-4}] export [/Table/54/1/3,/Table/54/1/4)
I180321 23:32:42.503685 34633 ccl/storageccl/export.go:118  [n1,s1,r80/1:/Table/54/1/{0-1}] export [/Table/54/1/0,/Table/54/1/1)
I180321 23:32:42.506262 34786 ccl/storageccl/export.go:118  [n1,s1,r81/1:/Table/54/1/{1-2}] export [/Table/54/1/1,/Table/54/1/2)
I180321 23:32:42.506644 34749 ccl/storageccl/export.go:118  [n1,s1,r88/1:/Table/54/1/{8-9}] export [/Table/54/1/8,/Table/54/1/9)
I180321 23:32:42.506904 34750 ccl/storageccl/export.go:118  [n1,s1,r83/1:/Table/54/1/{3-4}] export [/Table/54/1/3,/Table/54/1/4)
I180321 23:32:42.507120 34751 ccl/storageccl/export.go:118  [n1,s1,r84/1:/Table/54/1/{4-5}] export [/Table/54/1/4,/Table/54/1/5)
I180321 23:32:42.507337 34752 ccl/storageccl/export.go:118  [n1,s1,r87/1:/Table/54/1/{7-8}] export [/Table/54/1/7,/Table/54/1/8)
I180321 23:32:42.508855 34779 ccl/storageccl/export.go:118  [n1,s1,r82/1:/Table/54/1/{2-3}] export [/Table/54/1/2,/Table/54/1/3)
I180321 23:32:42.510144 34777 ccl/storageccl/export.go:118  [n1,s1,r86/1:/Table/54/1/{6-7}] export [/Table/54/1/6,/Table/54/1/7)
I180321 23:32:42.510481 34778 ccl/storageccl/export.go:118  [n1,s1,r85/1:/Table/54/1/{5-6}] export [/Table/54/1/5,/Table/54/1/6)
I180321 23:32:42.512785 34753 ccl/storageccl/export.go:118  [n1,s1,r47/1:/Table/5{1/1/970-4/1/0}] export [/Table/54/1,/Table/54/1/0)
I180321 23:32:42.514664 34654 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r89/1:/{Table/54/1/9-Max}] generated Raft snapshot 0fd58aa4 at index 11
E180321 23:32:42.515326 34654 storage/queue.go:778  [raftsnapshot,n1,s1,r89/1:/{Table/54/1/9-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r89: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.551366 34816 ccl/storageccl/export.go:118  [n1,s1,r89/1:/{Table/54/1/9-Max}] export [/Table/54/1/9,/Table/54/2)
I180321 23:32:42.557708 34821 ccl/storageccl/export.go:118  [n1,s1,r89/1:/{Table/54/1/9-Max}] export [/Table/54/2,/Table/54/3)
I180321 23:32:42.623813 30277 storage/replica_proposal.go:201  [n3,s3,r25/2:/Table/51/1/{20-40}] new range lease repl=(n3,s3):2 seq=4 start=1521675162.622041447,0 epo=1 pro=1521675162.622044715,0 following repl=(n2,s2):3 seq=3 start=1521675161.298111251,0 epo=1 pro=1521675161.298113954,0
I180321 23:32:42.896201 35265 storage/replica_command.go:862  [n1,s1,r89/1:/{Table/54/1/9-Max}] initiating a split of this range at key /Table/55 [r90]
I180321 23:32:42.907731 35272 storage/replica_command.go:862  [n1,s1,r90/1:/{Table/55-Max}] initiating a split of this range at key /Table/55/1/9 [r111]
I180321 23:32:42.910922 35191 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r90/1:/{Table/55-Max}] generated Raft snapshot 9d9855ec at index 10
E180321 23:32:42.913362 35191 storage/queue.go:778  [raftsnapshot,n1,s1,r90/1:/{Table/55-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r90: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:42.923930 35348 storage/replica_command.go:862  [n1,s1,r111/1:/{Table/55/1/9-Max}] initiating a split of this range at key /Table/55/2 [r112]
I180321 23:32:42.933593 35297 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r111/1:/{Table/55/1/9-Max}] generated Raft snapshot def89757 at index 11
E180321 23:32:42.939209 35297 storage/queue.go:778  [raftsnapshot,n1,s1,r111/1:/{Table/55/1/9-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r111: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
W180321 23:32:42.940699 35348 internal/client/txn.go:531  [n1,s1,r111/1:/{Table/55/1/9-Max}] failure aborting transaction: context canceled; abort caused by: result is ambiguous (context canceled)
I180321 23:32:43.033522 29535 storage/replica_proposal.go:201  [n1,s1,r53/1:/Table/51/1/3{20-40}] new range lease repl=(n1,s1):1 seq=4 start=1521675163.022930033,0 epo=1 pro=1521675163.022933299,0 following repl=(n2,s2):3 seq=3 start=1521675161.598868379,0 epo=1 pro=1521675161.598870832,0
I180321 23:32:43.068566 35532 storage/replica_command.go:862  [n1,s1,r111/1:/{Table/55/1/9-Max}] initiating a split of this range at key /Table/55/2 [r113]
I180321 23:32:43.092215 35593 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r113/1:/{Table/55/2-Max}] generated Raft snapshot bf23b360 at index 11
E180321 23:32:43.092928 35593 storage/queue.go:778  [raftsnapshot,n1,s1,r113/1:/{Table/55/2-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r113: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.153459 30305 storage/replica_proposal.go:201  [n3,s3,r5/3:/System/ts{d-e}] new range lease repl=(n3,s3):3 seq=3 start=1521675163.151722925,0 epo=1 pro=1521675163.151726002,0 following repl=(n1,s1):1 seq=2 start=1521675160.110204534,0 exp=1521675169.111067582,0 pro=1521675160.111134059,0
--- FAIL: test/TestBackupRestoreControlJob/pause (0.000s)
Test ended in panic.

------- Stdout: -------
I180321 23:32:43.164624 30768 sql/event_log.go:121  [n1,client=127.0.0.1:50120,user=root] Event: "create_database", target: 56, info: {DatabaseName:pause Statement:CREATE DATABASE pause User:root}
I180321 23:32:43.181235 35645 ccl/storageccl/export.go:118  [n2,s2,r36/3:/Table/51/1/1{20-40}] export [/Table/51/1/120,/Table/51/1/140)
I180321 23:32:43.181897 35811 ccl/storageccl/export.go:118  [n3,s3,r25/2:/Table/51/1/{20-40}] export [/Table/51/1/20,/Table/51/1/40)
I180321 23:32:43.182172 35779 ccl/storageccl/export.go:118  [n2,s2,r28/3:/Table/51/1/{40-60}] export [/Table/51/1/40,/Table/51/1/60)
I180321 23:32:43.183709 35785 ccl/storageccl/export.go:118  [n3,s3,r32/2:/Table/51/1/{80-100}] export [/Table/51/1/80,/Table/51/1/100)
I180321 23:32:43.182560 35754 ccl/storageccl/export.go:118  [n2,s2,r41/3:/Table/51/1/1{60-80}] export [/Table/51/1/160,/Table/51/1/180)
I180321 23:32:43.182669 35752 ccl/storageccl/export.go:118  [n2,s2,r44/3:/Table/51/1/2{00-20}] export [/Table/51/1/200,/Table/51/1/220)
I180321 23:32:43.184580 35810 ccl/storageccl/export.go:118  [n3,s3,r22/2:/Table/51{-/1/20}] export [/Table/51/1,/Table/51/1/20)
I180321 23:32:43.182733 35782 ccl/storageccl/export.go:118  [n2,s2,r39/3:/Table/51/1/1{40-60}] export [/Table/51/1/140,/Table/51/1/160)
I180321 23:32:43.186014 35826 ccl/storageccl/export.go:118  [n2,s2,r49/3:/Table/51/1/2{60-80}] export [/Table/51/1/260,/Table/51/1/280)
I180321 23:32:43.188338 35766 ccl/storageccl/export.go:118  [n2,s2,r30/3:/Table/51/1/{60-80}] export [/Table/51/1/60,/Table/51/1/80)
I180321 23:32:43.188373 35760 ccl/storageccl/export.go:118  [n2,s2,r46/3:/Table/51/1/2{20-40}] export [/Table/51/1/220,/Table/51/1/240)
I180321 23:32:43.188511 35807 ccl/storageccl/export.go:118  [n2,s2,r50/3:/Table/51/1/{280-300}] export [/Table/51/1/280,/Table/51/1/300)
I180321 23:32:43.188873 35753 ccl/storageccl/export.go:118  [n2,s2,r42/3:/Table/51/1/{180-200}] export [/Table/51/1/180,/Table/51/1/200)
I180321 23:32:43.189470 35759 ccl/storageccl/export.go:118  [n2,s2,r48/3:/Table/51/1/2{40-60}] export [/Table/51/1/240,/Table/51/1/260)
I180321 23:32:43.190671 35756 ccl/storageccl/export.go:118  [n2,s2,r34/3:/Table/51/1/1{00-20}] export [/Table/51/1/100,/Table/51/1/120)
I180321 23:32:43.191886 35726 ccl/storageccl/export.go:118  [n2,s2,r52/3:/Table/51/1/3{00-20}] export [/Table/51/1/300,/Table/51/1/320)
I180321 23:32:43.222595 35866 ccl/storageccl/export.go:118  [n1,s1,r64/1:/Table/51/1/5{00-20}] export [/Table/51/1/500,/Table/51/1/520)
I180321 23:32:43.225027 35936 ccl/storageccl/export.go:118  [n2,s2,r65/3:/Table/51/1/5{20-40}] export [/Table/51/1/520,/Table/51/1/540)
I180321 23:32:43.225678 35908 storage/replica_command.go:1784  [replicate,n2,s2,r42/3:/Table/51/1/{180-200}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r42:/Table/51/1/{180-200} [(n1,s1):1, (n2,s2):3, next=4]
I180321 23:32:43.254883 35919 storage/replica.go:3218  [n2,s2,r42/3:/Table/51/1/{180-200}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n2,s2):3] next=4
E180321 23:32:43.269724 35846 storage/queue.go:778  [replicate,n2,s2,r42/3:/Table/51/1/{180-200}] no removable replicas from range that needs a removal: [3*:26]
I180321 23:32:43.272151 36034 storage/store.go:2414  [replicaGC,n1,s1,r42/1:/Table/51/1/{180-200}] removing replica
I180321 23:32:43.272538 36034 storage/replica.go:836  [replicaGC,n1,s1,r42/1:/Table/51/1/{180-200}] removed 30 (20+10) keys in 0ms [clear=0ms commit=0ms]
I180321 23:32:43.286829 35983 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r83/1:/Table/54/1/{3-4}] generated Raft snapshot afaa94ee at index 17
E180321 23:32:43.288244 35983 storage/queue.go:778  [raftsnapshot,n1,s1,r83/1:/Table/54/1/{3-4}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r83: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.290968 36040 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r84/1:/Table/54/1/{4-5}] generated Raft snapshot 21bdf664 at index 17
E180321 23:32:43.291579 36040 storage/queue.go:778  [raftsnapshot,n1,s1,r84/1:/Table/54/1/{4-5}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r84: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.291941 36044 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r82/1:/Table/54/1/{2-3}] generated Raft snapshot 3cf79ddc at index 17
E180321 23:32:43.292825 36044 storage/queue.go:778  [raftsnapshot,n1,s1,r82/1:/Table/54/1/{2-3}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r82: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.293388 36048 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r85/1:/Table/54/1/{5-6}] generated Raft snapshot d2d6e7a9 at index 17
E180321 23:32:43.293955 36048 storage/queue.go:778  [raftsnapshot,n1,s1,r85/1:/Table/54/1/{5-6}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r85: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.294345 35985 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r81/1:/Table/54/1/{1-2}] generated Raft snapshot 04d1481c at index 17
E180321 23:32:43.294849 35985 storage/queue.go:778  [raftsnapshot,n1,s1,r81/1:/Table/54/1/{1-2}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r81: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.295192 36162 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r80/1:/Table/54/1/{0-1}] generated Raft snapshot dbc2a107 at index 18
E180321 23:32:43.295934 36162 storage/queue.go:778  [raftsnapshot,n1,s1,r80/1:/Table/54/1/{0-1}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r80: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.309697 35854 ccl/storageccl/export.go:118  [n3,s3,r22/2:/Table/51{-/1/20}] export [/Table/51/1,/Table/51/1/20)
I180321 23:32:43.312376 36104 ccl/storageccl/export.go:118  [n2,s2,r42/3:/Table/51/1/{180-200}] export [/Table/51/1/180,/Table/51/1/200)
I180321 23:32:43.314926 36270 ccl/storageccl/export.go:118  [n2,s2,r28/3:/Table/51/1/{40-60}] export [/Table/51/1/40,/Table/51/1/60)
I180321 23:32:43.315678 36280 ccl/storageccl/export.go:118  [n2,s2,r52/3:/Table/51/1/3{00-20}] export [/Table/51/1/300,/Table/51/1/320)
I180321 23:32:43.315736 36298 ccl/storageccl/export.go:118  [n2,s2,r50/3:/Table/51/1/{280-300}] export [/Table/51/1/280,/Table/51/1/300)
I180321 23:32:43.317627 36277 ccl/storageccl/export.go:118  [n2,s2,r49/3:/Table/51/1/2{60-80}] export [/Table/51/1/260,/Table/51/1/280)
I180321 23:32:43.318109 36156 ccl/storageccl/export.go:118  [n3,s3,r25/2:/Table/51/1/{20-40}] export [/Table/51/1/20,/Table/51/1/40)
I180321 23:32:43.318760 36278 ccl/storageccl/export.go:118  [n2,s2,r44/3:/Table/51/1/2{00-20}] export [/Table/51/1/200,/Table/51/1/220)
I180321 23:32:43.318769 36127 ccl/storageccl/export.go:118  [n1,s1,r53/1:/Table/51/1/3{20-40}] export [/Table/51/1/320,/Table/51/1/340)
I180321 23:32:43.320963 36315 ccl/storageccl/export.go:118  [n2,s2,r41/3:/Table/51/1/1{60-80}] export [/Table/51/1/160,/Table/51/1/180)
I180321 23:32:43.321732 36275 ccl/storageccl/export.go:118  [n2,s2,r39/3:/Table/51/1/1{40-60}] export [/Table/51/1/140,/Table/51/1/160)
I180321 23:32:43.322312 36160 ccl/storageccl/export.go:118  [n3,s3,r32/2:/Table/51/1/{80-100}] export [/Table/51/1/80,/Table/51/1/100)
I180321 23:32:43.322988 36279 ccl/storageccl/export.go:118  [n2,s2,r48/3:/Table/51/1/2{40-60}] export [/Table/51/1/240,/Table/51/1/260)
I180321 23:32:43.323905 36316 ccl/storageccl/export.go:118  [n2,s2,r34/3:/Table/51/1/1{00-20}] export [/Table/51/1/100,/Table/51/1/120)
I180321 23:32:43.324584 36276 ccl/storageccl/export.go:118  [n2,s2,r46/3:/Table/51/1/2{20-40}] export [/Table/51/1/220,/Table/51/1/240)
I180321 23:32:43.326259 36330 ccl/storageccl/export.go:118  [n1,s1,r64/1:/Table/51/1/5{00-20}] export [/Table/51/1/500,/Table/51/1/520)
I180321 23:32:43.328278 36129 ccl/storageccl/export.go:118  [n2,s2,r54/3:/Table/51/1/3{40-60}] export [/Table/51/1/340,/Table/51/1/360)
I180321 23:32:43.328600 36300 ccl/storageccl/export.go:118  [n2,s2,r59/3:/Table/51/1/4{20-40}] export [/Table/51/1/420,/Table/51/1/440)
I180321 23:32:43.329162 36299 ccl/storageccl/export.go:118  [n2,s2,r65/3:/Table/51/1/5{20-40}] export [/Table/51/1/520,/Table/51/1/540)
I180321 23:32:43.330185 36402 ccl/storageccl/export.go:118  [n2,s2,r30/3:/Table/51/1/{60-80}] export [/Table/51/1/60,/Table/51/1/80)
I180321 23:32:43.331142 36419 ccl/storageccl/export.go:118  [n2,s2,r68/3:/Table/51/1/5{60-80}] export [/Table/51/1/560,/Table/51/1/580)
I180321 23:32:43.331781 36441 ccl/storageccl/export.go:118  [n2,s2,r60/3:/Table/51/1/4{40-60}] export [/Table/51/1/440,/Table/51/1/460)
I180321 23:32:43.333484 36469 ccl/storageccl/export.go:118  [n2,s2,r58/3:/Table/51/1/4{00-20}] export [/Table/51/1/400,/Table/51/1/420)
I180321 23:32:43.334001 36440 ccl/storageccl/export.go:118  [n2,s2,r66/3:/Table/51/1/5{40-60}] export [/Table/51/1/540,/Table/51/1/560)
I180321 23:32:43.334362 36468 ccl/storageccl/export.go:118  [n2,s2,r56/3:/Table/51/1/{380-400}] export [/Table/51/1/380,/Table/51/1/400)
I180321 23:32:43.336019 36483 ccl/storageccl/export.go:118  [n2,s2,r61/3:/Table/51/1/4{60-80}] export [/Table/51/1/460,/Table/51/1/480)
I180321 23:32:43.337979 36411 ccl/storageccl/export.go:118  [n2,s2,r55/3:/Table/51/1/3{60-80}] export [/Table/51/1/360,/Table/51/1/380)
I180321 23:32:43.338458 36457 ccl/storageccl/export.go:118  [n2,s2,r63/3:/Table/51/1/{480-500}] export [/Table/51/1/480,/Table/51/1/500)
I180321 23:32:43.339099 36511 ccl/storageccl/export.go:118  [n3,s3,r26/2:/Table/51/1/{790-800}] export [/Table/51/1/790,/Table/51/1/800)
I180321 23:32:43.339771 36512 ccl/storageccl/export.go:118  [n3,s3,r91/2:/Table/51/1/8{00-10}] export [/Table/51/1/800,/Table/51/1/810)
I180321 23:32:43.339981 36382 ccl/storageccl/export.go:118  [n2,s2,r23/3:/Table/51/1/7{70-90}] export [/Table/51/1/770,/Table/51/1/790)
I180321 23:32:43.340802 36521 ccl/storageccl/export.go:118  [n2,s2,r78/3:/Table/51/1/7{20-40}] export [/Table/51/1/720,/Table/51/1/740)
I180321 23:32:43.351480 36540 ccl/storageccl/export.go:118  [n2,s2,r75/3:/Table/51/1/{680-700}] export [/Table/51/1/680,/Table/51/1/700)
I180321 23:32:43.353399 36543 ccl/storageccl/export.go:118  [n2,s2,r77/3:/Table/51/1/7{00-20}] export [/Table/51/1/700,/Table/51/1/720)
I180321 23:32:43.353533 36474 ccl/storageccl/export.go:118  [n2,s2,r74/3:/Table/51/1/6{60-80}] export [/Table/51/1/660,/Table/51/1/680)
I180321 23:32:43.354433 36535 ccl/storageccl/export.go:118  [n2,s2,r71/3:/Table/51/1/6{20-40}] export [/Table/51/1/620,/Table/51/1/640)
I180321 23:32:43.354090 36557 ccl/storageccl/export.go:118  [n3,s3,r27/2:/Table/51/1/8{10-30}] export [/Table/51/1/810,/Table/51/1/830)
I180321 23:32:43.355276 36520 ccl/storageccl/export.go:118  [n2,s2,r70/3:/Table/51/1/6{00-20}] export [/Table/51/1/600,/Table/51/1/620)
I180321 23:32:43.358791 36579 ccl/storageccl/export.go:118  [n2,s2,r29/3:/Table/51/1/8{30-50}] export [/Table/51/1/830,/Table/51/1/850)
I180321 23:32:43.359998 36542 ccl/storageccl/export.go:118  [n2,s2,r69/3:/Table/51/1/{580-600}] export [/Table/51/1/580,/Table/51/1/600)
I180321 23:32:43.361208 36446 ccl/storageccl/export.go:118  [n2,s2,r33/3:/Table/51/1/8{70-90}] export [/Table/51/1/870,/Table/51/1/890)
I180321 23:32:43.362696 36612 ccl/storageccl/export.go:118  [n2,s2,r37/3:/Table/51/1/9{10-30}] export [/Table/51/1/910,/Table/51/1/930)
I180321 23:32:43.363838 36493 ccl/storageccl/export.go:118  [n1,s1,r47/1:/Table/5{1/1/970-4/1/0}] export [/Table/51/1/970,/Table/51/2)
I180321 23:32:43.364730 36613 ccl/storageccl/export.go:118  [n2,s2,r40/3:/Table/51/1/9{30-50}] export [/Table/51/1/930,/Table/51/1/950)
I180321 23:32:43.364948 36348 ccl/storageccl/export.go:118  [n2,s2,r43/3:/Table/51/1/9{50-70}] export [/Table/51/1/950,/Table/51/1/970)
I180321 23:32:43.366443 36541 ccl/storageccl/export.go:118  [n2,s2,r73/3:/Table/51/1/6{40-60}] export [/Table/51/1/640,/Table/51/1/660)
I180321 23:32:43.367276 36615 ccl/storageccl/export.go:118  [n2,s2,r31/3:/Table/51/1/8{50-70}] export [/Table/51/1/850,/Table/51/1/870)
I180321 23:32:43.421924 36717 ccl/storageccl/export.go:118  [n2,s2,r35/3:/Table/51/1/{890-900}] export [/Table/51/1/890,/Table/51/1/900)
I180321 23:32:43.425598 36733 ccl/storageccl/export.go:118  [n2,s2,r101/2:/Table/51/1/9{00-10}] export [/Table/51/1/900,/Table/51/1/910)
I180321 23:32:43.426772 29522 storage/replica_proposal.go:201  [n1,s1,r46/1:/Table/51/1/2{20-40}] new range lease repl=(n1,s1):1 seq=4 start=1521675163.424396157,0 epo=1 pro=1521675163.424398990,0 following repl=(n2,s2):3 seq=3 start=1521675161.554782126,0 epo=1 pro=1521675161.554785192,0
I180321 23:32:43.455857 36858 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r86/1:/Table/54/1/{6-7}] generated Raft snapshot ab6a8ca3 at index 17
E180321 23:32:43.456875 36858 storage/queue.go:778  [raftsnapshot,n1,s1,r86/1:/Table/54/1/{6-7}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r86: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.457390 36810 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r88/1:/Table/54/1/{8-9}] generated Raft snapshot 14b70ecf at index 17
E180321 23:32:43.458029 36810 storage/queue.go:778  [raftsnapshot,n1,s1,r88/1:/Table/54/1/{8-9}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r88: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.458458 36814 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r87/1:/Table/54/1/{7-8}] generated Raft snapshot 7e2ef70f at index 17
E180321 23:32:43.459185 36814 storage/queue.go:778  [raftsnapshot,n1,s1,r87/1:/Table/54/1/{7-8}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r87: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.656177 37128 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r89/1:/Table/5{4/1/9-5}] generated Raft snapshot 93d0d86a at index 15
E180321 23:32:43.656738 37128 storage/queue.go:778  [raftsnapshot,n1,s1,r89/1:/Table/5{4/1/9-5}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r89: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.856376 37167 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r90/1:/Table/55{-/1/9}] generated Raft snapshot f7178b0e at index 16
E180321 23:32:43.857057 37167 storage/queue.go:778  [raftsnapshot,n1,s1,r90/1:/Table/55{-/1/9}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r90: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.956052 37429 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r111/1:/Table/55/{1/9-2}] generated Raft snapshot afd6c8c7 at index 23
E180321 23:32:43.956813 37429 storage/queue.go:778  [raftsnapshot,n1,s1,r111/1:/Table/55/{1/9-2}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r111: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:43.957323 37418 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r113/1:/{Table/55/2-Max}] generated Raft snapshot 25be6586 at index 13
E180321 23:32:43.958014 37418 storage/queue.go:778  [raftsnapshot,n1,s1,r113/1:/{Table/55/2-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r113: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
E180321 23:32:44.090974 37486 storage/queue.go:778  [replicate,n3,s3,r26/2:/Table/51/1/{790-800}] no removable replicas from range that needs a removal: [2*:29]
I180321 23:32:44.163722 30294 storage/replica_proposal.go:201  [n3,s3,r13/3:/Table/1{6-7}] new range lease repl=(n3,s3):3 seq=3 start=1521675164.161342111,0 epo=1 pro=1521675164.161343864,0 following repl=(n1,s1):1 seq=2 start=1521675160.110204534,0 exp=1521675169.111067582,0 pro=1521675160.111134059,0
I180321 23:32:44.256398 37561 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r83/1:/Table/54/1/{3-4}] generated Raft snapshot 9d3e93fb at index 17
E180321 23:32:44.257045 37561 storage/queue.go:778  [raftsnapshot,n1,s1,r83/1:/Table/54/1/{3-4}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r83: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.257486 37511 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r81/1:/Table/54/1/{1-2}] generated Raft snapshot 13fb3fbc at index 17
E180321 23:32:44.258080 37511 storage/queue.go:778  [raftsnapshot,n1,s1,r81/1:/Table/54/1/{1-2}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r81: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.258485 37451 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r85/1:/Table/54/1/{5-6}] generated Raft snapshot ace20856 at index 17
E180321 23:32:44.259009 37451 storage/queue.go:778  [raftsnapshot,n1,s1,r85/1:/Table/54/1/{5-6}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r85: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.259753 37501 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r84/1:/Table/54/1/{4-5}] generated Raft snapshot 2f7c0690 at index 17
E180321 23:32:44.260328 37501 storage/queue.go:778  [raftsnapshot,n1,s1,r84/1:/Table/54/1/{4-5}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r84: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.260709 37505 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r80/1:/Table/54/1/{0-1}] generated Raft snapshot 71ab26bc at index 18
E180321 23:32:44.261188 37505 storage/queue.go:778  [raftsnapshot,n1,s1,r80/1:/Table/54/1/{0-1}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r80: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.261586 37573 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r82/1:/Table/54/1/{2-3}] generated Raft snapshot a32e7cdf at index 17
E180321 23:32:44.262078 37573 storage/queue.go:778  [raftsnapshot,n1,s1,r82/1:/Table/54/1/{2-3}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r82: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.456510 37620 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r87/1:/Table/54/1/{7-8}] generated Raft snapshot 89227493 at index 17
E180321 23:32:44.457278 37620 storage/queue.go:778  [raftsnapshot,n1,s1,r87/1:/Table/54/1/{7-8}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r87: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.457736 37636 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r86/1:/Table/54/1/{6-7}] generated Raft snapshot 3eeafc2a at index 17
E180321 23:32:44.459979 37636 storage/queue.go:778  [raftsnapshot,n1,s1,r86/1:/Table/54/1/{6-7}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r86: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.460383 37626 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r88/1:/Table/54/1/{8-9}] generated Raft snapshot 0ac3808e at index 17
E180321 23:32:44.460949 37626 storage/queue.go:778  [raftsnapshot,n1,s1,r88/1:/Table/54/1/{8-9}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r88: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.557072 37669 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r89/1:/Table/5{4/1/9-5}] generated Raft snapshot 73116907 at index 15
E180321 23:32:44.557990 37669 storage/queue.go:778  [raftsnapshot,n1,s1,r89/1:/Table/5{4/1/9-5}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r89: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:44.634266 29521 storage/replica_proposal.go:201  [n1,s1,r40/1:/Table/51/1/9{30-50}] new range lease repl=(n1,s1):1 seq=4 start=1521675164.631838028,0 epo=1 pro=1521675164.631841670,0 following repl=(n2,s2):3 seq=3 start=1521675161.493443468,0 epo=1 pro=1521675161.493446208,0
I180321 23:32:44.693048 37688 storage/replica_command.go:1784  [replicate,n3,s3,r25/2:/Table/51/1/{20-40}] change replicas (REMOVE_REPLICA (n2,s2):3): read existing descriptor r25:/Table/51/1/{20-40} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4]
I180321 23:32:44.736800 37746 storage/replica.go:3218  [n3,s3,r25/2:/Table/51/1/{20-40}] proposing REMOVE_REPLICA((n2,s2):3): updated=[(n1,s1):1 (n3,s3):2] next=4
I180321 23:32:44.738426 30994 storage/store.go:3543  [n2,s2,r25/3:/Table/51/1/{20-40}] added to replica GC queue (peer suggestion)
I180321 23:32:44.739018 30994 storage/store.go:3543  [n2,s2,r25/3:/Table/51/1/{20-40}] added to replica GC queue (peer suggestion)
I180321 23:32:44.740371 29515 storage/replica_proposal.go:201  [n1,s1,r25/1:/Table/51/1/{20-40}] new range lease repl=(n1,s1):1 seq=5 start=1521675164.738312292,0 epo=1 pro=1521675164.738315254,0 following repl=(n3,s3):2 seq=4 start=1521675162.622041447,0 epo=1 pro=1521675162.622044715,0
I180321 23:32:44.743545 37535 storage/store.go:2414  [replicaGC,n2,s2,r25/3:/Table/51/1/{20-40}] removing replica
I180321 23:32:44.744181 37535 storage/replica.go:836  [replicaGC,n2,s2,r25/3:/Table/51/1/{20-40}] removed 30 (20+10) keys in 0ms [clear=0ms commit=0ms]
I180321 23:32:44.836994 29537 storage/replica_proposal.go:201  [n1,s1,r48/1:/Table/51/1/2{40-60}] new range lease repl=(n1,s1):1 seq=4 start=1521675164.833943590,0 epo=1 pro=1521675164.833947253,0 following repl=(n2,s2):3 seq=3 start=1521675161.480623869,0 epo=1 pro=1521675161.480626665,0
I180321 23:32:44.856113 37784 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r90/1:/Table/55{-/1/9}] generated Raft snapshot 692b4f55 at index 16
E180321 23:32:44.856864 37784 storage/queue.go:778  [raftsnapshot,n1,s1,r90/1:/Table/55{-/1/9}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r90: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.035167 29529 storage/replica_proposal.go:201  [n1,s1,r52/1:/Table/51/1/3{00-20}] new range lease repl=(n1,s1):1 seq=4 start=1521675165.033980406,0 epo=1 pro=1521675165.033984425,0 following repl=(n2,s2):3 seq=3 start=1521675161.537089389,0 epo=1 pro=1521675161.537092169,0
I180321 23:32:45.057347 37825 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r111/1:/Table/55/{1/9-2}] generated Raft snapshot dade5b72 at index 23
E180321 23:32:45.058054 37825 storage/queue.go:778  [raftsnapshot,n1,s1,r111/1:/Table/55/{1/9-2}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r111: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.059908 37924 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r113/1:/{Table/55/2-Max}] generated Raft snapshot 0208f2c9 at index 13
E180321 23:32:45.060595 37924 storage/queue.go:778  [raftsnapshot,n1,s1,r113/1:/{Table/55/2-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r113: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.237575 29531 storage/replica_proposal.go:201  [n1,s1,r39/1:/Table/51/1/1{40-60}] new range lease repl=(n1,s1):1 seq=4 start=1521675165.234906879,0 epo=1 pro=1521675165.234910679,0 following repl=(n2,s2):3 seq=3 start=1521675161.423802654,0 epo=1 pro=1521675161.423805154,0
I180321 23:32:45.256098 30432 storage/store.go:3562  [n3,s3,r36/2:/Table/51/1/{120-770}] added to replica GC queue (contacted deleted peer)
I180321 23:32:45.256098 30918 storage/store.go:3543  [n3,s3,r36/2:/Table/51/1/{120-770}] added to replica GC queue (peer suggestion)
I180321 23:32:45.256831 37961 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r81/1:/Table/54/1/{1-2}] generated Raft snapshot d96103d3 at index 18
E180321 23:32:45.257364 37961 storage/queue.go:778  [raftsnapshot,n1,s1,r81/1:/Table/54/1/{1-2}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r81: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.257604 37841 storage/store.go:2414  [replicaGC,n3,s3,r36/2:/Table/51/1/{120-770}] removing replica
I180321 23:32:45.257833 37841 storage/replica.go:836  [replicaGC,n3,s3,r36/2:/Table/51/1/{120-770}] removed 658 (650+8) keys in 0ms [clear=0ms commit=0ms]
I180321 23:32:45.257863 38024 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r85/1:/Table/54/1/{5-6}] generated Raft snapshot 676bb96a at index 17
E180321 23:32:45.258334 38024 storage/queue.go:778  [raftsnapshot,n1,s1,r85/1:/Table/54/1/{5-6}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r85: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.258690 38028 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r84/1:/Table/54/1/{4-5}] generated Raft snapshot 8fdc9847 at index 17
E180321 23:32:45.259157 38028 storage/queue.go:778  [raftsnapshot,n1,s1,r84/1:/Table/54/1/{4-5}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r84: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.259520 37915 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r80/1:/Table/54/1/{0-1}] generated Raft snapshot 44b0d1bf at index 18
E180321 23:32:45.260673 37915 storage/queue.go:778  [raftsnapshot,n1,s1,r80/1:/Table/54/1/{0-1}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r80: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.261056 37967 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r82/1:/Table/54/1/{2-3}] generated Raft snapshot 2197f1ed at index 17
E180321 23:32:45.261518 37967 storage/queue.go:778  [raftsnapshot,n1,s1,r82/1:/Table/54/1/{2-3}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r82: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.261825 38035 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r83/1:/Table/54/1/{3-4}] generated Raft snapshot 1cc811f9 at index 17
E180321 23:32:45.262365 38035 storage/queue.go:778  [raftsnapshot,n1,s1,r83/1:/Table/54/1/{3-4}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r83: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.297485 38042 storage/replica_command.go:1784  [replicate,n3,s3,r91/2:/Table/51/1/8{00-10}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r91:/Table/51/1/8{00-10} [(n1,s1):1, (n3,s3):2, next=3]
I180321 23:32:45.313039 38070 storage/replica.go:3218  [n3,s3,r91/2:/Table/51/1/8{00-10}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n3,s3):2] next=3
E180321 23:32:45.314164 37808 storage/queue.go:778  [replicate,n3,s3,r91/2:/Table/51/1/8{00-10}] no removable replicas from range that needs a removal: [2*:15]
I180321 23:32:45.316333 38098 storage/store.go:2414  [replicaGC,n1,s1,r91/1:/Table/51/1/8{00-10}] removing replica
I180321 23:32:45.316632 38098 storage/replica.go:836  [replicaGC,n1,s1,r91/1:/Table/51/1/8{00-10}] removed 20 (10+10) keys in 0ms [clear=0ms commit=0ms]
I180321 23:32:45.437626 29550 storage/replica_proposal.go:201  [n1,s1,r66/1:/Table/51/1/5{40-60}] new range lease repl=(n1,s1):1 seq=4 start=1521675165.435302651,0 epo=1 pro=1521675165.435305885,0 following repl=(n2,s2):3 seq=3 start=1521675161.766172015,0 epo=1 pro=1521675161.766174715,0
I180321 23:32:45.462610 38007 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r86/1:/Table/54/1/{6-7}] generated Raft snapshot 45db580e at index 17
E180321 23:32:45.464671 38007 storage/queue.go:778  [raftsnapshot,n1,s1,r86/1:/Table/54/1/{6-7}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r86: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.465846 38151 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r88/1:/Table/54/1/{8-9}] generated Raft snapshot 96224459 at index 17
E180321 23:32:45.466882 38151 storage/queue.go:778  [raftsnapshot,n1,s1,r88/1:/Table/54/1/{8-9}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r88: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.468593 38155 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r87/1:/Table/54/1/{7-8}] generated Raft snapshot 675da420 at index 17
E180321 23:32:45.469343 38155 storage/queue.go:778  [raftsnapshot,n1,s1,r87/1:/Table/54/1/{7-8}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r87: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.637281 29392 storage/replica_proposal.go:201  [n1,s1,r69/1:/Table/51/1/{580-600}] new range lease repl=(n1,s1):1 seq=4 start=1521675165.635578920,0 epo=1 pro=1521675165.635581943,0 following repl=(n2,s2):3 seq=3 start=1521675161.813752371,0 epo=1 pro=1521675161.813755039,0
I180321 23:32:45.656278 38244 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r89/1:/Table/5{4/1/9-5}] generated Raft snapshot bee05d90 at index 15
E180321 23:32:45.656959 38244 storage/queue.go:778  [raftsnapshot,n1,s1,r89/1:/Table/5{4/1/9-5}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r89: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:45.767671 30278 storage/replica_proposal.go:201  [n3,s3,r15/2:/Table/1{8-9}] new range lease repl=(n3,s3):2 seq=3 start=1521675165.765225433,0 epo=1 pro=1521675165.765228788,0 following repl=(n1,s1):1 seq=2 start=1521675160.110204534,0 exp=1521675169.111067582,0 pro=1521675160.111134059,0
I180321 23:32:45.856242 38267 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r90/1:/Table/55{-/1/9}] generated Raft snapshot 9962788c at index 16
E180321 23:32:45.857066 38267 storage/queue.go:778  [raftsnapshot,n1,s1,r90/1:/Table/55{-/1/9}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r90: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:46.010070 38333 storage/replica_raftstorage.go:520  [raftsnapshot,n2,s2,r79/3:/Table/51/1/7{40-70}] generated Raft snapshot 8bf74d6d at index 11
I180321 23:32:46.013240 38333 storage/store.go:3828  [raftsnapshot,n2,s2,r79/3:/Table/51/1/7{40-70}] streamed snapshot to (n3,s3):2: kv pairs: 38, log entries: 1, rate-limit: 8.0 MiB/sec, 3ms
I180321 23:32:46.013477 38272 storage/replica_raftstorage.go:726  [n3,s3,r79/2:{-}] applying Raft snapshot at index 11 (id=8bf74d6d, encoded size=4273, 1 rocksdb batches, 1 log entries)
I180321 23:32:46.013829 38272 storage/replica_raftstorage.go:732  [n3,s3,r79/2:/Table/51/1/7{40-70}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.039127 29530 storage/replica_proposal.go:201  [n1,s1,r28/1:/Table/51/1/{40-60}] new range lease repl=(n1,s1):1 seq=4 start=1521675166.035849316,0 epo=1 pro=1521675166.035852371,0 following repl=(n2,s2):3 seq=3 start=1521675161.297004121,0 epo=1 pro=1521675161.297006865,0
I180321 23:32:46.056182 38372 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r111/1:/Table/55/{1/9-2}] generated Raft snapshot 126426e7 at index 23
E180321 23:32:46.056859 38372 storage/queue.go:778  [raftsnapshot,n1,s1,r111/1:/Table/55/{1/9-2}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r111: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:46.058262 38376 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r113/1:/{Table/55/2-Max}] generated Raft snapshot ae78ecab at index 13
E180321 23:32:46.063046 38376 storage/queue.go:778  [raftsnapshot,n1,s1,r113/1:/{Table/55/2-Max}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r113: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:46.164725 38378 storage/replica_consistency.go:129  [replica consistency checker,n3,s3,r5/3:/System/ts{d-e}] triggering stats recomputation to resolve delta of {ContainsEstimates:true LastUpdateNanos:1521675160854641532 IntentAge:0 GCBytesAge:0 LiveBytes:-26168 LiveCount:-526 KeyBytes:-24064 KeyCount:-526 ValBytes:-2104 ValCount:-526 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0}
E180321 23:32:46.237425 38484 storage/queue.go:778  [replicate,n2,s2,r35/3:/Table/51/1/{890-900}] no removable replicas from range that needs a removal: [3*:31]
I180321 23:32:46.256534 30432 storage/store.go:3562  [n3,s3,r35/2:/{Table/51/1/8…-Max}] added to replica GC queue (contacted deleted peer)
I180321 23:32:46.256796 30918 storage/store.go:3543  [n3,s3,r35/2:/{Table/51/1/8…-Max}] added to replica GC queue (peer suggestion)
I180321 23:32:46.256996 38468 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r81/1:/Table/54/1/{1-2}] generated Raft snapshot a60908ee at index 18
I180321 23:32:46.257384 38452 storage/store.go:2414  [replicaGC,n3,s3,r35/2:/{Table/51/1/8…-Max}] removing replica
I180321 23:32:46.257609 38452 storage/replica.go:836  [replicaGC,n3,s3,r35/2:/{Table/51/1/8…-Max}] removed 118 (110+8) keys in 0ms [clear=0ms commit=0ms]
E180321 23:32:46.257623 38468 storage/queue.go:778  [raftsnapshot,n1,s1,r81/1:/Table/54/1/{1-2}] snapshot failed: (n3,s3):2: remote couldn't accept snapshot with error: [n3,s3],r81: cannot apply snapshot: snapshot intersects existing range [n3,s3,r35/2:/{Table/51/1/8…-Max}]
I180321 23:32:46.258118 38472 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r85/1:/Table/54/1/{5-6}] generated Raft snapshot e9bdfc20 at index 17
I180321 23:32:46.258771 38472 storage/store.go:3828  [raftsnapshot,n1,s1,r85/1:/Table/54/1/{5-6}] streamed snapshot to (n3,s3):2: kv pairs: 9, log entries: 7, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.259066 38475 storage/replica_raftstorage.go:726  [n3,s3,r85/2:{-}] applying Raft snapshot at index 17 (id=e9bdfc20, encoded size=2704, 1 rocksdb batches, 7 log entries)
I180321 23:32:46.259375 38475 storage/replica_raftstorage.go:732  [n3,s3,r85/2:/Table/54/1/{5-6}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.260480 38236 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r84/1:/Table/54/1/{4-5}] generated Raft snapshot 2472a9d8 at index 17
I180321 23:32:46.261649 38236 storage/store.go:3828  [raftsnapshot,n1,s1,r84/1:/Table/54/1/{4-5}] streamed snapshot to (n3,s3):2: kv pairs: 9, log entries: 7, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.262246 38240 storage/replica_raftstorage.go:726  [n3,s3,r84/2:{-}] applying Raft snapshot at index 17 (id=2472a9d8, encoded size=2704, 1 rocksdb batches, 7 log entries)
I180321 23:32:46.262610 38240 storage/replica_raftstorage.go:732  [n3,s3,r84/2:/Table/54/1/{4-5}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.264507 38391 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r80/1:/Table/54/1/{0-1}] generated Raft snapshot 7e13a02b at index 18
I180321 23:32:46.265405 38391 storage/store.go:3828  [raftsnapshot,n1,s1,r80/1:/Table/54/1/{0-1}] streamed snapshot to (n3,s3):2: kv pairs: 9, log entries: 8, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.265691 38515 storage/replica_raftstorage.go:726  [n3,s3,r80/2:{-}] applying Raft snapshot at index 18 (id=7e13a02b, encoded size=2765, 1 rocksdb batches, 8 log entries)
I180321 23:32:46.265997 38515 storage/replica_raftstorage.go:732  [n3,s3,r80/2:/Table/54/1/{0-1}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.266965 38395 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r82/1:/Table/54/1/{2-3}] generated Raft snapshot 6dc1a23c at index 17
I180321 23:32:46.267946 38395 storage/store.go:3828  [raftsnapshot,n1,s1,r82/1:/Table/54/1/{2-3}] streamed snapshot to (n3,s3):2: kv pairs: 9, log entries: 7, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.268246 38398 storage/replica_raftstorage.go:726  [n3,s3,r82/2:{-}] applying Raft snapshot at index 17 (id=6dc1a23c, encoded size=2703, 1 rocksdb batches, 7 log entries)
I180321 23:32:46.268545 38398 storage/replica_raftstorage.go:732  [n3,s3,r82/2:/Table/54/1/{2-3}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.269414 38460 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r83/1:/Table/54/1/{3-4}] generated Raft snapshot fe647aae at index 17
I180321 23:32:46.270499 38460 storage/store.go:3828  [raftsnapshot,n1,s1,r83/1:/Table/54/1/{3-4}] streamed snapshot to (n3,s3):2: kv pairs: 9, log entries: 7, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.270773 38463 storage/replica_raftstorage.go:726  [n3,s3,r83/2:{-}] applying Raft snapshot at index 17 (id=fe647aae, encoded size=2703, 1 rocksdb batches, 7 log entries)
I180321 23:32:46.271045 38463 storage/replica_raftstorage.go:732  [n3,s3,r83/2:/Table/54/1/{3-4}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.356237 38542 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r86/1:/Table/54/1/{6-7}] generated Raft snapshot 02bbac9d at index 17
I180321 23:32:46.357020 38542 storage/store.go:3828  [raftsnapshot,n1,s1,r86/1:/Table/54/1/{6-7}] streamed snapshot to (n3,s3):2: kv pairs: 9, log entries: 7, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.357836 38545 storage/replica_raftstorage.go:726  [n3,s3,r86/2:{-}] applying Raft snapshot at index 17 (id=02bbac9d, encoded size=2704, 1 rocksdb batches, 7 log entries)
I180321 23:32:46.358155 38545 storage/replica_raftstorage.go:732  [n3,s3,r86/2:/Table/54/1/{6-7}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.359244 38562 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r88/1:/Table/54/1/{8-9}] generated Raft snapshot 55066bb1 at index 17
I180321 23:32:46.359936 38562 storage/store.go:3828  [raftsnapshot,n1,s1,r88/1:/Table/54/1/{8-9}] streamed snapshot to (n3,s3):2: kv pairs: 9, log entries: 7, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.360211 38560 storage/replica_raftstorage.go:726  [n3,s3,r88/2:{-}] applying Raft snapshot at index 17 (id=55066bb1, encoded size=2704, 1 rocksdb batches, 7 log entries)
I180321 23:32:46.360549 38560 storage/replica_raftstorage.go:732  [n3,s3,r88/2:/Table/54/1/{8-9}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.361480 38579 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r87/1:/Table/54/1/{7-8}] generated Raft snapshot 8024f74d at index 17
I180321 23:32:46.362844 38579 storage/store.go:3828  [raftsnapshot,n1,s1,r87/1:/Table/54/1/{7-8}] streamed snapshot to (n3,s3):2: kv pairs: 9, log entries: 7, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.363187 38584 storage/replica_raftstorage.go:726  [n3,s3,r87/2:{-}] applying Raft snapshot at index 17 (id=8024f74d, encoded size=2704, 1 rocksdb batches, 7 log entries)
I180321 23:32:46.363502 38584 storage/replica_raftstorage.go:732  [n3,s3,r87/2:/Table/54/1/{7-8}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.656236 38570 storage/replica_raftstorage.go:520  [raftsnapshot,n1,s1,r89/1:/Table/5{4/1/9-5}] generated Raft snapshot 2df3d3c0 at index 15
I180321 23:32:46.657097 38570 storage/store.go:3828  [raftsnapshot,n1,s1,r89/1:/Table/5{4/1/9-5}] streamed snapshot to (n3,s3):2: kv pairs: 29, log entries: 5, rate-limit: 8.0 MiB/sec, 1ms
I180321 23:32:46.657453 38573 storage/replica_raftstorage.go:726  [n3,s3,r89/2:{-}] applying Raft snapshot at index 15 (id=2df3d3c0, encoded size=3007, 1 rocksdb batches, 5 log entries)
I180321 23:32:46.657751 38573 storage/replica_raftstorage.go:732  [n3,s3,r89/2:/Table/5{4/1/9-5}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180321 23:32:46.703877 38489 storage/replica_command.go:1784  [replicate,n3,s3,r32/2:/Table/51/1/{80-100}] change replicas (REMOVE_REPLICA (n2,s2):3

Please assign, take a look and update the issue accordingly.

Jira issue: CRDB-13756

The text was updated successfully, but these errors were encountered:

dt · 2018-08-17T16:43:43Z

haven't seen this one in 4 months

benesch · 2018-10-17T01:42:50Z

That would be because it was skipped for those four months. :)

https://github.com/cockroachdb/cockroach/blob/master/pkg/ccl/backupccl/backup_test.go#L858

The test was flaky and hence skipped for quite sometime. I wasn't able to fix it in it's current form because of the complexity of how we enable job control statements today. This will be revisited once we have a saner implementation. The folloup is tracked at cockroachdb#40638. Touches cockroachdb#40638. Fixes cockroachdb#24136. Release notes (bug fix): none.

adityamaru · 2022-03-14T12:03:26Z

This test is still skipped -- reopening https://github.com/cockroachdb/cockroach/blob/e450b31463f6c344918300f2a27d2947a80564e2/pkg/ccl/backupccl/backup_test.go#L1498

blathers-crl · 2022-03-14T12:03:54Z

cc @cockroachdb/bulk-io

blathers-crl · 2023-03-08T18:20:51Z

Hi @shermanCRL, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

98792: kvserver: unskip `TestNewTruncateDecision` r=erikgrinaker a=erikgrinaker Passed after 10k stress runs. Has been skipped since 2019, issue seems to have been fixed in the meanwhile. Resolves #38584. Epic: none Release note: None 98855: roachtest: enable schema changes in acceptance/version-upgrade r=fqazi a=fqazi Previously, due to flakes we disabled schema changes inside the version update test. This patch re-enables them, since we are confident that the workload itslef is now stable in a mixed version state. Fixes: #58489 Release note: None 99023: kv: add log scope to BenchmarkSingleRoundtripWithLatency r=arulajmani a=nvanbenschoten Informs #98887. Avoids mixing logs with benchmark results, which breaks benchdiff. Release note: None 99033: storepool: set last unavailable on gossip dead r=andrewbaptist a=kvoli Previously, the `LastUnavailable` time was set in most parts of the storepool when a store was considered either `Unavailable`, `Dead`, `Decommissioned` or `Draining`. When `LastUnavailable` is within the last suspect duration (30s default), the node is treated as suspect by other nodes in the cluster. `LastUnavailable` was not being set when a store was considered dead due to the store not gossiping its store descriptor. This commit updates the `status` storepool function to do just that. Informs: #98928 Release note: None 99039: pkg/ccl/backupccl: Remove TestBackupRestoreControlJob r=benbardin a=benbardin This test has was marked skipped for flakiness, in 2018. Fixes: #24136 Release note: None Co-authored-by: Erik Grinaker <[email protected]> Co-authored-by: Faizan Qazi <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]> Co-authored-by: Austen McClernon <[email protected]> Co-authored-by: Ben Bardin <[email protected]>

cockroach-teamcity added this to the 2.0 milestone Mar 21, 2018

cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Mar 21, 2018

petermattis modified the milestones: 2.0, 2.0.x Apr 5, 2018

tbg assigned dt Apr 18, 2018

tbg added the A-disaster-recovery label May 21, 2018

tbg added this to the 2.1 milestone Jul 22, 2018

dt closed this as completed Aug 17, 2018

benesch reopened this Oct 17, 2018

tbg closed this as completed Dec 9, 2019

adityamaru reopened this Mar 14, 2022

adityamaru added the skipped-test label Mar 14, 2022

adityamaru unassigned dt and spaskob Mar 14, 2022

adityamaru removed this from the 2.1 milestone Mar 14, 2022

blathers-crl bot added the T-disaster-recovery label Mar 14, 2022

exalate-issue-sync bot assigned dt Mar 21, 2022

shermanCRL added the GA-blocker label Mar 8, 2023

shermanCRL added the branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 label Mar 8, 2023

exalate-issue-sync bot assigned benbardin and unassigned dt Mar 14, 2023

benbardin mentioned this issue Mar 20, 2023

pkg/ccl/backupccl: Remove TestBackupRestoreControlJob #99039

Merged

craig bot closed this as completed in e57386b Mar 20, 2023

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

teamcity: failed tests on master: test/TestBackupRestoreControlJob #24136

teamcity: failed tests on master: test/TestBackupRestoreControlJob #24136

cockroach-teamcity commented Mar 21, 2018 •

edited by cockroach-jira-scripts

Loading

dt commented Aug 17, 2018

benesch commented Oct 17, 2018

adityamaru commented Mar 14, 2022

blathers-crl bot commented Mar 14, 2022

blathers-crl bot commented Mar 8, 2023

teamcity: failed tests on master: test/TestBackupRestoreControlJob #24136

teamcity: failed tests on master: test/TestBackupRestoreControlJob #24136

Comments

cockroach-teamcity commented Mar 21, 2018 • edited by cockroach-jira-scripts Loading

dt commented Aug 17, 2018

benesch commented Oct 17, 2018

adityamaru commented Mar 14, 2022

blathers-crl bot commented Mar 14, 2022

blathers-crl bot commented Mar 8, 2023

cockroach-teamcity commented Mar 21, 2018 •

edited by cockroach-jira-scripts

Loading