v1.1.3-pre2 crashes node on startup

[jsign]

When testing now v.1.1.3-pre2, I got one node working wihout problems, but in other two nodes they crash with the following tail logs:

Node 1:

2020-11-12T13:00:04.424Z        WARN    vm      vm/vm.go:522    Send actor error        {"from": "f1tudi422iiqzb6zrk3sw3ajuhky7fxi2avnx4ghi", "to": "f1d7mq36vf6osdhcd32i6k3wyb223mdjlxnafnala", "nonce": 9, "method": "0", "height": "229305", "error": "failed to transfer funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).send.func3\n        /lotus/chain/vm/vm.go:329\n  - transfer failed when deducting funds (78.989783599103 FIL): not enough funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).transfer\n        /lotus/chain/vm/vm.go:854"}
2020-11-12T13:00:04.425Z        WARN    vm      vm/vm.go:522    Send actor error        {"from": "f1l25nnnrhenzgtk6awm4cx3h3zpemo6mssvlducy", "to": "f1d7mq36vf6osdhcd32i6k3wyb223mdjlxnafnala", "nonce": 5, "method": "0", "height": "229305", "error": "failed to transfer funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).send.func3\n        /lotus/chain/vm/vm.go:329\n  - transfer failed when deducting funds (70.26928383038 FIL): not enough funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).transfer\n        /lotus/chain/vm/vm.go:854"}
2020-11-12T13:00:04.436Z        INFO    bs:sess session/session.go:459  No peers - broadcasting {"session": 3, "want-count": 64}
2020-11-12T13:00:04.459Z        INFO    sub     sub/incoming.go:98      new block over pubsub   {"cid": "bafy2bzaceauz5g4qj33e63qejzoqb5zlrc4ojghwbsza4y2k3b5fs5kchae24", "source": "12D3KooWH59Sk5LbFatAX8h3n3rrcot7wM9y1vdScB4nm3ytbNRr", "msgfetch": 0.03994851}
2020-11-12T13:00:04.516Z        INFO    chain   chain/sync_manager.go:197       new peer head: 12D3KooWBKzkiZXF9gApCHYdpgyCarvoDqMhaD4p9hyJDHXeek23 [bafy2bzaceauz5g4qj33e63qejzoqb5zlrc4ojghwbsza4y2k3b5fs5kchae24]
2020-11-12T13:00:04.517Z        INFO    chain   chain/sync_manager.go:413       deferring sync on [bafy2bzaceauz5g4qj33e63qejzoqb5zlrc4ojghwbsza4y2k3b5fs5kchae24]
2020-11-12T13:00:05.052Z        WARN    vm      vm/vm.go:522    Send actor error        {"from": "f1cstl6ote2oenn6jfecuaqd5ylgfgqtmxsgeq2qa", "to": "f1d7mq36vf6osdhcd32i6k3wyb223mdjlxnafnala", "nonce": 9, "method": "0", "height": "229305", "error": "failed to transfer funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).send.func3\n        /lotus/chain/vm/vm.go:329\n  - transfer failed when deducting funds (94.20209985561 FIL): not enough funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).transfer\n        /lotus/chain/vm/vm.go:854"}
2020-11-12T13:00:06.364Z        INFO    bs:sess session/session.go:459  No peers - broadcasting {"session": 4, "want-count": 45}
2020-11-12T13:00:06.380Z        INFO    sub     sub/incoming.go:98      new block over pubsub   {"cid": "bafy2bzacea5jn76owzne5yxnm3zkxws25nkkwp2p4b55ce4d3zkh3kxonvg4y", "source": "12D3KooWKPMG3wJreSHHvaLieR2VYgLBQ1He1RXuRG1suCXbhUg7", "msgfetch": 0.021484449}
2020-11-12T13:00:06.380Z        WARN    sub     sub/incoming.go:100     Received block with large delay 6 from miner f02528
2020-11-12T13:00:06.409Z        INFO    chain   chain/sync_manager.go:197       new peer head: 12D3KooWRaXYrSmiRDPzvLDSe7Qw6PPW1P9QLA3YZThTypY5zbpX [bafy2bzacea5jn76owzne5yxnm3zkxws25nkkwp2p4b55ce4d3zkh3kxonvg4y]
2020-11-12T13:00:06.409Z        INFO    chain   chain/sync_manager.go:413       deferring sync on [bafy2bzacea5jn76owzne5yxnm3zkxws25nkkwp2p4b55ce4d3zkh3kxonvg4y]
2020-11-12T13:00:08.135Z        INFO    messagepool     messagepool/messagepool.go:390  mpool ready
2020-11-12T13:00:08.137Z        WARN    vm      vm/vm.go:522    Send actor error        {"from": "f1li5konrkmfyibmrcskhr7g3tqeyq43to6h5ahoy", "to": "f05", "nonce": 4, "method": "2", "height": "229304", "error": "failed to transfer funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).send.func3\n        /lotus/chain/vm/vm.go:329\n  - transfer failed when deducting funds (0.371250620372154626 FIL): not enough funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).transfer\n        /lotus/chain/vm/vm.go:854"}
2020-11-12T13:00:08.137Z        WARN    sub     sub/incoming.go:583     error from message subscription: context canceled
2020-11-12T13:00:08.137Z        WARN    sub     sub/incoming.go:585     quitting HandleIncomingMessages loop
2020-11-12T13:00:08.137Z        WARN    sub     sub/incoming.go:60      quitting HandleIncomingBlocks loop
2020-11-12T13:00:08.138Z        WARN    peermgr peermgr/peermgr.go:138  closing peermgr done
2020-11-12T13:00:08.138Z        ERROR   chain   chain/sync.go:1498      failed to validate tipset: context canceled
2020-11-12T13:00:08.138Z        INFO    pubsub  [email protected]/pubsub.go:619   pubsub processloop shutting down
2020-11-12T13:00:08.138Z        INFO    chain   chain/sync.go:717       block validation        {"took": 43.643907674, "height": "229306", "age": 428.13834402}
2020-11-12T13:00:08.138Z        INFO    chain   chain/sync_manager.go:321       worker 0 done; took 46.122265606s
2020-11-12T13:00:08.138Z        WARN    peermgr peermgr/peermgr.go:156  exiting peermgr run
2020-11-12T13:00:08.138Z        INFO    chain   chain/sync.go:717       block validation        {"took": 43.642334624, "height": "229306", "age": 428.138509536}
2020-11-12T13:00:08.138Z        INFO    chain   chain/sync.go:717       block validation        {"took": 43.642321523, "height": "229306", "age": 428.138594544}
2020-11-12T13:00:08.138Z        INFO    chain   chain/sync.go:717       block validation        {"took": 43.64156095, "height": "229306", "age": 428.138633448}
ERROR: initializing node: starting node: GasEstimateMessageGas error: estimating gas used: message execution failed: exit SysErrInsufficientFunds(6), reason: failed to transfer funds (RetCode=6): transfer failed when deducting funds (0.371250620372154626 FIL): not enough funds (RetCode=6)

Node 2:

Nov 12 12:59:11 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:11.643Z        INFO        hello        hello/hello.go:180        time offset        {"offset": -0.137993173, "peerid": "12D3KooWH4jMECFWo16YHoYp8pVpxsHz8b528KmbN2HQt6LznQp7"}
Nov 12 12:59:11 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:11.851Z        INFO        hello        hello/hello.go:121        Got new tipset through Hello: [bafy2bzaceappvoiyohp3lor5ihpk4hpele4v5o4et46wiebc35uvulykslmhc bafy2bzacebunkak7rle76eqangrxyx7g7fupctcsnrwl7a36mcp4ilpodlqeg bafy2bzacebqmaxhtcia3qr3xbt6ef5vnxo5jeewzmdfs4sq4arfwysf5ktpoc] from 12D3KooWSysXYUpjU5M4aREUZvGzpd1BuqFGjmkGQ2fyiBC9jjiD
Nov 12 12:59:11 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:11.857Z        INFO        chain        chain/sync_manager.go:197        new peer head: 12D3KooWSysXYUpjU5M4aREUZvGzpd1BuqFGjmkGQ2fyiBC9jjiD [bafy2bzaceappvoiyohp3lor5ihpk4hpele4v5o4et46wiebc35uvulykslmhc bafy2bzacebunkak7rle76eqangrxyx7g7fupctcsnrwl7a36mcp4ilpodlqeg bafy2bzacebqmaxhtcia3qr3xbt6ef5vnxo5jeewzmdfs4sq4arfwysf5ktpoc]
Nov 12 12:59:11 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:11.917Z        INFO        hello        hello/hello.go:121        Got new tipset through Hello: [bafy2bzaceabwrt4eq4hfhvypyxtwecrw7z7paifchrrhkwlpxszrbpkufbekg bafy2bzacecuk5i34bhkux7chuyv7kzca6yvmpcn6kayitzrqmpxsowj7poalw bafy2bzaceawyinrhnujnkatwazyyyvvede32ju3mjkdapnskv5kr3colv4rgu bafy2bzacebpgpgdfspyubrzkjuagjx67pk62josmo6sbc4gnxxzheqpx4bn3q] from 12D3KooWFg4YwYeGrVZiqwZqkYci7PByApj2QAmEdCmPtE5pNvcV
Nov 12 12:59:11 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:11.946Z        INFO        chain        chain/sync.go:281        incoming tipset does not appear to be better than our best chain, ignoring for now        {"miners": ["f07830", "f01800", "f020330", "f018782"], "bestPweight": "4771294070", "bestTS": [{"/":"bafy2bzacebibgpeyxb7cpopzat7o4lmmr4odowx72cgbixpbwjgaxhbw3623w"},{"/":"bafy2bzaceaypatv2breh72ls44qcp4g3w4jwgulwwicvu276ehqrily6ijadu"},{"/":"bafy2bzaceaz2dqmxg3ulrun4tlg6uw35f6wsmqzh7ued64ekjvkge25thim6g"},{"/":"bafy2bzacecwarckkknghcuf3yi3mfcdqnfbajkxwwnlyca5xwu5gpnrawmkbc"},{"/":"bafy2bzacednu4l7kujej4ycuodpszsyd6qvhkb5erpqfy67377earwtf2nssk"},{"/":"bafy2bzacedhzc46de4m77jetbph7k7w2ziyea5slfwyjse3nlgai6euwkrbrc"},{"/":"bafy2bzacedngy4a5vm52f2zzrodxhrrbubt5kygtkd3qhmhbitakkij55h5ys"},{"/":"bafy2bzacebqd743pdi6mreywignkmmrcfrmnm7pphuzwmidwc2cfgh25nuzqc"}], "incomingWeight": "4551772040", "incomingTS": [{"/":"bafy2bzaceabwrt4eq4hfhvypyxtwecrw7z7paifchrrhkwlpxszrbpkufbekg"},{"/":"bafy2bzacecuk5i34bhkux7chuyv7kzca6yvmpcn6kayitzrqmpxsowj7poalw"},{"/":"bafy2bzaceawyinrhnujnkatwazyyyvvede32ju3mjkdapnskv5kr3colv4rgu"},{"/":"bafy2bzacebpgpgdfspyubrzkjuagjx67pk62josmo6sbc4gnxxzheqpx4bn3q"}]}
Nov 12 12:59:12 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:12.146Z        INFO        hello        hello/hello.go:180        time offset        {"offset": -0.161995935, "peerid": "12D3KooWSysXYUpjU5M4aREUZvGzpd1BuqFGjmkGQ2fyiBC9jjiD"}
Nov 12 12:59:12 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:12.160Z        INFO        hello        hello/hello.go:180        time offset        {"offset": -0.00079182, "peerid": "12D3KooWFg4YwYeGrVZiqwZqkYci7PByApj2QAmEdCmPtE5pNvcV"}
Nov 12 12:59:15 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:15.282Z        INFO        sub        sub/incoming.go:98        new block over pubsub        {"cid": "bafy2bzaceawzal7l3zqilrigiftln7bpqfbrxikqzp6fdxzxi2z6reszkgcqk", "source": "12D3KooWNsNMa4tKSyUoJY5tGtKWGW2d8wQrQ1eU5GWR19gDDXs6", "msgfetch": 0.099648959}
Nov 12 12:59:15 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:15.282Z        WARN        sub        sub/incoming.go:100        Received block with large delay 4305 from miner f016398
Nov 12 12:59:15 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:15.296Z        INFO        chain        chain/sync.go:281        incoming tipset does not appear to be better than our best chain, ignoring for now        {"miners": ["f016398"], "bestPweight": "4771294070", "bestTS": [{"/":"bafy2bzacebibgpeyxb7cpopzat7o4lmmr4odowx72cgbixpbwjgaxhbw3623w"},{"/":"bafy2bzaceaypatv2breh72ls44qcp4g3w4jwgulwwicvu276ehqrily6ijadu"},{"/":"bafy2bzaceaz2dqmxg3ulrun4tlg6uw35f6wsmqzh7ued64ekjvkge25thim6g"},{"/":"bafy2bzacecwarckkknghcuf3yi3mfcdqnfbajkxwwnlyca5xwu5gpnrawmkbc"},{"/":"bafy2bzacednu4l7kujej4ycuodpszsyd6qvhkb5erpqfy67377earwtf2nssk"},{"/":"bafy2bzacedhzc46de4m77jetbph7k7w2ziyea5slfwyjse3nlgai6euwkrbrc"},{"/":"bafy2bzacedngy4a5vm52f2zzrodxhrrbubt5kygtkd3qhmhbitakkij55h5ys"},{"/":"bafy2bzacebqd743pdi6mreywignkmmrcfrmnm7pphuzwmidwc2cfgh25nuzqc"}], "incomingWeight": "4768611651", "incomingTS": [{"/":"bafy2bzaceawzal7l3zqilrigiftln7bpqfbrxikqzp6fdxzxi2z6reszkgcqk"}]}
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.905Z        INFO        messagepool        messagepool/messagepool.go:390        mpool ready
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.906Z        WARN        vm        vm/vm.go:522        Send actor error        {"from": "f1mejl6qu7eshrmjbsdo3nsdefpyohp5now4yprii", "to": "f05", "nonce": 4, "method": "2", "height": "229295", "error": "failed to transfer funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).send.func3\n        /home/azureuser/lotus/chain/vm/vm.go:329\n  - transfer failed when deducting funds (0.121715731435929285 FIL): not enough funds (RetCode=6):\n    github.com/filecoin-project/lotus/chain/vm.(*VM).transfer\n        /home/azureuser/lotus/chain/vm/vm.go:854"}
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        WARN        sub        sub/incoming.go:583        error from message subscription: context canceled
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        WARN        sub        sub/incoming.go:585        quitting HandleIncomingMessages loop
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        WARN        sub        sub/incoming.go:60        quitting HandleIncomingBlocks loop
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        WARN        peermgr        peermgr/peermgr.go:138        closing peermgr done
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        INFO        chain        chain/sync.go:717        block validation        {"took": 68.120196937, "height": "229296", "age": 680.907850559}
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        INFO        chain        chain/sync.go:717        block validation        {"took": 68.09195684, "height": "229296", "age": 680.90786536}
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        INFO        chain        chain/sync.go:717        block validation        {"took": 68.093522385, "height": "229296", "age": 680.907870161}
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        ERROR        chain        chain/sync.go:1498        failed to validate tipset: context canceled
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.907Z        INFO        chain        chain/sync.go:717        block validation        {"took": 68.120141931, "height": "229296", "age": 680.907843558}
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.908Z        WARN        peermgr        peermgr/peermgr.go:156        exiting peermgr run
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.908Z        INFO        pubsub        [email protected]/pubsub.go:619        pubsub processloop shutting down
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: 2020-11-12T12:59:20.908Z        INFO        chain        chain/sync_manager.go:321        worker 0 done; took 1m8.151939354s
Nov 12 12:59:20 spaceracetwo34 lotus[1580563]: ERROR: initializing node: starting node: GasEstimateMessageGas error: estimating gas used: message execution failed: exit SysErrInsufficientFunds(6), reason: failed to transfer funds (RetCode=6): transfer failed when deducting funds (0.121715731435929285 FIL): not enough funds (RetCode=6)
Nov 12 12:59:21 spaceracetwo34 systemd[1]: lotus-daemon.service: Main process exited, code=exited, status=1/FAILURE
Nov 12 12:59:21 spaceracetwo34 systemd[1]: lotus-daemon.service: Failed with result 'exit-code'.