still getting segfaults on v1.7.1 (ubuntu 18.04)

[matthewdarwin]

Ubuntu 18.04
Using own compiled binaries (no changes to build.sh)

From syslog:

Apr  7 13:59:40 mainnet-public2 nodeos[5150]: info  2019-04-07T13:59:40.558 thread-0  producer_plugin.cpp:345       on_incoming_block    ] Received block 0a02d978c3b99827... #51721547
@ 2019-04-07T13:59:40.500 signed by eosiosg11111 [trxs: 20, lib: 51721213, conf: 0, latency: 58 ms]
Apr  7 13:59:41 mainnet-public2 nodeos[5150]: info  2019-04-07T13:59:41.106 thread-0  producer_plugin.cpp:345       on_incoming_block    ] Received block da97b2a1396dcd15... #51721548
@ 2019-04-07T13:59:41.000 signed by eosiosg11111 [trxs: 35, lib: 51721213, conf: 0, latency: 106 ms]
Apr  7 13:59:41 mainnet-public2 nodeos[5150]: info  2019-04-07T13:59:41.229 thread-0  producer_plugin.cpp:345       on_incoming_block    ] Received block fbbb5a6d0957b1ae... #51721549 @ 2019-04-07T13:59:41.500 signed by eosiosg11111 [trxs: 18, lib: 51721213, conf: 0, latency: -270 ms]
Apr  7 13:59:42 mainnet-public2 kernel: [338478.399065] show_signal_msg: 13 callbacks suppressed
Apr  7 13:59:42 mainnet-public2 kernel: [338478.399068] nodeos[5184]: segfault at 8 ip 
00000000004bd065 sp 00007f7af5fbe970 error 4 in nodeos[400000+289a000]

wasm-runtime = wabt
chain-state-db-size-mb = 24576
reversible-blocks-db-size-mb = 2048
http-validate-host = false
max-body-size = 1048576
p2p-max-nodes-per-host = 2
agent-name = "EOS Nation"
allowed-connection = any
max-clients = 150
connection-cleanup-period = 30
network-version-match = 0
sync-fetch-span = 2500
enable-stale-production = false
max-transaction-time = 30
max-irreversible-block-age = -1
txn-reference-block-lag = 0
abi-serializer-max-time-ms = 2000
verbose-http-errors = true
plugin = eosio::http_plugin
plugin = eosio::chain_api_plugin
plugin = eosio::net_api_plugin
plugin = eosio::producer_api_plugin
p2p-peer-address = eosbp-0.atticlab.net:9876
p2p-peer-address = seed1.eos42.io:9876
p2p-peer-address = node869-mainnet.eosauthority.com:9393
p2p-peer-address = peer1.eoshuobipool.com:18181
p2p-peer-address = fullnode.eoslaomao.com:443
p2p-peer-address = node2.liquideos.com:9876
p2p-peer-address = node1.eosnewyork.io:6987
p2p-peer-address = p2p.eossweden.org:9876
p2p-peer-address = peer1.mainnet.helloeos.com.cn:80
p2p-peer-address = m.jeda.one:3322
p2p-peer-address = node1.zbeos.com:9876
p2p-peer-address = bp.eosbeijing.one:8080
p2p-peer-address = node1.eoscannon.io:59876
p2p-peer-address = p2p.bitmars.one:8080
p2p-peer-address = mars.fnp2p.eosbixin.com:443
p2p-peer-address = 192.168.123.39:9876
peer-key = XXX
p2p-peer-address = 192.168.123.40:9876
peer-key = XXX
p2p-peer-address = 192.168.124.129:9876
peer-key = XXX
peer-private-key = XXX

[heifner]

Can you get a backtrace from the core file?

[heifner]

Possibly related #7084. Note the snapshot generation is only what tripped the read_delay_timer any slow down on write could trip the delay timer.

[matthewdarwin]

echo "thread apply all bt full" | gdb /usr/bin/nodeos core > bt.txt

bt.txt

[matthewdarwin]

bt2.txt
bt3.txt

[matthewdarwin]

bt4.txt

[heifner]

@matthewdarwin I believe this is fixed by: #7085
Could you possibly try that and see if it helps?

[matthewdarwin]

Will do... meanwhile here is another stack trace before that patch.

bt5.txt

[matthewdarwin]

it has been 24 hours and no crashes... so better than it plain v1.7.1 that's for sure. Can we have a v1.7.2 with this?

[matthewdarwin]

48 hours and no crashes for me.

[heifner]

Fixed by #7085