Restarting nodeos caused corruption in fork db (nodeos should exit instead of being stuck forever)

[matthewdarwin]

Seen on Jungle with nodeos 2.1.0-rc3.

stop nodeos, no errors:

info  2021-03-15T22:22:51.299 nodeos    resource_monitor_plugi:122    plugin_shutdown      ] shutdown...
info  2021-03-15T22:22:51.300 nodeos    resource_monitor_plugi:129    plugin_shutdown      ] exit shutdown
info  2021-03-15T22:22:51.351 nodeos    main.cpp:179                  main                 ] nodeos successfully exiting

on startup:

...
info  2021-03-15T22:22:57.741 nodeos    chain_api_plugin.cpp:98       plugin_startup       ] starting chain_api_plugin
info  2021-03-15T22:22:57.743 nodeos    net_api_plugin.cpp:53         plugin_startup       ] starting net_api_plugin
info  2021-03-15T22:22:57.743 nodeos    producer_plugin.cpp:932       plugin_startup       ] producer plugin:  plugin_startup() begin
info  2021-03-15T22:22:57.743 nodeos    producer_plugin.cpp:957       plugin_startup       ] Launching block production for 1 producers at 2021-03-15T22:22:57.743.
debug 2021-03-15T22:22:57.743 nodeos    producer_plugin.cpp:2047      schedule_production_ ] Waiting till another block is received
info  2021-03-15T22:22:57.743 nodeos    producer_plugin.cpp:968       plugin_startup       ] producer plugin:  plugin_startup() end
info  2021-03-15T22:22:57.743 nodeos    producer_api_plugin.cp:87     plugin_startup       ] starting producer_api_plugin
info  2021-03-15T22:22:57.743 nodeos    resource_monitor_plugi:94     plugin_startup       ] Creating and starting monitor thread
info  2021-03-15T22:22:57.743 nodeos    file_space_handler.hpp:112    add_file_system      ] /var/lib/nodeos/blocks's file system monitored. shutdown_available: 49065250810, capacity: 490652508160, threshold: 90
debug 2021-03-15T22:22:57.744 nodeos    producer_plugin.cpp:382       on_incoming_block    ] received incoming block 67350332 0403af3cbe05ed9e0d2d3715b3c6a5774b05e355eb3a0c58cd0c1a4e0a9eb9a5
debug 2021-03-15T22:22:57.753 nodeos    producer_plugin.cpp:249       on_block             ] Removed applied transactions before: 0, after: 0
warn  2021-03-15T22:22:57.753 nodeos    controller.cpp:1902           push_block           ] 3020000 fork_database_exception: Fork database exception
empty block log expects the first appended block to build off a block that is not the fork database root
    {}
    nodeos  controller.cpp:334 log_irreversible
error 2021-03-15T22:22:57.753 nodeos    producer_plugin.cpp:405       operator()           ] 3020000 fork_database_exception: Fork database exception
empty block log expects the first appended block to build off a block that is not the fork database root
    {}
    nodeos  controller.cpp:334 log_irreversible
rethrow
    {}
    nodeos  controller.cpp:1902 push_block
debug 2021-03-15T22:22:57.753 nodeos    producer_plugin.cpp:2047      schedule_production_ ] Waiting till another block is received
error 2021-03-15T22:22:57.753 nodeos    net_plugin.cpp:3278           process_signed_block ] ["eosn-jungle-blkrly14:9876 - 111e707" 10.88.64.244:9876]bad block exception #67350332 be05ed9e0d2d3715...: Fork database exception (3020000)
empty block log expects the first appended block to build off a block that is not the fork database root
rethrow
debug 2021-03-15T22:22:57.753 nodeos    producer_plugin.cpp:382       on_incoming_block    ] received incoming block 67350333 0403af3d583702f55b355db9edd322fbf4ee2f869700c297f22185745eeb2a4c
debug 2021-03-15T22:22:57.753 nodeos    producer_plugin.cpp:249       on_block             ] Removed applied transactions before: 0, after: 0
warn  2021-03-15T22:22:57.753 nodeos    controller.cpp:1902           push_block           ] 3020000 fork_database_exception: Fork database exception
empty block log expects the first appended block to build off a block that is not the fork database root
    {}
    nodeos  controller.cpp:334 log_irreversible
error 2021-03-15T22:22:57.753 nodeos    producer_plugin.cpp:405       operator()           ] 3020000 fork_database_exception: Fork database exception
empty block log expects the first appended block to build off a block that is not the fork database root
    {}
    nodeos  controller.cpp:334 log_irreversible
rethrow
    {}
    nodeos  controller.cpp:1902 push_block
debug 2021-03-15T22:22:57.753 nodeos    producer_plugin.cpp:2047      schedule_production_ ] Waiting till another block is received

etc....

Not sure what is the problem, but maybe nodeos should exit here instead of running forever and never moving forward?

[linhuang-blockone]

Matthew, do you have any steps to reproduce the problem? Does it always happen? I tried restarting nodeos but could not reproduce fork db corruption.

[matthewdarwin]

Sorry, the issue is not generally reproducible. I don't know what happened in this situation that caused the issue.

I mainly opened the issue because when a problem happens, nodeos should exit instead of continuing to run and never recovering.

[linhuang-blockone]

I was thinking about fixing the issue all together. Will make nodeos exit when the problem happens again, even if the root cause is not found.