changi node panicked at 'attempt to subtract with overflow'

[kuegi]

Summary

• node crashed during normal operation, after restart, it resynced a lot of blocks but didn't manage to catch up to tip again. (failure of sync likely is caused by all other nodes crashed too, I lost all connections)
• see attached crash report: thread '<unnamed>' panicked at 'attempt to subtract with overflow', ain-evm/src/block.rs:89:38

crashreport_chain.txt

Environment

• Node Version: 4.0.0 beta 2
• Block height on bug if applicable: last block before crash: 1717493 , managed to sync only to 1717395 afterwards
• OS with version: OSX

[prasannavl]

It does appear all the seed nodes for changi are down with the same log provided as above. We're starting to investigate.

[prasannavl]

Leaving some notes for info:

• beta1 will not sync freshly sync after 1712914. beta2 was to workaround those for now until the next beta was ready but has crashed the network
• Both the changi seed nodes had been upgraded to beta2, which caused the whole network to crash.
• Possible next steps:
  • We can just simply restart the seed nodes. However, we're investigating first to get to the bottom of the issue, before restarting, since it'll result in a chain from 1717493

[shohamc1]

Fixed by #2098