add an integration of nodeos for crash when the nodes are killed

[vzqzhang]

Change Description

Added a new integration test for nodeos. This test sets up one producing nodes and one "bridge" node using test_control_api_plugin. We send transactions to the bridge node. After the accounts are initialized and allocated with symbols, we kill the producing node. Then we flood hundreds transactions to the bridge node. At the end, we ill the bridge. It tests that no crashes happen when the nodes are killed.

Change Type

Select ONE

• Documentation

• Stability bug fix

• Other

• Other - special case

Consensus Changes

• Consensus Changes

API Changes

• API Changes

Documentation Additions

• Documentation Additions

[nksanthosh]

Since this test is not consistent in results - should we close the PR?

[heifner]

I asked if we could get automation to run this test a few hundred times to see if it fails a few times. If it does fail then it would be useful to keep. If it never fails, then I would like us to modify it such that it does cause develop to fail. We have hit shutdown bugs under load a number of times and it would be good to have a test for it.

[nksanthosh]

@kj4ezj Will you be able to run this test a few hundred times and report back on results? (not urgent - can be scheduled per your weekly workload)

[kj4ezj]

@kj4ezj Will you be able to run this test a few hundred times and report back on results?

This required code changes to the CI system, so I opened pull request 9551 based on this branch. I successfully completed a short test run in build 24759 with 21 runs in rounds of 14 runs each.

Finally, I kicked off build 24760 to perform 2100 runs across 30 rounds of 70 runs each (ten runs per operating system).