Restart node instead of panicing and exiting

[BrandonWeng]

Describe your changes and provide context

Currently the node will only switch into blocksync mode if the whole process is restarted, this change aims to add a detection go routine and restart the node if it's behind configurable

Testing performed to validate your change

Artificially introduced latency so that the node would fall behind, and see that it self restarted node

[BrandonWeng]

Error: stat foobar/config/genesis.json: no such file or directory
Usage:
  export [flags]

Flags:
      --chain-id string              Chain ID
      --for-zero-height              Export state to start at height zero (perform preproccessing)
      --height int                   Export state from a particular height (-1 means latest height) (default -1)
  -h, --help                         help for export
      --home string                  The application home directory (default "/tmp/TestExportCmd_HomeDir378377008/001")
      --jail-allowed-addrs strings   Comma-separated list of operator addresses of jailed validators to unjail

I think the test failures are unrelated

[yzang2019]

The test failures seems to be new, I think we should aim to fix them or at least figure out why they are failing. If they fail due to flakyness, then that's probably fine

[philipsu522]

Looks good. The only thing I'm slightly concerned about is that if we forcefully shut down the process (which I don't think it does) in the middle of a write to the database. This could cause some integrity issues and ultimately bork a machine. Can we:

1. make sure we pause and wait for all graceful shutdown routines
2. let it run repeatedly on a node for a while?

[BrandonWeng]

Looks good. The only thing I'm slightly concerned about is that if we forcefully shut down the process (which I don't think it does) in the middle of a write to the database. This could cause some integrity issues and ultimately bork a machine. Can we:

1. make sure we pause and wait for all graceful shutdown routines
2. let it run repeatedly on a node for a while?

1. In the StartInProcess call we actually should be gracefully shutting down the process whenever a restart/sig is sent (including database connections). https://github.com/sei-protocol/sei-cosmos/pull/218/files#diff-97289af2ab66016f168df6fccac70626d125f170e5cf0841fcdc8fa624999608R497-R521
2. Yeah I can spin up a chain with this change and let it run for a while (with a couple of the nodes with smaller instances so they fall behind)