Ensure service is always running

[devinmatte]

We can't afford for the service to stop running for a long period of time, so we need ways to ensure it is running

• Restarts
  • We should ensure that the service restarts every night when the T isn't running to clear any potential memory leaks and things that can be fixed with a restart
• Monitoring
  • Datadog should notify us when the service isn't currently running so we can intervene

[mathcolo]

We shouldn't be restarting to clear memory leaks—they're a critical indicator of something weird going on

better to OOM or at least observe unexpected process memory usage over time and fix root causes if we notice issues

[mathcolo]

https://www.martinfowler.com/ieeeSoftware/failFast.pdf !