Emergency shut off

[cmelone]

Problem/Opportunity Statement

if we ever get into a scenario where gantry is going completely haywire or is not resolving an OOM, it would be nice to have a mechanism to disable dynamic allocations for specific packages or the entire system. This shouldn’t require going into the container or having any knowledge of how the program works.

What would success / a fix look like?

A simple web page behind gh auth accessible to a small group that has a form to disable dynamic allocations on a package level. For subsequent predictions, it should return no cpu limits and 64GB memory limit.