define preStop hook to enable orderly shutdown of Invoker pod

[dgrove-oss]

This issue mainly concerns running in a mode where we deploy the Invoker pod via Kubernetes, but still use docker to schedule the individual action containers.

When an invoker exits in an orderly fashion, it attempts to cleanup its actionContainers by invoking removeAllActionContainers from a JVM shutdown hook (see https://github.com/apache/incubator-openwhisk/blob/0cb847c0906f58fee1166938977708d99261c1c5/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainerFactory.scala#L76).

If the invoker crashes (or its pod is terminated by Kubernetes for some other reason), this shutdown hook may not be executed or may be executed in a way that kills running actions. We should use the preStop lifecycle hook to enable a more orderly shutdown of the invoker and its running actions in the case when the invoker is still healthy and it is being descheduled by Kubernetes for some other reason.

Note that if the invoker pod is restarted by Kubernetes on the same worker node, one of its first actions will be to invoke removeAllActionContainers from the init method of DockerContainerFactory https://github.com/apache/incubator-openwhisk/blob/0cb847c0906f58fee1166938977708d99261c1c5/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainerFactory.scala#L73. So the leaked resources from a previously crashed invoker should be reclaimed when the node is re-used for a future instance of the invoker.

[dgrove-oss]

@xingzhou - does this capture what we discussed yesterday? Looking at the DockerContainerFactory code I think the orphaned containers would have eventually been collected, especially if the Invokers were deployed as a DaemonSet instead of an auto-scaling group.

[daisy-ycguo]

Thank you for looking at this issue. From the perspective of leaking resources, your solution makes sense. But how can we make sure the translations are completed ?

What I mean is, a invoker is just gotten killed while some actions are still being executed. Are we able to collect the result or resume the execution ?

[dgrove-oss]

If the invoker dies, the running actions are orphaned. In current system I believe the results are gone.

For cases when kubernetes is killing an otherwise healthy invoker, a combination of increasing the grace period and using a preStop hook to tell the invoker to stop scheduling new actions should limit the damage.

[dgrove-oss]

Closing as old...