PreStart and PostStop event hooks

[bgrant0607]

Many systems support event hooks for extensions. A few examples:
https://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/LifecycleManager
http://developer.android.com/guide/components/activities.html
http://upstart.ubuntu.com/cookbook/#event
https://coreos.com/docs/launching-containers/launching/getting-started-with-systemd/
http://elasticbox.com/documentation/configuring-and-managing-boxes/start-stop-and-upgrade-boxes/
http://git-scm.com/docs/githooks.html

docker stop and restart currently send SIGTERM followed by SIGKILL, similar to many other systems (e.g., Heroku: https://devcenter.heroku.com/articles/dynos#graceful-shutdown-with-sigterm), which provides an opportunity for applications to cleanly shut down, but lacks the ability to communicate the grace period duration or termination reason and doesn't directly provide support for notifying other processes or services.

As described in the (liveness probe issue)[https://github.com//issues/66], it would be useful to support multiple types of hook execution/notification mechanisms. It would also be useful to pass arguments from clients, such as "reason" (e.g., "cancel", "restart", "reload", "resize", "reboot", "move", "host_update", "probe_failure"). Another way "reason" could be handled is with user-defined events.

In addition to pre-termination notification, we should define other lifecycle hook points, probably at least pre- and post- start and terminate.

It would be useful for post-terminate to be passed the (termination reason)[https://github.com//issues/137], which could either be successful completion, a client-provided stop reason (see above), or detailed failure reason (exit, signal, OOM, container creation error, docker crash, machine crash, lost/ghost).

If the application generated an assertion failure or exception message, a post-termination hook could copy it to (/run/status.txt)[https://github.com//issues/139].

It would also be useful to be able to control (restart behavior)[https://github.com//issues/127] from a hook. We'd need a convenient way to carry over state from a previous execution. The simplest starting point would be for the user to keep it in a (volume)[https://github.com//issues/97].

[bgrant0607]

Docker issues requesting postcreate hooks:
moby/moby#3317
moby/moby#252

One of these mentioned that Docker planned to support hooks for every event, but that was quite a while ago, so I'm not sure whether that's still in plan.

[bgrant0607]

Just got support from tianon for runin/exec/enter support: moby/moby#1228
So we can use that to execute hooks inside containers

[bgrant0607]

I investigated Docker's event stream. It provides container id and event type (e.g., start, stop). It doesn't provide further details about the events. Also, it's obviously asynchronous with respect to the events. At least pre-start and post-termination event hook commands would be most useful if executed synchronously, inline with container execution.

Docker restart allows restarting the same command in the same container with the same container id and filesystem, even after the death of the previous process. There's no way to change the command executed AFAICT, however. If the forthcoming "runin"/"exec" allowed execution in dead containers, we could maybe use it for post-termination hooks. Pre-start hook commands look ugly without Docker support. No response yet to my docker-dev question about whether Docker is actually planning to add hook support.

We could override the container entrypoint with the pre-start hook command and then use "runin" to execute the real entrypoint, but the container's status, wait, restart, etc. would be broken. Creating a new container image that included the pre-start command, actual entrypoint, and post-termination hook might work, but we'd need to carefully propagate arguments, signals, exit status, etc.

The main reason for a post-start hook would be consistency, but it also might be convenient for start actions that don't block the start of the container's entrypoint, such as registration in a third-party discovery service, pushing events to pubsub, etc. "runin" should just work, though there may be a race if the application immediately terminated. Probably it would be useful to serialize execution with respect to later hooks on the same entity.

If we wrapped the application we could intercept SIGTERM in order to execute the pre-termination hook, and just pass the signal on to the application if no hook were specified. Again, we probably want to serialize with respect to the post-termination hook. It might not be super-useful to execute the pre-termination hook in the case that it weren't a planned container stop, but I'm not sure whether it's more natural to execute it prior to the post-termination hook regardless or whether it would be annoying for the pre-termination hook to execute when the application was already dead. OTOH, that case would probably need to be handled since the application could die at any time, including concurrently with the start of the pre-termination hook.

Asynchronous webhooks would be comparatively easy to support. The most difficult issue is what to do about auth. If generated by a command in the container, presumably they could authenticate as the container, so it looks appealing to only support command hooks. However, there are situations where we wouldn't have an obvious container to execute commands in, such as for pod lifecycle hooks or, even more obviously, replicationController and service lifecycle event hooks.

[smarterclayton]

We could override the container entrypoint with the pre-start hook command and then use "runin" to execute the real entrypoint, but the container's status, wait, restart, etc. would be broken. Creating a new container image that included the pre-start command, actual entrypoint, and post-termination hook might work, but we'd need to carefully propagate arguments, signals, exit status, etc.

We tried make this work, but it felt wrong on all sorts of levels and ultimately we felt there were two cases we wanted to handle:

• hooks that either need the container context (and as such executing outside the process namespace would be pointless), or if interrupted by container shutdown would not be internally inconsistent. Pre-termination is a good example
• hooks that should be outside of a container, because they need to continue to run even if a container fails. Deploy across multiple containers is a good example, or post-termination.

I don't think you need to subclass the entrypoint in the image - for the former, you could use "docker exec in" (add process to namespace) and for the latter, you can use restart-once containers (outside or inside a pod).

There's some crossover here between "event hooks" and "intent/application hooks" - reacting to changes to the infrastructure, vs reacting to user intent across multiple pods. The "intent hooks" tend to be things that I think of as wanting the auth/orchestration parts of the downward-facing API (I want to wait for this label query to reach X instances, and then update another replication controller to go to Y), as well as being more like regular jobs in their own right.

[thockin]

Is there more to do here, or should we close this and open new issues?

[bgrant0607]

I was waiting for #1445 to go in before closing this one.

[bgrant0607]

Removed from milestone and repurposed for PreStart and PostStop hooks, which I've always thought would be the most useful hooks to provide, for reasons discussed above and in several other issues.

Hooks are being discussed for Docker:
moby/moby#6982

Our users' hooks should execute within the container context. The case for PreStart executing within the container context is to initialize the filesystem, and the case for PostStop is extracting data from the filesystem.

What's attractive about Docker executing the hooks in the host context is that would be provide a hook for the management layer -- Kubelet in our case. The people who advocated executing them in the host context said that nsinit/exec could be then used to enter the container in the hook. That SGTM, since it would give us more control (e.g., allowing us to use Docker container restarts). It would still require the same changes to libcontainer and Docker, however, to decouple container lifetime from process lifetime.

/cc @vishh

[bgrant0607]

Proof-of-concept for prestart here: #3067 (comment)

[n4j]

@bgrant0607 This issue is open since long is it ok if we close this?

[timchenxiaoyu]

how to implement prestart hook ?

[adityajoshi12]

There is no updates on the prestart hook, you can use the init container as an alternative to prestart hook.

[sftim]

Are init containers a suitable replacement for preStart?

For Jobs, you can also use a pod failure policy to abort an entire Pod when a specific container fails. Perhaps extending that to other workload controllers, plus the existing init container support, is enough?

[debu99]

i prefer to make the pod zone awareness with environment variable in preStart, but init container can only share volume