Hooks for more customization

[tiborvass]

There are a few issues that ask for specific hooks, but none of them have apparently been fully addressed. The ones that are closed were promised a "generic" hooks mechanism for all commands/events.

A few usecases from reading the issues:

• shutdown hook (EXITPOINT ?) for gracefulness (Add ability to have a shutdown hook #2100)
• start hook (Hook on after container was started required #252 is a bit old, and Create postcreate hook #3317)

From kubernetes/kubernetes#140 (comment):

[...] we felt there were two cases we wanted to handle:

• hooks that either need the container context (and as such executing outside the process namespace would be pointless), or if interrupted by container shutdown would not be internally inconsistent. Pre-termination is a good example
• hooks that should be outside of a container, because they need to continue to run even if a container fails. Deploy across multiple containers is a good example, or post-termination.

Please feel free to comment with more.

It makes sense to offer hooks for greater customization, however, this might affect the current way of doing things (see the exec discussion in #3317 (comment))

For solutions, there was a mention of "predefined paths" for executables.

Some hooks (start and shutdown) would be more useful if they were executed synchronously (unlike how docker events fires currently).

We need to define what hooks should be implemented in docker and what should be done externally via docker events.

[tiborvass]

For reference: #4550

[foliea]

Perhaps that some docker options could also use the hook system ? I am thinking of docker run --rm. It would solve a few issues with that one (eg. conflict with -d) if we could use a on_exit hook.

[mrunalp]

Also, a hook to execute arbitrary code before we exec user command would be nice to allow customization in the namespaces.

[phemmer]

I keep running into things where I would find a pre-start hook very useful, so I'll throw in some of my thoughts on this.

I can see the possibility for several hooks

• pre-start - namespace is completely configured but the process has not been launched
• post-start - process has been launched
• pre-stop - just before sending a SIGTERM
• pre-kill - just before sending a SIGKILL
• post-stop - all processes have exited, but the namespaces & mounts still exist
• pre-pause
• post-pause
• pre-unpause
• post-upause

Now whether all these actually have a purpose or not is another matter. Personally I would find the pre-start and post-stop to be the most useful. My frequent use case is setting firewall rules for the container, and then cleaning them up.

I think each hook should not execute in the context of the container, but in the host context. If the hook needs container context, it can nsenter or nsinit.
The hook would be a binary or script, and it would receive a single argument of the container ID. If it needs any additional information, it should use the CLI and/or remote API to query the information it needs.

The hooks should be executed synchronously.
I also think the pre-* hooks should be able to abort an operation. For example, if the pre-start hook returns a non-0 exit status, the container should be marked as failed-to-start.

I see 2 possibilities for what level the hooks should be configured at, daemon level, so the hooks trigger on every container, and at the run level, where the hooks are passed when starting a container. I think both scenarios have valid use cases, and so both should probably be supported. Run level hooks would trigger after the daemon level hooks.

As for how to pass the hooks in. For the daemon, it would be passed as an argument with the path to the binary/script.
For run hooks, I see 2 possibilities:

1. Use multipart/related, and send each hook binary/script as additional parts after the JSON body.
2. Send the binary/script as a JSON attribute. This option has a few issues though as sending binary through JSON is rather cumbersome. And large JSON attributes just don't feel right.

 

I think hooks have a lot of potential to prevent feature bloat. By providing the capability for users to easily implement whatever they want, we can keep docker from getting a plethora of features which few use, and are difficult to maintain.

[grepory]

I detailed my use cases in #7847 -- but I think a generalized startup (that is to say post-startup) hook and a shutdown hook (that executes regardless of container exit status) would be nice. Use cases I can come up with quickly:

Service registration/discovery
Orchestration
Easier management of boot2docker volumes

[bgrant0607]

Hooks are critical to both implementation of higher-level management/orchestration systems, and (perhaps counterintuitively) portability of images across different environments, since they can provide whatever glue is necessary to adapt the container to its environment.

The hooks proposed by @phemmer SGTM. IMO, pre-start, pre-stop, and post-stop would be the most useful now (pre-pause and post-unpause would be useful eventually, too, once migration is fully supported).

In order to implement pre-start and post-stop, we need to decouple the namespace lifetime from the lifetime of the main process. That would be extremely useful to higher-level management systems, in general.

I also agree with @phemmer that the hooks should execute in the host context.

Users will often want their hooks to execute within the container context. The case for pre-start executing within the container context is to initialize the filesystem, and the case for post-stop is extracting data from the filesystem.

However, what's attractive about Docker executing the hooks in the host context is that would be provide a hook for the management layer -- Kubelet in our case. We need this, for example, in order to use Docker container restarts and still provide Kubernetes-specific functionality upon each restart.

/cc @vishh @erictune

[erictune]

As @bgrant0607 says above, both a management layer and a user (person who wants to run a container) both may want to run a hook point. How do two different sources provide hooks, and what is their order or precedence? Can there be mulitple instances of a hook or does the management layer wrap the user-provided hook command in its own command?

Also, a management layer may want to treat a user-specified hook as untrusted, and force it to run in a container (for resource isolation, protecting host filesystem). How should a management layer do this? Suppose hooks always run in host context, and the hook can optionally nsenter the container context (as suggested above). How should a management layer ensure that an arbitrary hook runs in container context? Parsing the user-provided hook is an unattractive option. Running nsenter unilaterally seems like it won't work.

Possible solution:

• Define separate host-context and container-context versions of each type of hook.
• the management layer provides the host-context hooks
• the user (person who wants to run this container) provides the container-context hooks.

Advantages:

• user provided hooks always run in container context, protecting the host filesystem and maintaining resource isolation.
• no conflict between two systems trying to define hooks.
• user provided hooks could be possibly be defined in the dockerfile, if they are something that applies to all uses of the container.
• Management hooks can be provided via the docker command line.

Can come up with a concrete example if that helps.

[phemmer]

@erictune I'm totally not following what you're getting at.

There's only 2 concievable ways you could set hooks in docker. At the daemon level (docker -d), and at the runtime level (docker run, docker start, docker build). I'm assuming by "management layer" you mean daemon level, and by "user specified hook" you mean runtime level. I don't get why the 2 would interact with each other at all.
Every hook system I've ever seen operates in layers. Layer 1 gets data, can manipulate it however it wants, and then the data gets passed to layer 2. Layer 1 has no clue that layer 2 even exists and vice versa. Each layer assumes that the next thing to receive the data is the core application itself.

[erictune]

I meant something different, I think. By "management layer", I mean an orchestration system that supports docker (such as Kubernetes or Mesos or a number of others). By "user", I mean a person that wants to run a container using an orchestration system.

The orchestration system might want to do some extra setup before running any container. It could use a "pre-start" hook.

The user may also want to do some setup before running her particular container. She could also use a "pre-start" hook.

But, the user should not need to see or understand the orchestration system's use of the hook, and the orchestration system should not need to grok the user's use of the hook.

[progrium]

This is weird because this is what the event stream is for. You can (and I'm about to) expose that event stream via a nice hook system as a container that doesn't need to be in docker core...

[jessfraz]

@progrium are you bind mounting the socket in the container? I would be interested in seeing it :)

[robhaswell]

@progrium the author has acknowledged docker events:

Some hooks (start and shutdown) would be more useful if they were executed synchronously (unlike how docker events fires currently).

There is also discussion here about the requirement for pre-start and pre-stop hooks.

[phemmer]

@progrium An event stream isn't a hook. A hook sits in between things. You can use it to capture data yes, but you can also use it to manipulate the environment. If you want to prevent docker from starting a container, your hook would return an error code. If you want to change a network setting before any processes are launched, your hook would make the change before continuing execution of the normal flow.

I've already coded up daemon level hooks. It was fairly simple. I have to add run-level (container) hooks. I know how I'm going to do it, but just havent yet. Will probably get it done this weekend.

[progrium]

Yes, you're both right. I missed that mention of events. If I know how Solomon thinks (and how I'd want it either way), this would not be directly implemented as shell hooks, but using some form of libchan/rpc accessible from containers that could then optionally expose them as hook scripts. Much like the container I'm working on.

And @jfrazelle, yes. I'll show you soon. Solomon wanted me to share something(s) for Docker Hack Day and I was thinking this.

[bgrant0607]

@progrium Requiring participation of the container, especially in a Docker-specific fashion, largely defeats the purpose of this feature.

[progrium]

Hmm, alright. I think one of us isn't following something. Can you
elaborate?

On Fri, Oct 17, 2014 at 5:15 PM, bgrant0607 [email protected]
wrote:

@progrium https://github.com/progrium Requiring participation of the
container, especially in a Docker-specific fashion, largely defeats the
purpose of this feature.

—
Reply to this email directly or view it on GitHub
#6982 (comment).

Jeff Lindsay
http://progrium.com

[tcurdt]

So I take it from this issue still being open, hooks/notifications of container's started or stopped are at this stage just merely wishful thinking? Right now I would have to listen to docker events manually (with e.g. https://github.com/deoxxa/docker-events)

Is that correct?

[progrium]

You have the events, post-event hooks are quite possible. In fact, here they are wrapped up nicely so you can create individual plugins:
https://vimeo.com/110835013
https://github.com/progrium/docker-plugins

This was made with a generic event-to-hook tool below, but I think if you're doing more than a one-off hook, docker-plugins might be a better way to go.
https://github.com/progrium/dockerhook

[tcurdt]

awesome! thanks, @progrium

[analytically]

👍

[tcurdt]

btw: while dockerhook and friends are great I ended up rolling my own slightly more specialised version. using it for managing nginx upstreams and reloads.

https://github.com/tcurdt/docker-upstream
https://registry.hub.docker.com/u/tcurdt/upstream/

seems to work fine - but still needs documentation.

[lukemarsden]

Just wanted to mention that we've released https://github.com/clusterhq/powerstrip to help solve the problem of adding blocking pre- and post- hooks to Docker.

The Powerstrip project is explicitly intended to be a way of prototyping Docker extensions. Please try it out and send feedback as github issues or PRs!

FYI @tcurdt @analytically @progrium @bgrant0607 @robhaswell @phemmer @jfrazelle @erictune @grepory @mrunalp @folieadrien @tiborvass

[peter-leonov]

👍

[luebken]

Any news on this? I find this topic pretty interesting.

[kirkstrobeck]

WANT

[jizhilong]

I also found a need for pre-start hook mechanism when trying to attach openstack-neutron ports to docker containers and do some customized network settings which is not modeld in libnetwork.
My final solution is to build such a 'prestart' hook based on docker's primitive mechanism, with no modification to docker-engine.
For those interested, this solution is now open sourced at docker-wait.

[pablo-ruth]

If you need pre-start and post-stop hooks, you can use go-init. it's a very lightweight init system for Docker (< 500kb) with support for lifecycle hooks.

[gaui]

One use-case for a pre-build Docker hook would be to emit a VERSION file with the following command: git describe --tags --long --always --dirty

[phlax]

adding a +1 to this feature request

my use case is to workaround docker support for checkpoint/restore

currently the docker implementation places checkpoint files inside the container directory. Which means that the snapshots are deleted with the container - making the checkpoint ephemeral and somewhat pointless.

if docker had post-create and pre-delete hooks for containers i would be able to bind mount a network directory to the snapshot dir or perhaps not use dockers snapshot/checkpoint feature at all and manually create/restore snapshots in the hooks