Support Lifecycle Hooks

[vishh]

Capturing requirements for Lifecycle hooks discussed during the summit.

Lifecycle hooks was agreed to be a useful feature and support for a few basic hooks is expected to be in place. Specifically, PreStart, PreStop and PostStop hooks will be introduced to start with. Supporting hooks might require separating the lifetime of namespaces from that of the processes in the container.

Hooks will be binaries that will be exec'ed. The Spec and current State of the container will be provided as arguments to the hook (via fds?). Are the exec hooks executed synchronously?

This issue is meant to define the Spec for Hooks and discuss the hooks that are necessary to begin with.

[LK4D4]

I think makes sense to add PreCheckpoint, PostCheckpoint, PreRestore, PostRestore at least for linux extension. ping @boucher @avagin
I think hooks must be synchronous and receive state as json to stdout.

[wking]

On Thu, Jun 25, 2015 at 11:06:52AM -0700, Alexander Morozov wrote:

I think makes sense to add PreCheckpoint, PostCheckpoint,
PreRestore, PostRestore at least for linux extension…

Hmm, do we even have checkpoint/restore lifecycle actions 1? Maybe
that should be a separate issue before we talk about hooks for them
;).

[boucher]

It would definitely be useful to have pre/post checkpoint/restore.

[philips]

@wking @LK4D4 @boucher Lets concentrate on what we know we must do: prestart and poststop before defining hooks for other stuff first.

@vishh What is a prestop hook used for in your mind? We had a debate on pre-stop and just held off on defining one in appc: appc/spec#188 (comment)

[mrunalp]

I agree that prestart and poststop are what we call agree on as useful and can start with adding those.

[mrunalp]

Should also we define what arguments are passed to these hooks as well? On Linux, we will have to pass the pid of the container process.

[LK4D4]

@mrunalp I thought we will pass whole state to all hooks.

[mrunalp]

@LK4D4 We could do that but wouldn't the state be implementation specific?

[LK4D4]

@mrunalp But it still will be a string, right?

[mrunalp]

Yes, that is one way to make it work; let the hooks interpret whatever is passed down. If we do decide to pass the whole state then it might be better to send it over a pipe rather than as an argument.

[LK4D4]

Totally agree on pipe(maybe stdin). Dunno how it will work in windows, probably the same.

[tnachen]

I think we also should draw out a timeline of the container life cycle i.e

namespace created -> pre-start -> user process launched -> process stopped -> post-stop -> cleanup

One reason is that I imagine there will be more hooks, since from a Mesos perspective we have more stages that requires more hooks for us to integrate, and it gets a bit more confusing as you need to read the description to figure out what each hook meant.

[tnachen]

Also can we name pre-start to something like pre-start-user-process and post-stop to post-stop-user-process to be more precise?

[wking]

On Wed, Aug 26, 2015 at 11:04:32PM -0700, Timothy Chen wrote:

I think we also should draw out a timeline of the container life cycle i.e

namespace created -> pre-start -> user process launched -> process stopped -> post-stop -> cleanup

+1

[wking]

On Wed, Aug 26, 2015 at 11:05:12PM -0700, Timothy Chen wrote:

Also can we name pre-start to something like pre-start-user-process
and post-stop to post-stop-user-process to be more precise?

With an explicit timeline (or graph, since it doesn't have to be
linear) early in the docs, I think the current names should be clear
enough (until we get more pre-start-something-else hooks).

[conqerAtapple]

We are essentially talking about an explicit container state machine. Once we have the state machine, it might be easier to give names and to add hooks.

[mrunalp]

@tnachen Yes, it makes sense to add such a timeline. I will make a PR for what we have right now in relation to the hooks. I think we probably don't want to rename these till there are concrete proposals for other pre/post hooks.

[tnachen]

Sounds good, I'll be creating more proposals as we're investigating how we can integrate Mesos with the runtime spec.

[wking]

On Thu, Aug 27, 2015 at 10:09:38AM -0700, Mrunal Patel wrote:

Yes, it makes sense to add such a timeline. I will make a PR for
what we have right now in relation to the hooks.

I didn't see a PR like that, and I wanted the timeline so I could link
from the lifecycle to the hook documentation as part of my
filename-consolidation (#126). So I added a more detailed version of
@tnachen's timeline as part of #126.

[liangchenye]

Should we define a priority In the 'pre-start' or 'post-stop'?
There might be multiply hooks, which should run first?

[wking]

On Mon, Sep 07, 2015 at 04:09:52AM -0700, 梁辰晔 (Liang Chenye) wrote:

Should we define a priority In the 'pre-start' or 'post-stop'?
There might be multiply hooks, which should run first?

The hook entries are arrays, so I think we should just execute them in
the listed order. More explicit wording in this regard would help
avoid confusion, although the current “remaining hooks” phrasing is a
start 1.

For a similar problem (explicitly linking mount order to the listed
array order), see #142.

[liangchenye]

Yes, similar.
I just 'copied' your word (listed order) and added a pull request.
#156

[vbatts]

We have basic hooks in now. This issue is largely a "tracker" for future hooks, like needed for CRIU.

[wking]

On Fri, Sep 25, 2015 at 04:00:59PM -0700, Mrunal Patel wrote:

@wking There was a discussion on IRC…

Ah, good :). Can you point me to the right day?

$ grep -ir post-start irclogs/2015/freenode/#opencontainers.*

isn't turning up anything…

[wking]

On Fri, Sep 25, 2015 at 04:10:32PM -0700, Jojy George Varghese wrote:

To complement the pre-start, we need a pre-stop hook (see diagram
for the symmetry).

The problem with pre-stop is that the stop is not necessarily being
triggered by the host-process managing the container. For example,
the process may just exit on its own. But as long as folks are clear
that these hooks only fire on ‘funC stop ’ (or whatever
triggers a host-initiated stop), I'm ok with them.

[mrunalp]

One more use case that we see while working with hooks is having pre/post mount hooks. These could be run before and after the new mount namespace is setup. WDYT?

[wking]

On Fri, Dec 04, 2015 at 04:22:27PM -0800, Mrunal Patel wrote:

One more use case that we see while working with hooks is having
pre/post mount hooks. These could be run before and after the new
mount namespace is setup. WDYT?

Is there a distinction between post-mount and pre-start? #231 doesn't
mention mounts at all, but I'd guess they'd fall under 7713efc's step
2. If so, the only difference between post-mount and pre-start would
be step 3, where the state.json gets written. That doesn't seem like
a useful enough distinction to me (but then, the distinction between
pre-start and post-start didn't seem useful enough to me either, #202,
1).

Pre-mount may be more useful, but if it's going to be a thing I think
we need to add enough detail to the lifecycle that we know what
pre-mount is coming after. And we need to figure out what is going to
go into its state JSON (do we require a container PID on Linux by that
point or not?).

[rhatdan]

I really need post-mount. I want a hook that is going to mount additional mount points to /sys. Currently if I use prestart, /sys gets mounted over my mount points.

[wking]

On Sun, Jan 17, 2016 at 06:07:08AM -0800, Daniel J Walsh wrote:

I really need post-mount. I want a hook that is going to mount
additional mount points to /sys. Currently if I use prestart, /sys
gets mounted over my mount points.

I think “more hooks” is the wrong solution. If you've provided /sys,
I don't think the runtime should be clobbering it (see points two and
three here 1). Of course, those points aren't currently spelled out
in the spec, which is pretty vague about the timing and clobbering
details of runtime-provided devices and mount points.

And I'd have expected pre-start to be run after the mount setup anyway
(it happens later in ccon 2).

But regardless of how this should be handled now, separate create/exec
actions (#299) should get you out of this problem, because /sys-level
mount tweaking should not be happening in the exec phase. That lets
you fix any broken start-phase functionality before calling exec. And
the only mount-tweaking done by exec will probably be mounting a new
/proc if the exec creates a new PID namespace 3. And the only thing
that will clobber (on my system anyway) is 4.

 See “/proc and PID namespaces”

[rhatdan]

If the start hooks happened after the mount then that would also fix my problem. I am trying to get additional kernel file systems mounted onto /sys. /sys/fs/cgroup to be exact. I don't want to take over the mounting of /sys or any other kernel file systems except for the cgroup hierarchy.

Currently /sys gets mounted after I mount /sys/fs/cgroup, so that my mounts disappear.

[wking]

@vishh suggested that I shift some discussion of hooks and a separate
create / (start or exec) over here [1,2]. See also #299 for more
background in splitting sandbox creation from process execution.
Linux runtimes launching a container process [3,4] will usually do an
IPC dance like 5 so the runtime process can spawn the pre-start
hooks after the container namespaces have been setup but before the
user-specified code is exec'ed (this may be an implemention detail
6, but I don't see how to avoid it 7). With the split proposed in
#299, the PID namespace will be created at start time (step (2) in
8), which means that any actions that need access to the container
PID (e.g. custom cgroup injection or state registration) will still
have to happen at start-time. So while splitting create from start
will make some things possible without hooks (e.g. manipulating the
mount namespace created by ‘create’), we won't be able to drop
pre-start hooks entirely 8.

[vishh]

(e.g. custom cgroup injection or state registration)

I don't get how running hooks between pid ns creation and user process startup is a necessity for either of these use cases. Can you elaborate more?

[wking]

On Fri, Mar 18, 2016 at 10:40:10AM -0700, Vish Kannan wrote:

(e.g. custom cgroup injection or state registration)

I don't get how running hooks between pid ns creation and user
process startup is a necessity for either of these use cases. Can
you elaborate more?

You can't add the container process to a cgroup until you have a PID,
and you can't get that PID (in the runtime namespace 1) until you
create the PID namespace.

More generally, if the ‘start’ call kicks off the process tree that
gives you the container process, it should be pretty clear that any
manipulation of that process has to happen after the ‘start’ call ;).
And things like cgroup injection and resource limits should be set
before you exec user-configured code, so that puts them after ‘start’
and before start's execvp (or whatever) in some sort of hook.

[vishh]

Why does one add a cgroup with a hook when the Spec already provides
ability to define cgroups?

On Fri, Mar 18, 2016 at 10:50 AM, W. Trevor King [email protected]
wrote:

On Fri, Mar 18, 2016 at 10:40:10AM -0700, Vish Kannan wrote:

(e.g. custom cgroup injection or state registration)

I don't get how running hooks between pid ns creation and user
process startup is a necessity for either of these use cases. Can
you elaborate more?

You can't add the container process to a cgroup until you have a PID,
and you can't get that PID (in the runtime namespace 1) until you
create the PID namespace.

More generally, if the ‘start’ call kicks off the process tree that
gives you the container process, it should be pretty clear that any
manipulation of that process has to happen after the ‘start’ call ;).
And things like cgroup injection and resource limits should be set
before you exec user-configured code, so that puts them after ‘start’
and before start's execvp (or whatever) in some sort of hook.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#20 (comment)

[wking]

On Fri, Mar 18, 2016 at 11:33:51AM -0700, Vish Kannan wrote:

Why does one add a cgroup with a hook when the Spec already provides
ability to define cgroups?

Maybe you want to use an alternative cgroup manager that's built on
higher level abstractions 1. Or maybe you want to use a new kernel
feature that hasn't landed in your runtime yet 2.

But is motivating the cgroups use case really that important? I find
it hard to imagine folks never needing access to the container PID
before user-specified code is exec'ed. The cgroups and
state-registration (e.g. if you want to maintain a cluster-wide
registry, but your runtime's builtin-registry doesn't handle that) are
just the most obvious cases I can think of, not an attempt at an
exhaustive list.

 Subject: Re: removal of cgroups from the OCI Linux spec
 Date: Thu, 29 Oct 2015 11:28:41 -0700
 Message-ID: <CAA_vbqSCEg+ByohgfR6Pqnzvy+j60=jmMNJi_UfnhB3WPtcjww@mail.gmail.com>

[vishh]

I'd prefer not considering such scenarios unless it cannot be solved
through other means.

On Fri, Mar 18, 2016 at 11:45 AM, W. Trevor King [email protected]
wrote:

On Fri, Mar 18, 2016 at 11:33:51AM -0700, Vish Kannan wrote:

Why does one add a cgroup with a hook when the Spec already provides
ability to define cgroups?

Maybe you want to use an alternative cgroup manager that's built on
higher level abstractions 1. Or maybe you want to use a new kernel
feature that hasn't landed in your runtime yet [2].

But is motivating the cgroups use case really that important? I find
it hard to imagine folks never needing access to the container PID
before user-specified code is exec'ed. The cgroups and
state-registration (e.g. if you want to maintain a cluster-wide
registry, but your runtime's builtin-registry doesn't handle that) are
just the most obvious cases I can think of, not an attempt at an
exhaustive list.

Subject: Re: removal of cgroups from the OCI Linux spec
Date: Thu, 29 Oct 2015 11:28:41 -0700
Message-ID: <CAA_vbqSCEg+ByohgfR6Pqnzvy+j60=
[email protected]>
[2]: #267

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#20 (comment)

[wking]

On Fri, Mar 18, 2016 at 12:11:12PM -0700, Vish Kannan wrote:

I'd prefer not considering such scenarios unless it cannot be solved
through other means.

Ok, so we just need to get to the bottom of a particular use-case.
How do you see attachment to an experimental cgroup (e.g. #267)
working before support lands in your chosen runtime? You could
solve that by using v2 cgroups 1 and letting the runtime join one
you'd already setup, but that seems like a major change if you're
already using v1 cgroups. How does the runtime distinguish between v1
and v2 cgroup users anyway? Is the logic for joining cgroups:

If a v2 cgroup at cgroupsPath exists, add the container process to
it. Otherwise add the container process to any v1 cgroups whose
path relative to their cgroup mount point exists.

Or are runtimes supposed to be hard-coded to always use v1 or v2
cgroups? Or are v1 runtimes supposed to limit attachments to cgroups
that are defined under linux.resources?

[vishh]

Why not run a custom runtime or move the process after it gets spawned?

On Fri, Mar 18, 2016 at 12:33 PM, W. Trevor King [email protected]
wrote:

On Fri, Mar 18, 2016 at 12:11:12PM -0700, Vish Kannan wrote:

I'd prefer not considering such scenarios unless it cannot be solved
through other means.

Ok, so we just need to get to the bottom of a particular use-case.
How do you see attachment to an experimental cgroup (e.g. #267)
working before support lands in your chosen runtime? You could
solve that by using v2 cgroups 1 and letting the runtime join one
you'd already setup, but that seems like a major change if you're
already using v1 cgroups. How does the runtime distinguish between v1
and v2 cgroup users anyway? Is the logic for joining cgroups:

If a v2 cgroup at cgroupsPath exists, add the container process to
it. Otherwise add the container process to any v1 cgroups whose
path relative to their cgroup mount point exists.

Or are runtimes supposed to be hard-coded to always use v1 or v2
cgroups? Or are v1 runtimes supposed to limit attachments to cgroups
that are defined under linux.resources?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#20 (comment)

[wking]

On Fri, Mar 18, 2016 at 12:35:19PM -0700, Vish Kannan wrote:

Why not run a custom runtime…

Because that's a moderately large change when all you want is:

$ echo $PID >/sys/fs/cgroup/a/b/c/tasks

And “you could fork the runtime and add support for that” is an
argument that applies to everything. If we go that route, where do
you draw the line? It it only stuff that would need pre-start hooks
that doesn't get supported in the spec?

… or move the process after it gets spawned?

I am suggesting we move the process after it's spawned (because you
need a PID to assign it). I'm just suggesting we assign it to cgroups
before we execvp (or whatever) the user-specified code. Assigning
resource limits before exec'ing the user-specified code seems like the
only safe approach to me, since you want to avoid an unconstrained
time window at the beginning of the user-specified code's execution.

[wking]

I think this is closable. The current spec position is to split create/start (#384) and keep hooks (closing #483), so callers have two ways to handle any hook-like things they need to do. Skimming the tail of this discussion, I don't see anything that isn't currently supported. And if there are unsupported use-cases, I think a more productive approach would be to open per-case issues with “the current hooks or create/start split don't support …”, and we can hash out whether additional hooks are needed. For example, see #644, where pre-/post-mount hooks were proposed, discussed, and rejected.

[vbatts]

Agreed