With pack build --run-image the user specified run image should have precedence over run images provided by extensions

[loewenstein]

Summary

If I call pack build --run-image my-run-image, but an extension changes the run image via the following Dockerfile, the application image base layers are taken from their-run-image.

FROM their-run-image

I haven't really tried this, but this is the expectation based on a Slack thread with @natalieparellano

---

Reproduction

Steps

1. Create a builder with run image extension
2. Call pack build --run-image my-run-image using that builder
3. Inspect the base layer of the resulting image

Current behavior

The extension provided run image is taken as base for the application image.

Expected behavior

The user provided run image should be taken as base for the application image.

Alternatively, when this is not feasible to implement, the build should fail, advising the user to remove the --run-image parameter or the extension.

Note: This is only an issue with extensions that fully replace the run image. An extension adding to the run image should be ok.

[loewenstein]

The use case I have in mind is, that the builder maintainer - likely in collaboration with buildpack maintainers - could have optimised run images for particular use cases:

• Java needs less OS packages than Node and Python
• Java Tomcat might benefit from a Shell, while Executable Jars don't need it
• Depending on the use of start scripts, Node might need things line Shell or /bin/env from coreutils
• ...

In all those cases, I would like to allow the user to know even better. Whatever the specific application might need, the developer should have the last say in providing it.

I realise that there are different use cases from that. First and foremost, extending a user provided run image (via --run-image) in an extension is just as valid as extending the default run image of a builder.

What I am not so sure is, whether there is a case to forbid the user to override the dynamic selection of a run image. Even for the Red Hat UBI work, while it might not make sense to override, users might still want or even have to in exceptional cases.

In short, maybe the two cases are

• run image extension
• run image exchange
  and have to be treated differently.

One option could be that the extension is responsible to handle it, then we would just need to make it transparent to the extension that the user has specified the run image that got passed in. Maybe we need to expand this and pass in the source of a run image more generally (could be user, builder or another extension and in case of extension it could be a replacement or an *extension).

[jabrown85]

I wonder what the UX should be. Maybe pack needs more info and not the extension?

pack build --run-image please/do-not:replace <img>
is the same as
pack build --run-image please/extend:me <img>

Obviously in the non-extension flow - the conclusion is pretty obvious. pack can't know if any of the 0 to many extensions on the chosen builder will extend or replace at the time of execution or do nothing and land the app on top of the given run-image.

Maybe something like this could verify either exact run-image or lineage of the resulting app image?
pack build --run-image please/do-not:replace --verify-run-image <img>

[loewenstein]

Well, all --verify-run-image could do is failing if an extension were to replace the run image, right? So the user would be left with not being able to use their run image or not being able to use the builder that includes the offending extension.

[jabrown85]

Well, all --verify-run-image could do is failing if an extension were to replace the run image, right? So the user would be left with not being able to use their run image or not being able to use the builder that includes the offending extension.

I was thinking it could be a verify-lineage sort of thing. The run image's layers must exist on the produced app image but it doesn't have to be the exact run image.

[loewenstein]

Understood. Still, the builder / extension maintainer has to decide if an image get's replaced or not and the user can only say if changing the image entirely is acceptable or not. An extension couldn't provide a default run image but accept that the user has the final word.

[BarDweller]

Hmm.. multiple scenarios here.. I'd agree the current flag is unclear and would need careful interpretation.

1. User wants to override run image, extension wants to switch the run image.

• In this case, if the run images don't match, the build should likely fail, unless we have some kind of 'force' argument, in which case there should likely be a warning that the extension choice is being ignored, and maybe the forcing should be recorded in the application image metadata to help distinguish that image for support purposes.
• We've coded in an env var for our run image switching extensions that pretty much allows for this (it's very handy during testing for new run images with existing builders), so there's definitely a call for something like this, but it's worth noting that if the run image isn't switched, that it's like a build has run and ignored the extension (but yet may have done things that only extension builds would have done), which makes for a messy situation, that really needs to only happen with the understanding of the end-user.

2. User wants to override the run image, extensions want to extend the run image

• This one is a little easier, as extension starts from the default run image identified by the builder metadata, so in theory that could be the one the user wants to use as the override. I think this closer matches the original usage of --run-image in an extension free build?
• If the image isn't in the set of run images declared by the builder, there should probably a warning or fail, depending on if the override is considered a request, or a forced choice.
• Again, still worth recording in the app image metadata that an override occurred.

3. User wants to swap the run image, based on the choice made by the extension(s).

• Scenario 1 really only works for user initiated builds, where the user knows pretty much what extensions may run, and what choices they might make, and is just cutting ahead to swap out the image. Eg, the UBI builder supports Node & Java, at multiple versions each, and we know which one the build will select, and override it. This isn't really practical if the replacements are needed in a wider scope.
• This would need a map of 'oldimage->newimage' to be passed, for entries absent in the map, the extensions choice is kept, but for anything in the map, the extensions choice is overridden to the map value.
• This would be quite handy ;) I could see users using it to extend the subset of official run images that their builds use to add in their own unique stuff, and have the builds use the appropriate run image based on the choice made during the build.
• As with all overrides, I'd like to see the override choice being recorded in the app image metadata, to help distinguish those images from those built using the tested run images ;)

Hmm.. I like the map thing =)

[natalieparellano]

I think this one could benefit from some synchronous discussion - put it in future topics for Working Group 🙂

[loewenstein]

Agreed @natalieparellano. Although I would also say that this will benefit hugely, if we enforce an asynchronous discussion. An asynchronous discussion more likely will result in the status quo, the different use case and what would need to be changed to balance requirements to be properly documented ;)

Would "Build & Run Image Selection" be a good separate chapter for the spec maybe? That one could list and plot the different sources of image changes (dynamic selection and extension)... Just a thought though, because a PR doing this could be the optimal place to have the discussion and pour the discussion results directly into documentation.

[natalieparellano]

Would "Build & Run Image Selection" be a good separate chapter for the spec maybe?

I tend to agree, it is really hard to determine how the run image is selected by reading the spec.

We discussed this in Working Group today (3/28), we think the current behavior (--run-image is an input to extensions but does not override extensions) should be explained better in the existing documentation and/or help text for pack, but we should continue to explore alternatives that might make this more configurable.