UBI Image Support (Builder&Run Images, Extensions)

[BarDweller]

With the recent completion of experimental extension support in the CNCF Buildpack project, we can now look at UBI based stacks.

Red Hat Universal Base Images (UBI) are designed to be a foundation for cloud-native and web applications use cases developed in containers. Unlike the existing Ubuntu based Paketo stacks, UBI images are built from portions of Red Hat Enterprise Linux. For more info, see https://developers.redhat.com/products/rhel/ubi

There are a wide variety of UBI images offered by Red Hat, offering pre configured deployments of various runtimes (such as Java), and offering cutdown images designed for extension (ubi-minimal).

While it would be possible to use UBI images in a similar fashion to the existing Ubuntu based images, and download application dependencies via buildpacks, and store them within layers, it would be more normal (where possible) to use the dependencies supplied via the configured repositories within the images. Eg, rather than downloading a JVM/JDK, one would be installed via 'yum'. Dependencies installed via yum in this manner can help customers meet support requirements for their application by ensuring the resulting application is sitting on a supportable runtime. Often this is a key consideration for customers when choosing to use UBI based applications in production.

Until recently, CNCF Buildpacks were unable to affect the Builder image, or Run image at a system level (eg, you could not make modifications that required root from a buildpack, such as installing packages via yum). Recent updates to the Buildpacks project have added 'extensions', these are special variants of buildpacks that are able to modify the Builder image (and substitute the Run image), via Dockerfiles generated during the execution of the Extensions. (In the future, the goal is to also allow modification of the Run image via the same mechanism).

Thus, via extensions, it becomes possible to have a Builder that contains such extensions, that can install dependencies such as Java, Maven, NodeJS, etc via yum. These extensions are very similar to buildpacks, participating in the 'detect' phase, and being allowed to issue 'provides' to the buildplan (but critically not 'requires').. then extensions that are selected to become part of a build run a 'generate' task (rather than a buildpacks 'build' task), that may output Dockerfiles to modify the Builder image, and/or substitute the Run image. Just like buildpacks, the detect & generate invocations get access to the source project to determine what actions they wish to take. For more information on extensions see.. https://github.com/buildpacks/spec/blob/main/image_extension.md

UBI Image support was on the Roadmap for 2022, but was mostly blocked by the extension work upstream.

With that functionality now complete, it's proposed to create a Builder that uses UBI, that will produce applications that run on UBI.

The Builder should (where possible)

• obtain dependencies used during the build via extensions via yum rather than via downloads
  (behavior to be overridable via env vars to disable globally/disable per dependency)
• switch the Run image to a precanned Run image that has required dependencies (eg JVM/JDK preinstalled)

As part of this, we would need new repositories for the extensions (which should be just thought of as slightly different flavor buildpacks, as they share 90% of their definition with them). Extension implementations will likely be using the same frameworks as existing buildpacks, and have logic that will seem very familiar. There may be scope for extracting common function from existing buildpacks into libraries that can be driven from both extensions and buildpacks (Consider 'is this a java project' type functions).

So far, we have prototyped extensions that can supply Java & Node JS,

At a (very rough) guess, this will overall require..

• A new repo for the UBI Builder image(s)
• A new repo for the UBI stack
  • This looks like it may require an update to https://github.com/paketo-buildpacks/jam to allow for multiple run images in a 'stack'.
  • At a guess, the builder base will be from ubi-minimal, then the various run images would be based from existing ubi runtime base images.
• New repos for the proposed extensions.
  • Expecting at least extensions for NodeJS, Java, Maven, possibly more..
• Whatever magic/pipeline/actions need to be put in place to allow these to be built & published to the appropriate docker repo.

Open questions:

• Should there be a single UBI Builder, or is there merit to having equivs for 'full' / 'base' / 'tiny' as per the Ubuntu Builders?
• Testing:
  • The choice between just repackaging existing buildpacks, and actually making sure they work in a UBI environment (it's possible, maybe even likely, that there will be version differences for system dependencies that may mean existing buildpacks may need adapting, or updating.. the challenge is in identifying where these differences cause issues)
  • We could have a limited UBI Builder where we slowly add in buildpacks we've tested, gradually building back to a comprehensive set.
• 'Stacks' with the pending removal of stacks as a concept by CNCF Buildpacks (https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) .. how should Paketo mark buildpacks as compatible with Ubuntu/UBI based images, if such marking is required?

Hmm.. this has gotten a little long =)
TLDR: UBI Images.. what do you expect of them?! How do we go about introducing them ?! =)

Thanks in advance for your thoughts & input!

[robdimsdale]

TL;DR

What do you expect of UBI images?

I think you are best-placed to answer that! You have the most consumer feedback and the most experience with prototyping UBI support. You know what it should look like; you know best what consumers want.

How do we go about introducing UBI images to the Paketo Buildpacks ecosystem and community?

I think if you (collectively) are willing to commit to helping maintain them, your high-level proposal above makes sense to me. I'd start with the smallest change set possible (a single stack + builder, stack extensions for the specific languages you want to support from day 1) and over time we can identify areas for refactoring and growing UBI support to other languages.

Detailed thoughts

Maintenance / Organizational support

Adding UBI support increases the surface area of Paketo substantially. In order for this to be sustainable for the current Paketo community, I think we would need to increase the number of contributors/maintainers in various areas. Specifically in the builders space, and probably stacks, but also maybe in some of the language families (java, node) too. This might not be as big a hurdle as it seems, given that we've seen community interest in becoming contributors of stacks/tooling, java, and nodejs.

To put this more bluntly, as a maintainer of builders, I'm in favor of this work but you have to help us support it :)

Also, for what it's worth, I don't think we'd need to add more contributors to the tooling space, because I'd expect the builders/language family teams to own most of the CI etc to build, test, and publish the stack/buildpack/builder images. To me, the increase in surface area to the tooling team feels more sustainable than the increase in surface area for builders.

Number of stacks/builders (full/base/tiny)

I'd suggest starting with one stack + builder that meets the requirements you have today. We can always add more stacks and builders and it's easier to add more than to deprecate and remove them. Perhaps I'm missing the point of your question though.

Buildpack/stack compatibility

I don't have the best mental model of stack extensions yet, but I think that if a stack (plus one or more stack extensions) provides a dependency (e.g. node), and the corresponding buildpack that would typically provide this dependency (e.g node-engine) supports no-op success if the dependency is already provided via the stack extension, then I think this should be fairly straight-forward.

I'd expect the downstream buildpacks to declare explicit support for the UBI stack (or just * if it really should work on any stack). If I understand correctly, this is essentially what you all did here and here.

Testing

If UBI stacks are to become a fully-supported, first-class citizen of Paketo Buildpacks, then I don't think there's any question that we should integration test them. At the very least, I'd expect the builder to have integration tests for the buildpacks it supports, and realistically I'd probably advocate for all the supported component buildpacks to have integration tests against the UBI stack (and its stack extensions).

Offline

If the stack extensions install packages via yum, how does this work in an offline/air-gapped environment? Does yum have built-in support for mirrors? Is this even a use-case you're concerned about?

[BarDweller]

I think you are best-placed to answer that! You have the most consumer feedback and the most experience with prototyping UBI support. You know what it should look like; you know best what consumers want.

Yeah I have a pretty good idea of what I'd consider a good Paketo UBI image, just trying here to gather any additional input =)

To put this more bluntly, as a maintainer of builders, I'm in favor of this work but you have to help us support it :)

Understood.. we're not considering this as a drive-by code dump ;p

Offline

How do other Paketo buildpacks handle offline/air-gapped ? deps via url? or via actual file?

As far as yum goes, the easy route would be an env var with the details for a local repo that the extension would configure before yum is used. Alternatively, if the rpm packages are available via another route (http? mounted volume?) then they could be pulled from there.. although with rpms having transient deps, that quickly leads to mirroring a whole bunch of stuff either way. This shouldn't be a new thing to people using yum etc in airgapped/offline envs.

There does come a point with airgapped envs where it becomes slightly more sensible to build your own custom builder with the resources you want all contained within it. Then you only have the one control point to manage for the developers, rather than an arbitrary set of rpms/tarballs/transient deps to manage.

[loewenstein]

This looks like it may require an update to https://github.com/paketo-buildpacks/jam to allow for multiple run images in a 'stack'.

Should we consider to convince upstream that multiple run images are useful and should be supported by the spec right away, instead of going for something jam specific?

I'd like the notion of a single Paketo Jammy Builder for example (and stack as long as this exists as a concept) and just using different run images or later extensions to provide the more and more dependencies that the more and more buildpacks require.

Same would of course be true for the single Paketo UBI Builder.

[BarDweller]

Should we consider to convince upstream that multiple run images are useful and should be supported by the spec right away, instead of going for something jam specific?

If you mean CNCF Buildpacks as the upstream, then they plan to remove the stack concept altogether.
Functionally, CNCF Buildpacks support multiple run images today via use of Extensions.

The jam specific part here is more around creating required Build & Run base images. Today jam is able to build a single pair of build+run, was just considering here that if, for ubi, we need build + multiple run, then it would make sense to add this to jam rather than fudge something together for the other run images.

I'd like the notion of a single Paketo Jammy Builder for example (and stack as long as this exists as a concept) and just using different run images or later extensions to provide the more and more dependencies that the more and more buildpacks require.

I'm not sure this will give you much different to what you can already achieve today with the existing selection of builder images with their included selection of buildpacks. From a builder perspective, the extensions merely allow us to install dependencies via rpm/dnf (or even apt if you are on ubuntu/debian), rather than installing them from a tarball. This also means the extensions are not placing the build time deps into layers, but they are part of the normal system dirs like /usr/local etc.

From the application run image perspective, the extensions (currently **) allow us to 'switch' to a new image that will be used instead of the default 'stack' run image. For UBI, this means we'll switch to a run image that already has the runtime installed. UBI has a variety of these available already, and we'll just need to add the appropriate label etc via a docker layer to make those into viable run images we can switch to.

(** eventually run images will be modifiable via extensions in the same manner as the builder today, with the caveat that any applications produced in this manner lose the ability to rebase to newer run images).

There may be scope for the ubuntu stacks to use extensions to switch to runtime-specific run images to cut down the layer count though..

The main flaw with using switched run images like this is that you must have a run image that's suitable to use.. Eg. For this proposal, we won't support multi-runtime applications initially.. (for example, an app that wants both node & java).. but they could be handled by the user creating a custom node-java-run-image and the extensions honoring an env var that would swap to the user specified run-image

[loewenstein]

What I meant is in particular, that a builder.toml has exactly one build and exactly one run image. Creating multiple run images with jam just gives us the capabilities to produce them, but no means other than user provided --run-image flags to chose one that is not the one in builder.toml.

But you might be right, extensions will be a new way to select the run image that's part if the builder actually.

[BarDweller]

What I meant is in particular, that a builder.toml has exactly one build and exactly one run image. Creating multiple run images with jam just gives us the capabilities to produce them, but no means other than user provided --run-image flags to chose one that is not the one in builder.toml.

Aye, that's still the case today, that builder.toml defines a stack with a build-image and a run-image, the build image is the base for the builder image, and the run-image is the default one that will be used if no extensions switch to alternate run images. I suspect that the --run-image flag just switches that default run-image, and won't override any image being switched to via extensions.. I'll need to go test that, as potentially it may be cleaner if that flag were to also override extension switching behavior. The additional run images are currently not referenced from builder.toml, but still need to be built just like the default run image today.

But you might be right, extensions will be a new way to select the run image that's part if the builder actually.

Indeed.. I also imagine that eventually the stack block in the builder.toml will either go away entirely, or be replaced with something that allows for the up-front declaration of images to switch to.. Looks like the stack removal stuff is being looked at at the moment in the CNCF Buildpacks project, so I'll see what I can find out. :-)