Log4j2 hotpatching in Bottlerocket 1.5.0

[bcressey]

What's new?

In Bottlerocket 1.5.0, we added the ability to hotpatch Java processes inside containers. Bottlerocket is not affected by the log4j2 CVEs; however, Bottlerocket customers may be using the logging library within their containers.

The hotpatch functionality works by running OCI hooks during container startup. When the container is created, the hotpatch agent is copied in along with a small log4j2 hotpatch file. For a short period of time after the container starts, the hotpatch agent checks whether any Java process with a supported version is running in the container. If so, it attempts to apply the hotpatch to that process.

The hotpatch functionality is disabled by default. This is an optional mitigation that is provided to customers looking for defense in depth.

Why would I want to use this?

It’s important to be clear that hotpatching is not a substitute for patching. Containers that include Java applications should be updated to new versions that include vendor patches for CVE-2021-44228 and related CVEs as quickly as possible.

However, in some cases, you may not have complete control over the patching process. Containers created by third-party vendors may not have updates available, and you may not have the source code or infrastructure available to rebuild them on your own. In these cases, the hotpatch functionality provides a possible way to mitigate CVE-2021-44228.

It’s also important to be aware of the limitations. Only certain versions of the JDK are supported, and only Java processes that start shortly after the container starts will be hotpatched.

How do I turn it on?

Hotpatching can be enabled for new launches of Bottlerocket by including the following settings in user data.

[settings.oci-hooks]
log4j-hotpatch-enabled = true

For existing hosts running the latest version of Bottlerocket, hotpatching can be enabled using the API client.

apiclient set oci-hooks.log4j-hotpatch-enabled=true

Enabling the setting at runtime has no effect on running containers. Newly-launched containers will be hotpatched.

Help! I found a problem!

If hotpatching causes a problem in your environment, it can be disabled by omitting the user data at launch, or by setting oci-hooks.log4j-hotpatch-enabled to false at runtime.

The hotpatch functionality depends on several open source projects. The issue could be in the log4j2 hotpatch, or in Bottlerocket’s hotpatch agent, or in Bottlerocket itself.

If the root cause is unclear, please reach out through AWS Premium Support or open a GitHub issue for further guidance.

[bcressey]

This functionality will be removed in Bottlerocket 1.15.0. As mentioned above, the hotpatching feature was never a substitute for patching the vulnerability. It has been twenty months since CVE-2021-44228 was discovered, and Bottlerocket has had ten feature releases since then. We expect that the vast majority of Java applications have been patched by this time. We’re removing the feature to reduce the patching surface area of the distro and to remove complexity from the path of starting containers. For backwards compatibility, the setting will continue to be supported but will have no effect beyond printing a deprecation warning.

We’re not aware of any use of the hotpatch agent outside of Bottlerocket, and accordingly we plan to end-of-life the hotpatch agent by November 2023. Please open an issue if this affects you.