Problem with running Kaniko in own containers

[syphernl]

While troubleshooting issues with another Semantic-Release plugin I stumbled upon the fact that Kaniko overrides the contents of the container it runs in while building a (multi-stage) container.
Before Kaniko runs there are 189 files in /usr/bin and after the run there are only 153, of which a bunch not included in the first run (but are part of installs done in the Dockerfile that Kaniko is building).

Now it is clear as to why they said that simply adding the Kaniko binary to another container is not recommended.. The docs state

It unpacks directly into its own container root and may overwrite anything already there

Reproduction scenario

• Create a container with semantic-release + plugins + Kaniko executor + git
• Before Kaniko executor ran: run git --version. Works, shows version.
• Build a (multistage) image using this plugin, make sure the last step of the app container is apk del git (or apt-get purge git).
• After Kaniko executor ran: run git --version. Git cannot be found anymore

Sure you can choose to not remove git explicitly, then it will be part of the container and still work afterwards but this will likely lead to issues if you are building a Debian image with an Alpine base or the other way around. Or same platform but with different versions (e.g. Alpine 3.12 vs 3.20)

Potential fix

In this issue similar problems are discussed and seems that chroot or proot are used as workarounds.
Since the latter requires SYS_PTRACE using chroot is probably a better idea..