Use a tiny base image for Docker builds

[pugnascotia]

Closes #51670, closes #50838.

Introduce a tiny base image for Docker builds. It aims to create a basic filesystem with as little as possible, which is mostly glibc, busybox and bash. A statically-built curl is also provided.

We still use CentOS 8 as a base. All the fun stuff happens in the Dockerfile.

Note: The comment history below covers a time period when there was an intermediate base image, and a time when the new parts of the build existed in scripts. The current form of the build is a result of discussions with Docker about how official images should be written.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[mark-vieira]

It'd be nice for changes like this to include a summary of the impact. Things like:

• Total image size before/after
• Compressed image size before/after
• Total build time (including time to build the builder and base images) before/after

We also have a number of items/questions that still need addressing if we choose to peruse this further:

• How do we manage this base image? Do we publish it separately from the ES image or do we just include all the stuff to build it in the Docker context we publish here. Would doing the latter be kosher wrt to Dockerhub policies?
• How do we integrate this into the build now that we have to first build a base image. How does this impact build times given that building Docker images is already quite expensive and now we are adding another step.
• Does this have any consequences for Cloud which consumes these images as well?
• Do we expect maintaining what is essentially a forked version of the CentOS image build to be costly going forward?

All and all there are a lot of added complexity here, and that's still w/o all the build integration stuff completely fleshed out. I think we need to consider to end benefit here, which is why I ask for the overall numbers to provide some context in making that decision.

[pugnascotia]

One thing to bear in mind here is that a reduction in size isn't strictly the goal. Rather, we want to reduce the amount of unnecessary stuff, which can e.g. trigger vulnerability warnings in security scans.

Anyway, the uncompessed image size, as reported by docker images, is 792MB on master, and 517MB with these changes, so 275MB smaller.

The compressed size is 383MB before and 290MB after, so 93MB smaller.

On my laptop, with all Docker images purged, the time to build the base image is 3m 16s (which included downloading the centos:7 image). With layer caching it is significantly faster on subsequent builds. That said, since there's nothing secret in the base image, and it's unlikely to change frequently, I had anticipated publishing it to Docker Hub as usual, perhaps by Infra if they were willing to own.

I haven't engaged with Cloud yet as I wanted to validate the approach thus far. They would definitely need to make changes though, since the base image only provides user and group management utilities from BusyBox, which are usable but less full-featured than other utils. The changes to the Dockerfile here reflect that. I don't anticipate any changes being difficult though. I already included pigz on Cloud's behalf. We'd actually have to engage with them more tightly using this base image, since it doesn't provide a package manager.

In terms on maintenance, I see this similarly to bitnami/minidev, another stripped-down base distro that Bitnami uses for their Elasticsearch image. We might also choose to use it as a basis for e.g. Logstash at some point down the line.

BTW I haven't forgotten about docker-slim - there are some changes in the works that might make it usable.

[mark-vieira]

One thing to bear in mind here is that a reduction in size isn't strictly the goal. Rather, we want to reduce the amount of unnecessary stuff, which can e.g. trigger vulnerability warnings in security scans.

Thanks, Rory. That's important context and I would have known that had I actually read #51670 😉

That said, since there's nothing secret in the base image, and it's unlikely to change frequently, I had anticipated publishing it to Docker Hub as usual, perhaps by Infra if they were willing to own.

👍 That would save us a lot of redundant effort and headache in our build if we could go this route. We'd need to figure out a lot of details here but IMO that would my preference I think so we could rip out all that additional complexity from our build and basically the only thing that would change in the ES repo is the base image used in our Dockerfile.

[pugnascotia]

Still looking for feedback on this PR.

[joshbressers]

Can we still run "yum update" inside the container after these changes?

[pugnascotia]

No yum or rpm I'm afraid. It's true that it's handy when you're fooling around in an image to be able to do that, but it adds a lot of MBs to the image, as well as bundling an entire development environment (Python). The main goal is being more careful about what's in the final image, and we'd prefer to avoid shipping unnecessary libraries and binaries.

[jasontedor]

@elasticmachine update branch

[elasticmachine]

merge conflict between base and head

[jasontedor]

@pugnascotia I like the direction here, but I'll withhold a thorough review until after you take this out of draft stage. Let me know it's ready for a thorough review.

[pugnascotia]

I've updated this PR with master, which unfortunately means the Dockerfile now combines the UBI work and the tiny image work. It's not pretty, and we could split it so that we have separate files for UBI and non-UBI...but there are steps in common, which would increase the maintenance burden. I'l like to hear everyone's opinion here, particularly @mark-vieira, @rjernst and @jasontedor.

[mark-vieira]

I don't think it's a big deal. I prefer this to duplication since Docker doesn't have a good solution for "sharing" logic aside from base images which isn't ideal for this use case.

[mieciu]

LGTM in behalf of Cloud.
We're ready to incorporate those changes 👍

[pugnascotia]

@jasontedor Latest stats:

Measure	CentOS 8	Scratch
docker images size	771MB	558MB
Number of files	27829	18797
Number of executables in /usr/{bin,sbin}	582	431
docker save ... | gzip size	379M	311M

In terms of reducing attack surface, we would ship about a third fewer files, notably omitting the standard CentOS stuff like systemd and yum, as well as python and a bunch of libraries.

We should see a greater size reduction on ARM, because the CentOS image ships a whole bunch of device drivers (at least I think that's what they are?), which adds hundreds of megabytes to the image. I actually can't remember when I last tried this build on ARM, however, so I should do that before merging.

[pugnascotia]

OK, I spun up an ARM images in AWS and checked that the branch builds OK, which it does. The docker images size is about the same. I forgot to capture any more stats before I terminated the instance, unfortunately, but I don't expect much variation.

Entertainingly, the ARM version of centos:8 is now only 255MB, vs 215MB for x86_64, whereas I'm pretty sure it was much higher for centos:7. But still, we're saving space.

So...I guess we're good to go. Eek.

[tarekziade]

@pugnascotia have we considered using rhel-minimal as a base ? it strips yum, python and more stuff as well, but provides microdnf

[pugnascotia]

Can you point me at the image? We can only use an official image on Docker Hub, because one of requirements for publishing official images is that you extend an official image (or scratch).

[tarekziade]

Ah I see, I was thinking about https://catalog.redhat.com/software/containers/rhel-minimal/5b4a52bbbed8bd2fe667e449
or
https://hub.docker.com/r/redhat/ubi8-minimal

so yeah no "official" image

[michelkluger]

is there a slim/alpine version of elastic search I could use for my CI? instead of pulling 640mb?

[mark-vieira]

is there a slim/alpine version of elastic search I could use for my CI? instead of pulling 640mb?

We do not offer an image based on alpine linux for a couple of reasons. Firstly, the majority of the size of the image is due to the included JDK, so using a "slimmer" base image won't actually reduce the size that much. Also, we found that there were compatibility issues when running against musl libc vs glibc.