Include the default log4j config in Docker images

[pugnascotia]

At present we ship a custom log4j configuration in our Docker images. This done to make the images more "Docker-like", since the convention with Docker is to write all container output to stdout and stderr.

This is not ideal, because:

• It is more difficult for Filebeat to use the right event.dataset value
• There can be drift between the configurations
• Users can't easily choose to use the default logging config and have to bind-mount their own config
• Writing to stdout / stderr doesn't extend to the logging configurations that plugins can supply

Elasticsearch should make it easy to pick what style of logging to use in Docker. Ideally, the stdout/stderr logging config should be automatically derived from the default configs, so as to minimise drift.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Logging)

[pugnascotia]

I spent some time time thinking about this. I did consider whether we should simply stop tailoring logging for Docker in v8.0. This would be a simple solution, but users relying on the current logging arrangement would then have to recreate it themselves. Logging to stderr / stdout remains the expected default behaviour in the Docker world.

So then I wondered about shipping a log4j2.stack.properties file and a log4j2.docker.properties file in the Docker image, and changing the entrypoint script to symlink the appropriate file as log4j2.properties at runtime. This would only happen if there were no existing log4j2.properties file, so a user mounting their own config directory or properties file would be unaffected. A new environment variable would indicate the preferred logging style, for example ES_LOG_STYLE=stack or something like that. We might want to build the image with a default symlink pointing to the docker config, so that ES can still start if the entrypoint didn't run ES directly. This strategy with symlinks would avoid making changes in the LogConfigurator class.

There's also the question of whether it's possible to automatically generate a Docker log4j2.properties from the default copy. I actually wondered whether it would be easier to concoct some kind of meta-config, and generate both the stack and docker templates from that. Just a though, and probably an impractical one. It'll still likely be easier to somehow parse the stack config, and output a Docker variant.

[pugnascotia]

One other possibility would be to convert the standard logging config to a docker-style config at runtime. While more complex, this would pick up any log configurations for installed plugins as well. Making this work in practise would likely be tricky, however.

[pugnascotia]

In the course of looking into this, I started wondering if we should make it possible to direct GC logs to stderr, instead of to file as we do at present. It's a separate concern, and I think we'd need to make it opt-in because the usual settings that we apply result in a lot of logs, and stderr obviously has no file rotation.