Make it possible to use Stack logging in Docker #65778
Conversation
Closes elastic#62758. Include the Stack log4j config in the Docker image, in order to make it possible to write logs in a container environment in the same way as for an archive or package deployment. This is useful in situations where the user is bind-mounting the logs directory and has their own arrangements for log shipping.
pugnascotia
added
>enhancement
:Delivery/Packaging
|
Pinging @elastic/es-delivery (Team:Delivery) |
There was a problem hiding this comment.
This doesn't seem to address the configuration drift issue as described in #62758. So we still have the problem of multiple logging config files that need to be kept in since. If we still intend to address that we should open a separate issue for that.
Second, and mostly from curiosity, is there a specific use case for using the file-based logging? Is this something cloud requested?
Rather than maintaining a separate log4j config specifically for Docker, we can generate a config by processing the ES distribtution's default configuration so that everything is emitted to the console.
|
@mark-vieira yes, Cloud don't want to write to the console since they use filebeat to ship their logs, and would prefer to be able to use the default logging config. I also make an attempt at transforming the default config into a console-only config. The major constraint on the implementation here is the need to publish reproducible build files to Docker hub. It means that all you have to work with is the archive distro of ES and whatever you put in the build context. For that reason I opted for a jshell script, since the v8.0 image has almost nothing useful in it otherwise. We could probably change it to a standard Java class if you think that using jshell is too gonzo, but we'd still need to put the source in the Docker context in order to provide reproducible builds. Same thing goes for Iron Bank - they build the images, not us. |
|
BTW I ran the Docker test for the default and UBI distros locally, and also build the Iron Bank image. |
|
@elasticmachine run elasticsearch-ci/2 - known issue. |
My main concern is lack of test coverage. I say let's just punt on that for now. We don't have to solve the config drift issue as part of this PR. My preference however would be to use some kind of log4j2 construct here. A custom "tee" appender perhaps that could send logging to one of two locations based on some config/env var/etc. |
|
Looking at the RoutingAppender that you pointed me at, I believe we'd need to configure it for every appender that we currently, which would be pretty ugly. I'd rather make the transformer script testable. What about moving it to |
That's probably messy to call |
|
@mark-vieira OK, we're now using an executable jar instead of jshell. @breskeby any thoughts on improving the new sub-project? |
|
@breskeby I'd particularly like a review on the new gradle config in this PR. |
There was a problem hiding this comment.
lgtm. some minor remarks.
I remember there were memory leaking issues reported with using Groovy template expansion as we rely on here in our docker file setups atm. Therefore in the long term ideally we would move away from google template expansion (but totally out of scope here, just as a general comment as we also havn't received related issues reported for the es build afaik).
distribution/docker/build.gradle
Outdated
|
|
||
| /** | ||
| * Used in the Dockerfile template to wrap commands in a retry loop. It can't be | ||
| * a closure because Gradle can't effectively cache those. |
There was a problem hiding this comment.
Can you elaborate what can't be cached here?
distribution/docker/build.gradle
Outdated
| * Used in the Dockerfile template to wrap commands in a retry loop. It can't be | ||
| * a closure because Gradle can't effectively cache those. | ||
| */ | ||
| class Retry { |
There was a problem hiding this comment.
In general I'm in favour to have class definitions living in buildSrc somewhere no matter how small they are. To keep our build scripts a bit cleaner and smaller. Also these classes should change less often than the script itself.
distribution/docker/build.gradle
Outdated
| @@ -100,7 +108,7 @@ ${indent}${exitKeyword} \$exit_code""" | |||
| 'docker_base' : base.name().toLowerCase(), | |||
| 'version' : VersionProperties.elasticsearch, | |||
| 'major_minor_version' : "${major}.${minor}", | |||
| 'retry_loop' : retry_loop | |||
| 'retry' : Retry, | |||
There was a problem hiding this comment.
I think the , is too much here
|
@breskeby I moved the retry class to The logic for build the the |
|
@elasticmachine update branch |
Backport of elastic#65778. Closes elastic#62758. Include the Stack log4j config in the Docker image, in order to make it possible to write logs in a container environment in the same way as for an archive or package deployment. This is useful in situations where the user is bind-mounting the logs directory and has their own arrangements for log shipping. To use stack logging, set the environment variable `ES_LOG_STYLE` to `file`. It can also be set to `console`, which is the same as not specifying it at all. The Docker logging config is now auto-generated at image build time, by running the default config through a transformer program when preparing the distribution in an image builder step. Also, in the docker distribution `build.gradle`, I changed a helper closure into a class with a static method in order to fix an issue where the Docker image was always being rebuilt, even when there were no changes.
Backport of #65778. Closes #62758. Include the Stack log4j config in the Docker image, in order to make it possible to write logs in a container environment in the same way as for an archive or package deployment. This is useful in situations where the user is bind-mounting the logs directory and has their own arrangements for log shipping. To use stack logging, set the environment variable `ES_LOG_STYLE` to `file`. It can also be set to `console`, which is the same as not specifying it at all. The Docker logging config is now auto-generated at image build time, by running the default config through a transformer program when preparing the distribution in an image builder step. Also, in the docker distribution `build.gradle`, I changed a helper closure into a class with a static method in order to fix an issue where the Docker image was always being rebuilt, even when there were no changes.
Closes #62758.
Include the Stack log4j config in the Docker image, in order to make it
possible to write logs in a container environment in the same way as for
an archive or package deployment. This is useful in situations where the
user is bind-mounting the logs directory and has their own arrangements
for log shipping.
To use stack logging, set the environment variable
ES_LOG_STYLEtofile. It can also be set toconsole, which is the same as notspecifying it at all.
The Docker logging config is now auto-generated at image build time, by
running the default config through a transformer program when preparing
the distribution in an image builder step.
Implementation notes:
build.gradle, I changed a helper closure into a class with a static method in order to work around an issue where the Docker image was always being rebuilt, even when there were no changes.