Elastic Log Driver: Support "docker logs" via local buffer

[fearful-symmetry]

This has been a long-standing issue in #13990 which we'll want to address sooner rather than later.

Right now, the docker plugin has no support fort the docker logs command, which is a must for cloud adoption. We need to entirely re-implement this behavior in the plugin. This behavior needs to be entirely local, as one of the primary use cases for docker logs is grabbing logs when upstream elasticsearch outputs are down. I've talked with @urso about this, and our best bet to implement this in a short period of time would be to generate a log file for each configured pipeline that we can spit back when the user asks for it. There's also the fs-backed buffer that @faec is working on, although it might not be ready yet. This leaves us with two remaining questions:

• Where in the pipeline do we siphon logs?
• How do we write and manage log files? Does a "reaper" process periodically prune them?

Keep in mind we need to support the following options:

• A "since" date that reports logs after a cutoff
• A "follow" option
• A count of logs to return

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[urso]

This leaves us with two remaining questions:

• Where in the pipeline do we siphon logs?
• How do we write and manage log files? Does a "reaper" process periodically prune them?

Why not follow the JSON logging driver logic 1 by 1 here? We have a go-routine per container. Each would write to it's own directory (named based on container name) into a JSON file that is rotated after 10MB. After the go-routine has written the file it would use libbeat to published the log line as an event.
When the container is deleted we delete the directory.
For example see: https://github.com/cpuguy83/docker-log-driver-test/blob/master/

If we have a way to store the log file in the container directory, then docker will take care of deleting all resources for us. If we decide that we need to keep the files in a separate directory we need to make some limits configurable (e.g. delete log files after container is stopped for ).

[andresrc]

can we close this issue or is there anything missing? Thanks!

[businessbean]

Does this mean that we will have the Recent Log Entries section populated if the Elasticsearch nodes are running in Docker containers (on Kubernetes)?