Docker logging plugin

[exekias]

We aim to provide an official logging plugin for Docker, and push it here: https://store.docker.com/search?type=plugin&category=logging

We don't need to name it Filebeat, but internally it would use most of Filebeat internals. For the initial version it should be able to:

• Retrieve Docker logs
• Include metadata using the same fields as add_docker_metadata
• Make use of spooling to support (short) ES outages
• Allow to have the same experience with hints as with Filebeat

Non-goals for the first version that are interesting for the future:

• Add capability to read logs back in (from ES or the spooling itself), used by docker logs command

Some initial works for it can be found here: https://github.com/elastic/beats/compare/master...exekias:filebeat-docker-plugin?expand=1

[drewr]

Love it!

👋 @elastic/infra

[hholst80]

Which log driver do you intend to support? Only the json-file driver? Or will any driver that allows logs to be replayed via the Docker API be supported? Sorry I misunderstood this. You want to provide a new log driver for Docker that emits data to ES directly, correct?

[exekias]

Hi @hholst80, yes, this would be deployed into docker as a logging plugin and sends logs directly to ES

[fearful-symmetry]

Hey folks. Talked with @exekias about this for a while, and I'd like to pick this back up, since it's a request from cloud and others.

I'm leaning towards the idea of a new beat (in the beats repo, along with everything else), since it would be a good way to dogfood our newbeat dev process, and also because there's no reason I see not to ship a light-weight specialized thing, as opposed to "a whole filebeat." Plugins are normally deployed with (more or less) a single docker plugin command, so this needs to be a fairly simple utility with little need for additional user tweaking.

After Talking with Carlos, it seems that plugins are just a 'weird container' but the build and deployment system is a bit odd, so we can also avoid cluttering up the filebeats directory with additional tooling, and also simplify the deployment process if we have a specialized beat with less knobs to tweak.

Does anyone else here have strong feelings? My one worry with making a new beat is well, we already have a lot, but things like heartbeat/winlogbeat are already 'small and specialized' to some extent.

I'd like to get started with this and pick up where Carlos left off.

[fearful-symmetry]

Continued by #13990