Container logs integration

[ChrsMark]

At the moment we don't have a dedicated integration for container logs similar to what we have for the kubernetes' container logs.

It would be nice to have a dedicated package for it. It can be either a container generic one or it can be a new datastream under docker package. I think having a generic one for all types of containers would be preferable.

[BobDu]

I look the document about filebeat and fleet integrations.
If I use filebeat, can choose collect container logs by namespace , pod label, etc....
But if use k8s integration, only start collect all container logs, no switch to off some namespace or pod.

[joshdover]

It would be nice to have a dedicated package for it. It can be either a container generic one or it can be a new datastream under docker package. I think having a generic one for all types of containers would be preferable.

Any reason why we wouldn't want both? From a discoverability angle, I do think users are going to look for this in the Docker package and may be confused why logs aren't supported if they don't see it there.

[ChrsMark]

Hey @BobDu! Sorry for the late replay here, this slipped from my notice (was on vacation that time of your post, sorry).
Your observation is correct and I agree we should address this.
Just to be sure, do you mean that you would like to have the option to specify conditions and only collect logs if those match? I would appreciate if you could provide an example here :).

Fair point @joshdover! From that perspective it is better to have a dedicated data_stream under each of the target environments (docker, k8s etc).

We can extend https://github.com/elastic/integrations/tree/main/packages/docker by using the https://github.com/elastic/elastic-agent/tree/main/internal/pkg/composable/providers/docker provider. Quite similarly to what we do with the k8s provider.

@gizas @mlunadia what do you think about prioritizing this? It seems to be a low hanging fruit.

[ChrsMark]

So I see 2 parts here:

1. support conditions in container_logs data_stream of k8s package. Add by [Kubernetes] Add condition configuration option to container logs data stream #4324 ✅
2. add container_logs data_stream for the docker package. We can extend https://github.com/elastic/integrations/tree/main/packages/docker by using the https://github.com/elastic/elastic-agent/tree/main/internal/pkg/composable/providers/docker provider. Quite similarly to what we do with the k8s provider. PR: Add docker_logs datastream for docker container logs collection #4716 ✅

[ChrsMark]

@gizas since we have addressed both parts, shall we close this issue?

[ChrsMark]

@gizas I'm gonna close this now. Feel free to re-open if there is anything we missed here.