Add support for multiple sets of hints on autodiscover

[vjsamuel]

Enhancement

What does this PR do?

This PR allows defining multiple sets of annotations similar to how processors are defined. This functionality already exists in heartbeat. This PR standardizes the utility and adds it to logs and metrics hints builder as well.

Why is it important?

A container can always have more than one kind of endpoint that requires different kinds of processing needs. We need to provide a uniform experience across all beats from a hints perspective.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

• Check for regressions:
  • Use heartbeat with hints enabled and setup a pod with multiple monitors, this should continue working
  • Use metricbeat or filebeat with hints enabled and setup a pod with hints as usually configured, this should continue working.
  • Repeat the previous test with container-specific hints.
• Test the new feature:
  • Use metricbeat or filebeat with hints enabled, and setup a pod with multiple sets of hints (there are some examples in the docs added in this PR), check that multiple modules are instantiated for the pod.
  • Repeat the previous test with container-specific hints.

Related issues

• Add support for heartbeat hints builder #13119 - PR where this feature was added for Heartbeat

Use cases

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #18883 updated]

• Start Time: 2020-06-22T06:43:43.409+0000

• Duration: 80 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9481
Skipped	1574
Total	11055

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[exekias]

@ChrsMark @jsoriano what do you think about this one? @vjsamuel could you please explain a little bit about the use case that you are trying to solve?

[vjsamuel]

this allows use cases like:

co.elastic.metrics/1.module: prometheus
co.elastic.metrics/1.hosts: ${data.host}:8080/metrics
co.elastic.metrics/1.period: 1m

co.elastic.metrics/2.module: prometheus
co.elastic.metrics/2.hosts: ${data.host}:8080/app/metrics
co.elastic.metrics/2.period: 10s

this is currently not doable.

[jsoriano]

I think this feature makes sense, also thinking on cases like kafka, where we may want to do something like this:

co.elastic.metrics/1.module: kafka
co.elastic.metrics/1.metricsets: consumergroup,partition
co.elastic.metrics/1.hosts: ${data.host}:9092

co.elastic.metrics/2.module: kafka
co.elastic.metrics/2.metricsets: broker
co.elastic.metrics/2.hosts: ${data.host}:8778

So this enhancement has my +1, but I also think that it increases the complexity and we have to be very careful.

We have to be clear on how this feature would work, for example, would a mix of numbered and unnumbered configurations work? and if it is, what configs would it produce?
Something like the following would be allowed? would it create two configs or only one? what would be the period?

co.elastic.metrics/module: prometheus
co.elastic.metrics/hosts: ${data.host}:8080/metrics
co.elastic.metrics/period: 1m
co.elastic.metrics/1.module: prometheus
co.elastic.metrics/1.hosts: ${data.host}:8080/app/metrics

We also need to be clear on how we are going to document this. Autodiscover is already quite powerful, but also sometimes difficult to understand by some users. Currently we document the possible annotations with its whole name (e.g: co.elastic.metrics/module), we are going to think very well how to add the documentation for this feature.

And we are also going to need thorough testing, this feature opens the gate to many kinds of cases.

[jsoriano]

@vjsamuel thanks for remarking that this is a feature already available in heartbeat, I see this as an strong point to make it available as is in other beats 🙂 And thanks for improving the documentation!

I have been taking a quick look to the PR that introduced hints in Heartbeat (#13119), and there you mentioned that Heartbeat seems to be a unique use case as a user might want more than one check on his/her pod/container. Out of curiosity, what made you change your opinion? Is there a use case worth mentioning for other beats?

I am going to add this PR to the manual test plan so we will make additional manual testing before releasing it, I have added some notes for it in the how to test section in the issue description, please review it and feel free to modify it.

[vjsamuel]

What helped change my mind? One of the reasons is that the prometheus endpoint having the jvm/go runtime metrics causes a stop the world pause to get the memory snapshot. for latency sensitive applications we prefer not to scrape these endpoints at 5s granularity. However, we do have application metrics that need such scrape intervals. so, the natural tendency is to push those metrics to a different registry and scrape them at a lower granularity as compared to the runtime metrics.

I had no good reason to do it for filebeat but its a great feature :)

[jsoriano]

LGTM, improvements in docs and tests cover my original concerns, thanks @vjsamuel!

[jsoriano]

jenkins, run the tests please