Skip k8s processor for events coming from Agent

[ChrsMark]

What does this PR do?

This PR adds a condition to the add_kubernetes_metadata processor so as to be skipped in case the event is produced by an Agent setup.

Why is it important?

So as to avoid having the add_kubernetes_metadata processor being enabled by default when Filebeat/Metricbeat is deployed via Elastic Agent.

Closes #29767

[botelastic]

This pull request doesn't have a Team:<team> label.

[mergify]

This pull request does not have a backport label. Could you fix it @ChrsMark? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-01-12T11:58:15.770+0000

• Duration: 121 min 47 sec

• Commit: eb7f8eb

Test stats 🧪

Test	Results
Failed	0
Passed	47850
Skipped	4284
Total	52134

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ruflin]

@blakerouse @ph I wonder if there is a nicer way to add this condition for example as part of the transpiler?

It feels strange that we have to make a change in the default beats configs instead of making it in Elastic Agent.

[ChrsMark]

Another option would be to use another default configuration for Beat's config like we do for docker with filebeat.docker.yaml https://github.com/elastic/beats/blob/3c864d26bccdb580a2a7985e9fa902f968cda431/filebeat/filebeat.docker.yml. We can have different configs to be used by Beats when spawn by Elastic Agent.

[ruflin]

I like this idea a lot. This would potentially also solve some other issues where we use the transpiler to set some defaults instead of just modifying a config file. It would also decouple Beats run under Elastic Agent more cleary. It means we would have an filebeat.elastic-agent.yml file?

@cmacknz For awareness.

[ChrsMark]

It means we would have an filebeat.elastic-agent.yml file?

Yes. Not sure how easy it would be technically to happen though.

[blakerouse]

Simply adding -c filebeat.elastic-agent.yml here would do it:

beats/x-pack/elastic-agent/spec/filebeat.yml

Lines 3 to 9 in 56423a0

	args: [
	"-E", "setup.ilm.enabled=false",
	"-E", "setup.template.enabled=false",
	"-E", "management.enabled=true",
	"-E", "logging.level=debug",
	"-E", "gc_percent=${FILEBEAT_GOGC:100}"
	]

[ph]

Yes, what is proposed by @blakerouse seems to be the easiest way to have a completely different set of default options.

[ChrsMark]

So what could be the next step here? I guess I can close this PR if we agree on going with the filebeat.elastic-agent.yml option (I'm +1 for this) and keep #29767 open until we introduce the decoupling? Does this fall to Agent's team ownership?

[ruflin]

I think the control plane team should take this over @jlind23 . Seems we all agree the config file would be a better approach.

@blakerouse If we have the separate config, it seems we can get rid of all the other -E options and just put it in the config file directly?

[blakerouse]

@ruflin Yes we could move them into the configuration file.

[ph]

@ruflin @blakerouse I wonder, if this could open a security risk if a user is able to modify the configuration file?

[blakerouse]

@ph It is possible that it could allow them to add something, but we already have that today with the default configuration anyway.

[ruflin]

I'm closing this PR as I think we all agree we take a different path forward. @ChrsMark Thanks for kicking this off and leading us to the right path.

@ph Could you take the lead in opening up a follow up PR or issue so we can continue the conversation there?

[ph]

created elastic/elastic-agent#90 for the solution please comment on it. I will check with @jlind23 to prioritize it.