Add Fleet agent.id to Agent monitoring data #26548

andrewkroh · 2021-06-28T20:42:59Z

What does this PR do?

This is a follow up to #26394 which set the agent.id field to the Fleet agent ID for
for integrations that are run by Agent, but that didn't cover the Filebeat and Metricbeat
processes that are executed by Agent for monitoring itself. This covers those processes.

Relates #26394

Why is it important?

The agent.id of the running processes should match what we show to users in the Fleet UI.

Checklist

My code follows the style guidelines of this project
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates [Elastic Agent] Use Fleet agent ID as agent.id #26394

elasticmachine · 2021-06-28T20:43:02Z

Pinging @elastic/agent (Team:Agent)

This is a follow up to elastic#26394 which set the agent.id field to the Fleet agent ID for for integrations that are run by Agent, but that didn't cover the Filebeat and Metricbeat processes that are executed by Agent for monitoring itself. This covers those processes. Relates elastic#26394

elasticmachine · 2021-06-28T22:01:09Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #26548 updated
Start Time: 2021-06-29T20:42:32.387+0000
Duration: 81 min 4 sec
Commit: 46fdc46

Test stats 🧪

Test	Results
Failed	0
Passed	6972
Skipped	16
Total	6988

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6972
Skipped	16
Total	6988

mergify · 2021-06-29T08:13:06Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/agent/monitoring-agent-id upstream/feature/agent/monitoring-agent-id
git merge upstream/master
git push upstream feature/agent/monitoring-agent-id

mergify · 2021-06-29T20:18:45Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/agent/monitoring-agent-id upstream/feature/agent/monitoring-agent-id
git merge upstream/master
git push upstream feature/agent/monitoring-agent-id

michel-laterman · 2021-06-29T21:59:25Z

x-pack/elastic-agent/pkg/agent/operation/monitoring.go

+				"add_fields": map[string]interface{}{
+					"target": "agent",
+					"fields": map[string]interface{}{
+						"id": o.agentInfo.AgentID(),


out of curiosity, why is this processor not part of the input processors above?

It could go there, but by putting here it should be automatically applied to every input. In effect the processor is appended to the list of each input's processors. This way just means less code duplication.

Apparently in the config transformation the root processors is dropped or not carried over to the final config so that didn't work. I opened #26776 to move add_fields into each input.

This is a follow up to #26394 which set the agent.id field to the Fleet agent ID for for integrations that are run by Agent, but that didn't cover the Filebeat and Metricbeat processes that are executed by Agent for monitoring itself. This covers those processes. Relates #26394 (cherry picked from commit 7b66597)

…26591) * Add Fleet agent.id to Agent monitoring data (#26548) This is a follow up to #26394 which set the agent.id field to the Fleet agent ID for for integrations that are run by Agent, but that didn't cover the Filebeat and Metricbeat processes that are executed by Agent for monitoring itself. This covers those processes. Relates #26394 (cherry picked from commit 7b66597) * Fix changelog merge Co-authored-by: Andrew Kroh <[email protected]>

* master: (25 commits) fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415) macos for metricbeat to run in the extended meta-stage (elastic#26573) Packaging: add arm7 platform in the main pipeline (elastic#26575) [Heartbeat] Skip flakey timer queue test (elastic#26592) Update to "read_pipeline" permission (elastic#26465) (elastic#26580) API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582) Add Fleet agent.id to Agent monitoring data (elastic#26548) Add kinesis metricset (elastic#25989) Refactor of system/memory metricset (elastic#26334) Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219) [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873) docs: Hint for the error "Error extracting container id" (elastic#25824) [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) ...

An earlier commit elastic#26548 set the same value using a single global processor, but that doesn't seem to make it into the final config so it had no effect. So this sets the agent.id at the same places where elastic_agent.id is added.

This is a follow up to #26394 which set the agent.id field to the Fleet agent ID for for integrations that are run by Agent, but that didn't cover the Filebeat and Metricbeat processes that are executed by Agent for monitoring itself. This covers those processes. Relates #26394 (cherry picked from commit 7b66597)

…26776) An earlier commit #26548 set the same value using a single global processor, but that doesn't seem to make it into the final config so it had no effect. So this sets the agent.id at the same places where elastic_agent.id is added.

…26776) An earlier commit #26548 set the same value using a single global processor, but that doesn't seem to make it into the final config so it had no effect. So this sets the agent.id at the same places where elastic_agent.id is added. (cherry picked from commit aa62bba)

…26776) (#26849) An earlier commit #26548 set the same value using a single global processor, but that doesn't seem to make it into the final config so it had no effect. So this sets the agent.id at the same places where elastic_agent.id is added. (cherry picked from commit aa62bba) Co-authored-by: Andrew Kroh <[email protected]>

…26776) (#26850) An earlier commit #26548 set the same value using a single global processor, but that doesn't seem to make it into the final config so it had no effect. So this sets the agent.id at the same places where elastic_agent.id is added. (cherry picked from commit aa62bba) Co-authored-by: Andrew Kroh <[email protected]>

andrewkroh added enhancement review Agent Team:Elastic-Agent Label for the Agent team backport-v7.14.0 Automated backport with mergify labels Jun 28, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 28, 2021

andrewkroh force-pushed the feature/agent/monitoring-agent-id branch from cc106e7 to 3023c56 Compare June 28, 2021 20:44

andrewkroh requested a review from michalpristas June 28, 2021 20:44

Merge branch 'master' into feature/agent/monitoring-agent-id

20574a5

andrewkroh requested a review from a team June 29, 2021 12:38

Merge branch 'master' into feature/agent/monitoring-agent-id

46fdc46

michel-laterman approved these changes Jun 29, 2021

View reviewed changes

andrewkroh merged commit 7b66597 into elastic:master Jun 29, 2021

mergify bot mentioned this pull request Jun 29, 2021

[7.x](backport #26548) Add Fleet agent.id to Agent monitoring data #26591

Merged

andrewkroh mentioned this pull request Jul 7, 2021

Set agent.id to Fleet Agent ID for each metric/log monitoring input #26776

Merged

6 tasks

mergify bot mentioned this pull request Jul 7, 2021

[7.14](backport #26548) Add Fleet agent.id to Agent monitoring data #26777

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Fleet agent.id to Agent monitoring data #26548

Add Fleet agent.id to Agent monitoring data #26548

andrewkroh commented Jun 28, 2021

elasticmachine commented Jun 28, 2021

elasticmachine commented Jun 28, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

mergify bot commented Jun 29, 2021

mergify bot commented Jun 29, 2021

michel-laterman Jun 29, 2021

andrewkroh Jun 29, 2021

andrewkroh Jul 7, 2021

Add Fleet agent.id to Agent monitoring data #26548

Add Fleet agent.id to Agent monitoring data #26548

Conversation

andrewkroh commented Jun 28, 2021

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Jun 28, 2021

elasticmachine commented Jun 28, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

mergify bot commented Jun 29, 2021

mergify bot commented Jun 29, 2021

michel-laterman Jun 29, 2021

Choose a reason for hiding this comment

andrewkroh Jun 29, 2021

Choose a reason for hiding this comment

andrewkroh Jul 7, 2021

Choose a reason for hiding this comment

elasticmachine commented Jun 28, 2021 •

edited by jenkins-beats-ci bot

Loading