Cherry-pick #21994 to 7.10: [Elastic Agent] Fix missing elastic_agent event data

[blakerouse]

Cherry-pick of PR #21994 to 7.10 branch. Original message:

What does this PR do?

Fixes the events pushed by Elastic Agent monitoring to have the elastic_agent.* fields. This also updates the fields from elastic.agent.id to elastic_agent.id to match the datasource that was change to elastic_agent.

Why is it important?

So all events can be coordinated to an Agent.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes Elastic agent monitoring logs/metrics doesn't have elastic.agent.* fields #21864

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [Pull request #22033 updated]

• Start Time: 2020-10-21T12:06:22.853+0000

• Duration: 38 min 27 sec

Test stats 🧪

Test	Results
Failed	4
Passed	1368
Skipped	4
Total	1376

Test errors

Expand to view the tests failures

• Name: Build&Test / x-pack/elastic-agent-build / TestFleetGateway – application

  • Age: 1
  • Duration: 0
  • Error Details: Failed

• Name: Build&Test / x-pack/elastic-agent-build / TestFleetGateway/send_no_event_and_receive_no_action – application

  • Age: 1
  • Duration: 0
  • Error Details: Failed

• Name: Build&Test / x-pack/elastic-agent-build / TestFleetGateway/Successfully_connects_and_receives_a_series_of_actions – application

  • Age: 1
  • Duration: 0
  • Error Details: Failed

• Name: Build&Test / x-pack/elastic-agent-build / TestFleetGateway/Periodically_communicates_with_Fleet – application

  • Age: 1
  • Duration: 0
  • Error Details: Failed

Steps errors

Expand to view the steps failures

• Name: x-pack/elastic-agent-build - mage build test

  • Description: mage build test

  • Duration: 12 min 53 sec

  • Start Time: 2020-10-21T12:31:50.181+0000

  • log

Log output

Expand to view the last 100 lines of log output

[2020-10-21T12:43:44.082Z] c1e54eec4b57: Pulling fs layer
[2020-10-21T12:43:44.344Z] c1e54eec4b57: Verifying Checksum
[2020-10-21T12:43:44.344Z] c1e54eec4b57: Download complete
[2020-10-21T12:43:44.605Z] c1e54eec4b57: Pull complete
[2020-10-21T12:43:44.605Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2020-10-21T12:43:44.605Z] Status: Downloaded newer image for alpine:3.4
[2020-10-21T12:43:46.824Z] + python .ci/scripts/pre_archive_test.py
[2020-10-21T12:43:46.824Z] Copy ./x-pack/elastic-agent/build into build/x-pack/elastic-agent/build
[2020-10-21T12:43:46.834Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22033/src/github.com/elastic/beats/build
[2020-10-21T12:43:47.141Z] + rm -rf ve
[2020-10-21T12:43:47.141Z] + find . -type d -name vendor -exec rm -r {} ;
[2020-10-21T12:43:47.159Z] Recording test results
[2020-10-21T12:43:48.013Z] Stashed 1 file(s)
[2020-10-21T12:43:48.324Z] + tar --version
[2020-10-21T12:43:48.634Z] + tar --exclude=test-build-artifacts-x-pack/elastic-agent-build.tgz -czf test-build-artifacts-x-pack/elastic-agent-build.tgz .
[2020-10-21T12:43:51.213Z] [Google Cloud Storage Plugin] Found 1 files to upload from pattern: test-build-artifacts-x-pack/elastic-agent-build.tgz
[2020-10-21T12:43:51.610Z] [Google Cloud Storage Plugin] Uploading: test-build-artifacts-x-pack/elastic-agent-build.tgz
[2020-10-21T12:43:52.894Z] + python .ci/scripts/search_system_tests.py
[2020-10-21T12:43:52.945Z] [INFO] system-tests=''. If no empty then let's create a tarball
[2020-10-21T12:43:53.275Z] Client: Docker Engine - Community
[2020-10-21T12:43:53.275Z]  Version:           19.03.13
[2020-10-21T12:43:53.275Z]  API version:       1.40
[2020-10-21T12:43:53.275Z]  Go version:        go1.13.15
[2020-10-21T12:43:53.275Z]  Git commit:        4484c46d9d
[2020-10-21T12:43:53.275Z]  Built:             Wed Sep 16 17:02:36 2020
[2020-10-21T12:43:53.275Z]  OS/Arch:           linux/amd64
[2020-10-21T12:43:53.275Z]  Experimental:      false
[2020-10-21T12:43:53.275Z] 
[2020-10-21T12:43:53.275Z] Server: Docker Engine - Community
[2020-10-21T12:43:53.275Z]  Engine:
[2020-10-21T12:43:53.275Z]   Version:          19.03.13
[2020-10-21T12:43:53.275Z]   API version:      1.40 (minimum version 1.12)
[2020-10-21T12:43:53.275Z]   Go version:       go1.13.15
[2020-10-21T12:43:53.275Z]   Git commit:       4484c46d9d
[2020-10-21T12:43:53.275Z]   Built:            Wed Sep 16 17:01:06 2020
[2020-10-21T12:43:53.275Z]   OS/Arch:          linux/amd64
[2020-10-21T12:43:53.275Z]   Experimental:     false
[2020-10-21T12:43:53.275Z]  containerd:
[2020-10-21T12:43:53.275Z]   Version:          1.3.7
[2020-10-21T12:43:53.275Z]   GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
[2020-10-21T12:43:53.275Z]  runc:
[2020-10-21T12:43:53.275Z]   Version:          1.0.0-rc10
[2020-10-21T12:43:53.275Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-10-21T12:43:53.275Z]  docker-init:
[2020-10-21T12:43:53.275Z]   Version:          0.18.0
[2020-10-21T12:43:53.275Z]   GitCommit:        fec3683
[2020-10-21T12:43:57.994Z] Failed in branch x-pack/elastic-agent-build
[2020-10-21T12:44:02.900Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-10-21T12:44:02.910Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-10-21T12:44:02.973Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-22033-3/source/source.tgz
[2020-10-21T12:44:02.992Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-22033-3/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-22033/source.tgz
[2020-10-21T12:44:14.218Z] + tar --version
[2020-10-21T12:44:14.514Z] + tar -xpf source.tgz
[2020-10-21T12:44:27.094Z] + rm source.tgz
[2020-10-21T12:44:27.144Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22033/src/github.com/elastic/beats
[2020-10-21T12:44:27.153Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22033/src/github.com/elastic/beats/x-pack-elastic-agent-windows-windows-2019-1603284074611
[2020-10-21T12:44:27.209Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22033/src/github.com/elastic/beats/x-pack-elastic-agent-build-1603284227876
[2020-10-21T12:44:27.538Z] + cat
[2020-10-21T12:44:27.538Z] + /usr/local/bin/runbld ./runbld-test-reports --job-name elastic+beats+pull-request
[2020-10-21T12:44:27.538Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-10-21T12:44:34.162Z] runbld>>> runbld started
[2020-10-21T12:44:34.162Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-10-21T12:44:35.558Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-10-21T12:44:35.820Z] runbld>>> Matches in the system config:
[2020-10-21T12:44:35.820Z] runbld>>> - Matched ^elastic\+beats
[2020-10-21T12:44:35.820Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-10-21T12:44:37.207Z] runbld>>> Debug logging enabled.
[2020-10-21T12:44:37.207Z] runbld>>> Storing result
[2020-10-21T12:44:37.207Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-10-21T12:44:37.207Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201021124436-601E46EC
[2020-10-21T12:44:37.207Z] runbld>>> Adding system facts.
[2020-10-21T12:44:38.595Z] runbld>>> Adding vcs info for the latest commit:  717bf6e4d596c4a119e696e5ccb69c7f6519a8e6
[2020-10-21T12:44:38.595Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-10-21T12:44:38.595Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-10-21T12:44:38.595Z] Processing JUnit reports with runbld...
[2020-10-21T12:44:38.595Z] + echo 'Processing JUnit reports with runbld...'
[2020-10-21T12:44:38.857Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-10-21T12:44:38.857Z] runbld>>> DURATION: 33ms
[2020-10-21T12:44:38.857Z] runbld>>> STDOUT: 40 bytes
[2020-10-21T12:44:38.857Z] runbld>>> STDERR: 49 bytes
[2020-10-21T12:44:38.857Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-10-21T12:44:38.857Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-22033
[2020-10-21T12:44:39.802Z] runbld>>> Storing build metadata: 
[2020-10-21T12:44:39.802Z] runbld>>> Adding test report.
[2020-10-21T12:44:39.802Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-22033/src/github.com/elastic/beats
[2020-10-21T12:44:40.745Z] runbld>>> Found 2 test output files
[2020-10-21T12:44:41.008Z] runbld>>> Test output logs contained: Errors: 0 Failures: 4 Tests: 1376 Skipped: 0
[2020-10-21T12:44:41.268Z] runbld>>> Storing result
[2020-10-21T12:44:41.268Z] runbld>>> FAILURES: 4
[2020-10-21T12:44:42.209Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-10-21T12:44:42.209Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201021124436-601E46EC
[2020-10-21T12:44:42.470Z] runbld>>> Email notification disabled by environment variable.
[2020-10-21T12:44:42.470Z] runbld>>> Slack notification disabled by environment variable.
[2020-10-21T12:44:48.772Z] Running on worker-1244230 in /var/lib/jenkins/workspace/Beats_beats_PR-22033
[2020-10-21T12:44:48.838Z] [INFO] getVaultSecret: Getting secrets
[2020-10-21T12:44:48.886Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-10-21T12:44:50.742Z] + chmod 755 generate-build-data.sh
[2020-10-21T12:44:50.742Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22033/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22033/runs/3 FAILURE 2306504
[2020-10-21T12:44:50.742Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22033/runs/3/steps/?limit=10000 -o steps-info.json
[2020-10-21T12:44:51.442Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22033/runs/3/tests/?status=FAILED -o tests-errors.json