Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #18973 to 7.x: Switch the GRPC communication where Agent is running the server and the beats are connecting back to Agent #19108

Merged
merged 2 commits into from
Jun 12, 2020

Conversation

blakerouse
Copy link
Contributor

Cherry-pick of PR #18973 to 7.x branch. Original message:

What does this PR do?

This completes the switch of the GRPC protocol so that the beat application will connect to Agent using the GRPC server running inside the Agent process. This uses the new protocol that allows bi-directional communication for both configuration, status, and actions.

This also completely removes the old GRPC code and protocol files, which are no longer used or needed.

Why is it important?

To only require 1 listening port on the host machine running the GRPC. To use the new elastic-agent-client code to make it easier for new applications to be controlled by Agent.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

…he beats are connecting back to Agent (elastic#18973)

* Update libbeat fleet management to use elastic-agent-client.

* Work on switching to the new GRPC.

* More on refactor

* Add back the state.

* Format and check

* Add changelog.

* Cleanup.

* mage fmt in x-pack/libbeat.

* Update go.mod.

* Fix from review.

* Fix NewFromConfig comment

* Update go.mod

* Fix imports.

* Fix some locking issues from review.

* Fix lots of issues, add unit testing to cover restarts on failure reporting or crashes.

* Update go.sum

* Fix TestConfigurableRun.

* Fix range over registered apps in GRPC server using RWLock, switch to sync.Map.

* Fix config.

* Fix tests to work on Windows.

* Fix operation start to not start the same application twice.

* Fix enabling and disabling of monitoring.

(cherry picked from commit 0c15394)
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 10, 2020
@blakerouse blakerouse added Team:Ingest Management and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 10, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ingest-management (Team:Ingest Management)

@blakerouse blakerouse self-assigned this Jun 10, 2020
@elasticmachine
Copy link
Collaborator

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19108 opened]

  • Start Time: 2020-06-10T14:55:18.166+0000

  • Duration: 71 min 14 sec

Test stats 🧪

Test Results
Failed 0
Passed 6273
Skipped 1058
Total 7331

Steps errors

Expand to view the steps failures

  • Name: Make -C filebeat testsuite

    • Description: make -C filebeat testsuite

    • Duration: 7 min 11 sec

    • Start Time: 2020-06-10T15:19:05.787+0000

    • log

  • Name: Install docker-compose 1.21.0

    • Description: .ci/scripts/install-docker-compose.sh

    • Duration: 1 min 57 sec

    • Start Time: 2020-06-10T15:18:24.565+0000

    • log

  • Name: Make -C libbeat testsuite

    • Description: make -C libbeat testsuite

    • Duration: 8 min 16 sec

    • Start Time: 2020-06-10T15:18:28.896+0000

    • log

  • Name: Install docker-compose 1.21.0

    • Description: .ci/scripts/install-docker-compose.sh

    • Duration: 1 min 57 sec

    • Start Time: 2020-06-10T15:18:22.257+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-10T16:06:01.514Z] + exit 0
[2020-06-10T16:06:02.001Z] + curl -sSLo codecov https://codecov.io/bash
[2020-06-10T16:06:02.265Z] + FILE=auditbeat/build/coverage/full.cov
[2020-06-10T16:06:02.265Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-06-10T16:06:02.265Z] + FILE=filebeat/build/coverage/full.cov
[2020-06-10T16:06:02.265Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-06-10T16:06:02.265Z] + FILE=heartbeat/build/coverage/full.cov
[2020-06-10T16:06:02.265Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-06-10T16:06:02.265Z] + FILE=libbeat/build/coverage/full.cov
[2020-06-10T16:06:02.265Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-06-10T16:06:02.265Z] + FILE=metricbeat/build/coverage/full.cov
[2020-06-10T16:06:02.265Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-06-10T16:06:02.265Z] + FILE=packetbeat/build/coverage/full.cov
[2020-06-10T16:06:02.265Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-06-10T16:06:02.265Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-06-10T16:06:02.265Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-06-10T16:06:02.265Z] + FILE=journalbeat/build/coverage/full.cov
[2020-06-10T16:06:02.265Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-06-10T16:06:04.811Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats
[2020-06-10T16:06:05.151Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-10T16:06:05.174Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Lint
[2020-06-10T16:06:05.324Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-10T16:06:05.476Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-10T16:06:05.736Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-10T16:06:05.885Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-10T16:06:06.037Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-10T16:06:06.191Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-10T16:06:06.348Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-10T16:06:06.502Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Filebeat-oss
[2020-06-10T16:06:06.656Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-06-10T16:06:06.812Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Libbeat-oss
[2020-06-10T16:06:06.963Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-10T16:06:07.116Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-10T16:06:07.268Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-10T16:06:07.423Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-10T16:06:07.574Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-10T16:06:07.723Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-10T16:06:07.874Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-10T16:06:08.026Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-10T16:06:08.183Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-10T16:06:08.346Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-10T16:06:08.502Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-10T16:06:08.662Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-10T16:06:08.812Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-10T16:06:08.962Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-06-10T16:06:09.117Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-10T16:06:09.272Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-10T16:06:09.425Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-10T16:06:09.575Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Heartbeat-Windows
[2020-06-10T16:06:09.728Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Functionbeat-Windows
[2020-06-10T16:06:09.876Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-10T16:06:10.025Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-10T16:06:10.174Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-06-10T16:06:10.609Z] + cat
[2020-06-10T16:06:10.609Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-10T16:06:10.609Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-10T16:06:17.197Z] runbld>>> runbld started
[2020-06-10T16:06:17.197Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-06-10T16:06:18.583Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19108' in order of occurrence in the config (last value wins).
[2020-06-10T16:06:19.972Z] runbld>>> Debug logging enabled.
[2020-06-10T16:06:19.972Z] runbld>>> Storing result
[2020-06-10T16:06:19.972Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-10T16:06:19.972Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200610160619-3F2697AE
[2020-06-10T16:06:19.972Z] runbld>>> Adding system facts.
[2020-06-10T16:06:20.915Z] runbld>>> Adding vcs info for the latest commit:  20fc28681ae6ae8db2654a1410d32d4bb3eef1c0
[2020-06-10T16:06:21.487Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-10T16:06:21.487Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-10T16:06:21.487Z] Processing JUnit reports with runbld...
[2020-06-10T16:06:21.487Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-10T16:06:21.747Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-10T16:06:21.747Z] runbld>>> DURATION: 18ms
[2020-06-10T16:06:21.747Z] runbld>>> STDOUT: 40 bytes
[2020-06-10T16:06:21.747Z] runbld>>> STDERR: 49 bytes
[2020-06-10T16:06:21.747Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-10T16:06:21.747Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats
[2020-06-10T16:06:23.134Z] runbld>>> Storing build metadata: 
[2020-06-10T16:06:23.134Z] runbld>>> Adding test report.
[2020-06-10T16:06:23.134Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats
[2020-06-10T16:06:24.518Z] runbld>>> Found 97 test output files
[2020-06-10T16:06:24.518Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-06-10T16:06:24.518Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-06-10T16:06:24.518Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-06-10T16:06:24.518Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-06-10T16:06:24.518Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-06-10T16:06:24.518Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-06-10T16:06:24.779Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-06-10T16:06:24.779Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-06-10T16:06:25.722Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 7181 Skipped: 855
[2020-06-10T16:06:25.722Z] runbld>>> Storing result
[2020-06-10T16:06:25.722Z] runbld>>> FAILURES: 0
[2020-06-10T16:06:25.982Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-10T16:06:25.982Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200610160619-3F2697AE
[2020-06-10T16:06:31.707Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19108
[2020-06-10T16:06:31.875Z] [INFO] getVaultSecret: Getting secrets
[2020-06-10T16:06:31.948Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-10T16:06:32.860Z] + chmod 755 generate-build-data.sh
[2020-06-10T16:06:32.860Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19108/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19108/runs/1 FAILURE 4274423
[2020-06-10T16:06:32.860Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19108/runs/1/steps/?limit=10000 -o steps-info.json
[2020-06-10T16:06:34.204Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19108/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-06-10T16:06:34.454Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19108/runs/1/log/ -o pipeline-log.txt

Copy link
Contributor

@michalpristas michalpristas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

havent tested but backport looks identical

@blakerouse blakerouse merged commit 681da1c into elastic:7.x Jun 12, 2020
@blakerouse blakerouse deleted the backport_18973_7.x branch June 12, 2020 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants