[Elastic Agent] Add ability to communicate with Kibana through service token #28096

blakerouse · 2021-09-23T13:05:45Z

What does this PR do?

Adds ability for the libbeat Kibana client to communicate with Kibana using an ES service token. This is used by the container sub-command with the KIBANA_FLEET_SERVICE_TOKEN environment variable to communicate with Kibana using the provided ES service token.

Why is it important?

This allows a spawned container of Elastic Agent that will bootstrap Fleet in Kibana to use a service token instead of using a username/password authenticated super user.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes Add support for using a service token to call Kibana Fleet Setup API #28024

botelastic · 2021-09-23T13:05:52Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2021-09-23T13:06:54Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-09-23T13:09:44Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-11T16:00:47.505+0000
Duration: 69 min 45 sec
Commit: 8e66586

Test stats 🧪

Test	Results
Failed	0
Passed	20466
Skipped	1519
Total	21985

Steps errors

Expand to view the steps failures

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

Took 9 min 20 sec . View more details here
Description: make -C generator/_templates/metricbeat test test-package

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

Took 6 min 43 sec . View more details here
Description: make -C generator/_templates/metricbeat test test-package

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

Took 6 min 4 sec . View more details here
Description: make -C generator/_templates/metricbeat test test-package

`Archive system tests files`

Took 0 min 0 sec . View more details here
Description: mage packageSystemTests

`generator-beat-test - make -C generator/_templates/beat test test-package`

Took 10 min 21 sec . View more details here
Description: make -C generator/_templates/beat test test-package

`generator-beat-test - make -C generator/_templates/beat test test-package`

Took 6 min 58 sec . View more details here
Description: make -C generator/_templates/beat test test-package

`generator-beat-test - make -C generator/_templates/beat test test-package`

Took 6 min 54 sec . View more details here
Description: make -C generator/_templates/beat test test-package

`Archive system tests files`

Took 0 min 0 sec . View more details here
Description: mage packageSystemTests

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 2'

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

ruflin · 2021-09-23T13:12:41Z

x-pack/elastic-agent/pkg/agent/cmd/setup_config.go

+				Host:         envWithDefault("http://kibana:5601", "KIBANA_FLEET_HOST", "KIBANA_HOST"),
+				Username:     envWithDefault("elastic", "KIBANA_FLEET_USERNAME", "KIBANA_USERNAME", "ELASTICSEARCH_USERNAME"),
+				Password:     envWithDefault("changeme", "KIBANA_FLEET_PASSWORD", "KIBANA_PASSWORD", "ELASTICSEARCH_PASSWORD"),
+				ServiceToken: envWithDefault("", "KIBANA_FLEET_SERVICE_TOKEN", "FLEET_SERVER_SERVICE_TOKEN"),


As it is the fleet-server service toke, should we indicate this here by calling it KIBANA_FLEET_SERVER_SERVICE_TOKEN ?

It doesn't have to be a Fleet Server token, it could be any token that has the correct permissions. It could be a different token then the one provided to Fleet Server.

I think its best to keep the name KIBANA_FLEET_SERVICE_TOKEN because KIBANA_FLEET_{name} is the same for all other environment variables that control Elastic Agent containers communication to Kibana.

It also matches with the configuration that can be used:

kibana: fleet: service_token: abc123

@joshdover Do we check on the Fleet side if it is a fleet-server service token or if it has the correct permissions?

We'll be checking that is has correct permissions, so it technically doesn't have to be the elastic/fleet-server service token. It just needs to have the "application privilege" that is being granted to the service account by default as defined in this issue: elastic/kibana#112647

So by default, Kibana would accept the elastic/fleet-server service account's token or any credentials for a super user (basic, apikey, etc.)

ruflin · 2021-09-23T13:14:03Z

@joshdover @blakerouse As soon as all components are in place in the different projects it would be good to have an e2e test confirm that this works as expected end to end.

blakerouse · 2021-09-23T13:18:51Z

@joshdover @blakerouse As soon as all components are in place in the different projects it would be good to have an e2e test confirm that this works as expected end to end.

I filed an issue to track it - elastic/e2e-testing#1613

…lity to pass service token to container subcommand.

jlind23 · 2021-10-07T08:43:47Z

@blakerouse anything missing in this PR apart of the e2e test?

joshdover

Changes LGTM and I have been able to verify this works against elastic/kibana#113932

blakerouse · 2021-10-11T15:59:34Z

@jlind23 No was waiting on confirmation that it worked from @joshdover. Now that I know it works, I will work on getting it merged.

blakerouse · 2021-10-11T16:00:03Z

/test

blakerouse · 2021-10-11T18:12:26Z

None of the issues seem to be related to the PR. Looks like unrelated changes that are in master or an issue with the Windows 8 CI runner. Going to merge.

…e token (#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. (cherry picked from commit fbca813)

…e token (#28096) (#28349) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. (cherry picked from commit fbca813) Co-authored-by: Blake Rouse <[email protected]>

…e token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt.

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (#28236) * [cloud][docker] use the private docker namespace (#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (#28297) * seccomp: allow clone3 syscall for x86 (#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (#28253) * Fix build tags for Go 1.17 (#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <[email protected]> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (#28325) * Redis: remove deprecated fields (#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (#27878) partial fix for #27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <[email protected]> Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Ugo Sangiorgi <[email protected]> Co-authored-by: Dan Kortschak <[email protected]> Co-authored-by: Arnaud Lefebvre <[email protected]> Co-authored-by: Aleksandr Maus <[email protected]> Co-authored-by: apmmachine <[email protected]> Co-authored-by: Michael Katsoulis <[email protected]> Co-authored-by: Noémi Ványi <[email protected]> Co-authored-by: Blake Rouse <[email protected]> Co-authored-by: LaZyDK <[email protected]> Co-authored-by: Marius Iversen <[email protected]> Co-authored-by: Andrea Spacca <[email protected]> Co-authored-by: Mariana Dima <[email protected]> Co-authored-by: Andrew Cholakian <[email protected]>

…e token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt.

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (elastic#28236) * [cloud][docker] use the private docker namespace (elastic#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297) * seccomp: allow clone3 syscall for x86 (elastic#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (elastic#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (elastic#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (elastic#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (elastic#28253) * Fix build tags for Go 1.17 (elastic#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (elastic#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <[email protected]> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (elastic#28325) * Redis: remove deprecated fields (elastic#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (elastic#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (elastic#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (elastic#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (elastic#27878) partial fix for elastic#27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <[email protected]> Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Ugo Sangiorgi <[email protected]> Co-authored-by: Dan Kortschak <[email protected]> Co-authored-by: Arnaud Lefebvre <[email protected]> Co-authored-by: Aleksandr Maus <[email protected]> Co-authored-by: apmmachine <[email protected]> Co-authored-by: Michael Katsoulis <[email protected]> Co-authored-by: Noémi Ványi <[email protected]> Co-authored-by: Blake Rouse <[email protected]> Co-authored-by: LaZyDK <[email protected]> Co-authored-by: Marius Iversen <[email protected]> Co-authored-by: Andrea Spacca <[email protected]> Co-authored-by: Mariana Dima <[email protected]> Co-authored-by: Andrew Cholakian <[email protected]>

blakerouse added Team:Elastic-Agent Label for the Agent team v7.16.0 backport-v7.16.0 Automated backport with mergify labels Sep 23, 2021

blakerouse self-assigned this Sep 23, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 23, 2021

blakerouse marked this pull request as ready for review September 23, 2021 13:06

ruflin reviewed Sep 23, 2021

View reviewed changes

blakerouse mentioned this pull request Sep 23, 2021

Add test for Kibana bootstrap using service token elastic/e2e-testing#1613

Open

blakerouse added 3 commits October 6, 2021 09:16

Add ability to communicate with Kibana through service token. Add abi…

ce8f4fa

…lity to pass service token to container subcommand.

Add changelog entry.

0863caa

Fix go fmt.

8e66586

blakerouse force-pushed the agent-container-kibana-service-token branch from 74955ae to 8e66586 Compare October 6, 2021 13:21

joshdover approved these changes Oct 11, 2021

View reviewed changes

joshdover mentioned this pull request Oct 11, 2021

[Fleet] Allow elastic/fleet-server service account to call setup API elastic/kibana#112648

Closed

5 tasks

blakerouse merged commit fbca813 into elastic:master Oct 11, 2021

blakerouse deleted the agent-container-kibana-service-token branch October 11, 2021 18:12

mergify bot mentioned this pull request Oct 11, 2021

[7.x](backport #28096) [Elastic Agent] Add ability to communicate with Kibana through service token #28349

Merged

dedemorton mentioned this pull request Dec 6, 2021

Add 7.16 release notes for Elastic Agent and Fleet elastic/observability-docs#1319

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Elastic Agent] Add ability to communicate with Kibana through service token #28096

[Elastic Agent] Add ability to communicate with Kibana through service token #28096

blakerouse commented Sep 23, 2021 •

edited

Loading

botelastic bot commented Sep 23, 2021

elasticmachine commented Sep 23, 2021

elasticmachine commented Sep 23, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`Archive system tests files`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`Archive system tests files`

`Error signal`

ruflin Sep 23, 2021

blakerouse Sep 23, 2021

ruflin Sep 23, 2021

joshdover Sep 23, 2021

ruflin commented Sep 23, 2021

blakerouse commented Sep 23, 2021

jlind23 commented Oct 7, 2021

joshdover left a comment

blakerouse commented Oct 11, 2021

blakerouse commented Oct 11, 2021

blakerouse commented Oct 11, 2021

[Elastic Agent] Add ability to communicate with Kibana through service token #28096

[Elastic Agent] Add ability to communicate with Kibana through service token #28096

Conversation

blakerouse commented Sep 23, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

botelastic bot commented Sep 23, 2021

elasticmachine commented Sep 23, 2021

elasticmachine commented Sep 23, 2021 • edited by jenkins-beats-ci bot Loading

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package

generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package

generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package

Archive system tests files

generator-beat-test - make -C generator/_templates/beat test test-package

generator-beat-test - make -C generator/_templates/beat test test-package

generator-beat-test - make -C generator/_templates/beat test test-package

Archive system tests files

Error signal

💚 Flaky test report

🤖 GitHub comments

ruflin Sep 23, 2021

Choose a reason for hiding this comment

blakerouse Sep 23, 2021

Choose a reason for hiding this comment

ruflin Sep 23, 2021

Choose a reason for hiding this comment

joshdover Sep 23, 2021

Choose a reason for hiding this comment

ruflin commented Sep 23, 2021

blakerouse commented Sep 23, 2021

jlind23 commented Oct 7, 2021

joshdover left a comment

Choose a reason for hiding this comment

blakerouse commented Oct 11, 2021

blakerouse commented Oct 11, 2021

blakerouse commented Oct 11, 2021

blakerouse commented Sep 23, 2021 •

edited

Loading

elasticmachine commented Sep 23, 2021 •

edited by jenkins-beats-ci bot

Loading

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`Archive system tests files`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`Archive system tests files`

`Error signal`