Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Observability] Use Fleet's default data output when onboarding integrations using auto-detect flow #201158

Merged
merged 4 commits into from
Nov 25, 2024
Merged

[Observability] Use Fleet's default data output when onboarding integrations using auto-detect flow #201158

merged 4 commits into from
Nov 25, 2024

Conversation

thomheymann
Copy link
Contributor

@thomheymann thomheymann commented Nov 21, 2024 •
edited by kibanamachine
Loading

Resolves #199751

Summary

Use Fleet's default data output when onboarding integrations using auto-detect flow.

Screenshot

Fleet output settings

Screenshot 2024-11-21 at 15 10 24

Generated Agent config

$ cat /Library/Elastic/Agent/elastic-agent.yml

outputs:
  default:
    type: elasticsearch
    hosts:
      - https://192.168.1.73:9200
    ssl.ca_trusted_fingerprint: c48c98cdf7f85d7cc29f834704011c1002b5251d594fc0bb08e6177544fe304a
    api_key: b1BkR1Q1TUIyOUpfMWhaS2NCUXA6SS1Jb3dncGVReVNpcEdzOGpSVmlzQQ==
    preset: balanced

Testing

  1. Go to Fleet > Settings > Outputs
  2. Edit the default output and enter dummy data into the "Elasticsearch CA trusted fingerprint" field
  3. Go through the auto-detect onboarding flow
  4. Inspect the elastic-agent.yml file written to disk. It should contain the default output configured in Fleet including ca_trusted_fingerprint setting

@thomheymann thomheymann changed the title Use Fleet default output config for auto-detect onboarding [Observability] Use Fleet's default data output when onboarding integrations using auto-detect flow Nov 21, 2024
@thomheymann thomheymann marked this pull request as ready for review November 21, 2024 15:15
@thomheymann thomheymann requested a review from a team as a code owner November 21, 2024 15:15
@thomheymann thomheymann requested a review from a team November 21, 2024 15:15
@thomheymann thomheymann requested a review from a team as a code owner November 21, 2024 15:15
@thomheymann thomheymann added backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) release_note:skip Skip the PR/issue when compiling release notes labels Nov 21, 2024
@botelastic botelastic bot added ci:project-deploy-observability Create an Observability project Team:Fleet Team label for Observability Data Collection Fleet team labels Nov 21, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@github-actions GitHub Actions
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

nchaulet
nchaulet approved these changes Nov 21, 2024
View reviewed changes
Copy link
Member

@nchaulet nchaulet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one suggestion otherwise LGTM 🚀

mykolaharmash
mykolaharmash reviewed Nov 22, 2024
View reviewed changes
Copy link
Contributor

@mykolaharmash mykolaharmash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested locally and it works as expected, though I have a conceptual question. It feels unintuitive that user would need to modify fleet settings in order to establish connection between ES and a standalone agent.

Seems like there is no way for us to get the CA fingerprint directly from the cluster during auto detect flow, but maybe this should be an explicit step in our auto_detect cli script to optionally provide the fingerprint, wdyt? Perhaps we could show this step only if ES host is using https schema...

@thomheymann
Copy link
Contributor Author

I've tested locally and it works as expected, though I have a conceptual question. It feels unintuitive that user would need to modify fleet settings in order to establish connection between ES and a standalone agent.

Seems like there is no way for us to get the CA fingerprint directly from the cluster during auto detect flow, but maybe this should be an explicit step in our auto_detect cli script to optionally provide the fingerprint, wdyt? Perhaps we could show this step only if ES host is using https schema...

Users will not have to do this. It's just a quick way of testing that the CA settings are applied correctly in a local dev environment. When users download the distribution and setup Kibana these settings will be applied automatically.

@elasticmachine
Copy link
Contributor

elasticmachine commented Nov 25, 2024 •
edited
Loading

💚 Build Succeeded

  • Buildkite Build
  • Commit: 79d5f78
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-201158-79d5f78dbf64

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
fleet 1303 1304 +1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
fleet 81 82 +1
Unknown metric groups

API count

id before after diff
fleet 1428 1430 +2

History

@mykolaharmash
Copy link
Contributor

Users will not have to do this. It's just a quick way of testing that the CA settings are applied correctly in a local dev environment. When users download the distribution and setup Kibana these settings will be applied automatically.

Ah, nice, thank you for clarifying!

@thomheymann thomheymann merged commit 845aa8c into elastic:main Nov 25, 2024
26 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12008848720

@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
❌ 8.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 201158

Questions ?

Please refer to the Backport tool documentation

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 26, 2024
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 201158 locally

@thomheymann thomheymann mentioned this pull request Nov 26, 2024
Merged
@thomheymann
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

thomheymann added a commit to thomheymann/kibana that referenced this pull request Nov 26, 2024
@thomheymann
[Observability] Use Fleet's default data output when onboarding integ…
7a68652 
…rations using auto-detect flow (elastic#201158)

Resolves elastic#199751

## Summary

Use Fleet's default data output when onboarding integrations using
auto-detect flow.

## Screenshot

### Fleet output settings

<img width="1411" alt="Screenshot 2024-11-21 at 15 10 24"
src="https://github.com/user-attachments/assets/ac193552-7f18-4566-a84b-061df45c13f3">

### Generated Agent config

```
$ cat /Library/Elastic/Agent/elastic-agent.yml

outputs:
  default:
    type: elasticsearch
    hosts:
      - https://192.168.1.73:9200
    ssl.ca_trusted_fingerprint: c48c98cdf7f85d7cc29f834704011c1002b5251d594fc0bb08e6177544fe304a
    api_key: b1BkR1Q1TUIyOUpfMWhaS2NCUXA6SS1Jb3dncGVReVNpcEdzOGpSVmlzQQ==
    preset: balanced
```

## Testing

1. Go to Fleet > Settings > Outputs
2. Edit the default output and enter dummy data into the "Elasticsearch
CA trusted fingerprint" field
3. Go through the auto-detect onboarding flow
4. Inspect the `elastic-agent.yml` file written to disk. It should
contain the default output configured in Fleet including
`ca_trusted_fingerprint` setting

(cherry picked from commit 845aa8c)

# Conflicts:
#	x-pack/plugins/observability_solution/observability_onboarding/server/routes/flow/route.ts
paulinashakirova pushed a commit to paulinashakirova/kibana that referenced this pull request Nov 26, 2024
@thomheymann @paulinashakirova
[Observability] Use Fleet's default data output when onboarding integ…
3ad2404 
…rations using auto-detect flow (elastic#201158)

Resolves elastic#199751

## Summary

Use Fleet's default data output when onboarding integrations using
auto-detect flow.

## Screenshot

### Fleet output settings

<img width="1411" alt="Screenshot 2024-11-21 at 15 10 24"
src="https://github.com/user-attachments/assets/ac193552-7f18-4566-a84b-061df45c13f3">

### Generated Agent config

```
$ cat /Library/Elastic/Agent/elastic-agent.yml

outputs:
  default:
    type: elasticsearch
    hosts:
      - https://192.168.1.73:9200
    ssl.ca_trusted_fingerprint: c48c98cdf7f85d7cc29f834704011c1002b5251d594fc0bb08e6177544fe304a
    api_key: b1BkR1Q1TUIyOUpfMWhaS2NCUXA6SS1Jb3dncGVReVNpcEdzOGpSVmlzQQ==
    preset: balanced
```

## Testing

1. Go to Fleet > Settings > Outputs
2. Edit the default output and enter dummy data into the "Elasticsearch
CA trusted fingerprint" field
3. Go through the auto-detect onboarding flow
4. Inspect the `elastic-agent.yml` file written to disk. It should
contain the default output configured in Fleet including
`ca_trusted_fingerprint` setting
thomheymann added a commit that referenced this pull request Nov 27, 2024
@thomheymann
[8.x] [Observability] Use Fleet's default data output when onboarding…
c7ecbec 
… integrations using auto-detect flow (#201158) (#201833)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Observability] Use Fleet's default data output when onboarding
integrations using auto-detect flow
(#201158)](#201158)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Thom
Heymann","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-25T11:14:49Z","message":"[Observability]
Use Fleet's default data output when onboarding integrations using
auto-detect flow (#201158)\n\nResolves #199751\r\n\r\n##
Summary\r\n\r\nUse Fleet's default data output when onboarding
integrations using\r\nauto-detect flow.\r\n\r\n## Screenshot\r\n\r\n###
Fleet output settings\r\n\r\n<img width=\"1411\" alt=\"Screenshot
2024-11-21 at 15 10
24\"\r\nsrc=\"https://github.com/user-attachments/assets/ac193552-7f18-4566-a84b-061df45c13f3\">\r\n\r\n###
Generated Agent config\r\n\r\n```\r\n$ cat
/Library/Elastic/Agent/elastic-agent.yml\r\n\r\noutputs:\r\n
default:\r\n type: elasticsearch\r\n hosts:\r\n -
https://192.168.1.73:9200\r\n ssl.ca_trusted_fingerprint:
c48c98cdf7f85d7cc29f834704011c1002b5251d594fc0bb08e6177544fe304a\r\n
api_key:
b1BkR1Q1TUIyOUpfMWhaS2NCUXA6SS1Jb3dncGVReVNpcEdzOGpSVmlzQQ==\r\n preset:
balanced\r\n```\r\n\r\n## Testing\r\n\r\n1. Go to Fleet > Settings >
Outputs\r\n2. Edit the default output and enter dummy data into the
\"Elasticsearch\r\nCA trusted fingerprint\" field\r\n3. Go through the
auto-detect onboarding flow\r\n4. Inspect the `elastic-agent.yml` file
written to disk. It should\r\ncontain the default output configured in
Fleet including\r\n`ca_trusted_fingerprint`
setting","sha":"845aa8c47c858a6ccf83390bb38f332424ce2221","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport
missing","Team:Fleet","v9.0.0","backport:prev-minor","ci:project-deploy-observability"],"number":201158,"url":"https://github.com/elastic/kibana/pull/201158","mergeCommit":{"message":"[Observability]
Use Fleet's default data output when onboarding integrations using
auto-detect flow (#201158)\n\nResolves #199751\r\n\r\n##
Summary\r\n\r\nUse Fleet's default data output when onboarding
integrations using\r\nauto-detect flow.\r\n\r\n## Screenshot\r\n\r\n###
Fleet output settings\r\n\r\n<img width=\"1411\" alt=\"Screenshot
2024-11-21 at 15 10
24\"\r\nsrc=\"https://github.com/user-attachments/assets/ac193552-7f18-4566-a84b-061df45c13f3\">\r\n\r\n###
Generated Agent config\r\n\r\n```\r\n$ cat
/Library/Elastic/Agent/elastic-agent.yml\r\n\r\noutputs:\r\n
default:\r\n type: elasticsearch\r\n hosts:\r\n -
https://192.168.1.73:9200\r\n ssl.ca_trusted_fingerprint:
c48c98cdf7f85d7cc29f834704011c1002b5251d594fc0bb08e6177544fe304a\r\n
api_key:
b1BkR1Q1TUIyOUpfMWhaS2NCUXA6SS1Jb3dncGVReVNpcEdzOGpSVmlzQQ==\r\n preset:
balanced\r\n```\r\n\r\n## Testing\r\n\r\n1. Go to Fleet > Settings >
Outputs\r\n2. Edit the default output and enter dummy data into the
\"Elasticsearch\r\nCA trusted fingerprint\" field\r\n3. Go through the
auto-detect onboarding flow\r\n4. Inspect the `elastic-agent.yml` file
written to disk. It should\r\ncontain the default output configured in
Fleet including\r\n`ca_trusted_fingerprint`
setting","sha":"845aa8c47c858a6ccf83390bb38f332424ce2221"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201158","number":201158,"mergeCommit":{"message":"[Observability]
Use Fleet's default data output when onboarding integrations using
auto-detect flow (#201158)\n\nResolves #199751\r\n\r\n##
Summary\r\n\r\nUse Fleet's default data output when onboarding
integrations using\r\nauto-detect flow.\r\n\r\n## Screenshot\r\n\r\n###
Fleet output settings\r\n\r\n<img width=\"1411\" alt=\"Screenshot
2024-11-21 at 15 10
24\"\r\nsrc=\"https://github.com/user-attachments/assets/ac193552-7f18-4566-a84b-061df45c13f3\">\r\n\r\n###
Generated Agent config\r\n\r\n```\r\n$ cat
/Library/Elastic/Agent/elastic-agent.yml\r\n\r\noutputs:\r\n
default:\r\n type: elasticsearch\r\n hosts:\r\n -
https://192.168.1.73:9200\r\n ssl.ca_trusted_fingerprint:
c48c98cdf7f85d7cc29f834704011c1002b5251d594fc0bb08e6177544fe304a\r\n
api_key:
b1BkR1Q1TUIyOUpfMWhaS2NCUXA6SS1Jb3dncGVReVNpcEdzOGpSVmlzQQ==\r\n preset:
balanced\r\n```\r\n\r\n## Testing\r\n\r\n1. Go to Fleet > Settings >
Outputs\r\n2. Edit the default output and enter dummy data into the
\"Elasticsearch\r\nCA trusted fingerprint\" field\r\n3. Go through the
auto-detect onboarding flow\r\n4. Inspect the `elastic-agent.yml` file
written to disk. It should\r\ncontain the default output configured in
Fleet including\r\n`ca_trusted_fingerprint`
setting","sha":"845aa8c47c858a6ccf83390bb38f332424ce2221"}}]}]
BACKPORT-->
@kibanamachine kibanamachine added v8.18.0 and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Nov 27, 2024
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
@thomheymann @CAWilson94
[Observability] Use Fleet's default data output when onboarding integ…
370a99e 
…rations using auto-detect flow (elastic#201158)

Resolves elastic#199751

## Summary

Use Fleet's default data output when onboarding integrations using
auto-detect flow.

## Screenshot

### Fleet output settings

<img width="1411" alt="Screenshot 2024-11-21 at 15 10 24"
src="https://github.com/user-attachments/assets/ac193552-7f18-4566-a84b-061df45c13f3">

### Generated Agent config

```
$ cat /Library/Elastic/Agent/elastic-agent.yml

outputs:
  default:
    type: elasticsearch
    hosts:
      - https://192.168.1.73:9200
    ssl.ca_trusted_fingerprint: c48c98cdf7f85d7cc29f834704011c1002b5251d594fc0bb08e6177544fe304a
    api_key: b1BkR1Q1TUIyOUpfMWhaS2NCUXA6SS1Jb3dncGVReVNpcEdzOGpSVmlzQQ==
    preset: balanced
```

## Testing

1. Go to Fleet > Settings > Outputs
2. Edit the default output and enter dummy data into the "Elasticsearch
CA trusted fingerprint" field
3. Go through the auto-detect onboarding flow
4. Inspect the `elastic-agent.yml` file written to disk. It should
contain the default output configured in Fleet including
`ca_trusted_fingerprint` setting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nchaulet nchaulet nchaulet approved these changes

@mykolaharmash mykolaharmash mykolaharmash approved these changes

Assignees
No one assigned
Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.18.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Observability Onboarding] ES unsigned certificates don't work properly
5 participants
@thomheymann @elasticmachine @mykolaharmash @kibanamachine @nchaulet