[UII] Support integration-level outputs #189125

jen-huang · 2024-07-24T18:55:30Z

Summary

Resolves #143905. This PR adds support for integration-level outputs. This means that different integrations within the same agent policy can now be configured to send data to different locations. This feature is gated behind enterprise level subscription.

For each input, the agent policy will configure sending data to the following outputs in decreasing order of priority:

Output set specifically on the integration policy
Output set specifically on the integration's parent agent policy (including the case where an integration policy belongs to multiple agent policies)
Global default data output set via Fleet Settings

Integration-level outputs will respect the same rules as agent policy-level outputs:

Certain integrations are disallowed from using certain output types, attempting to add them to each other via creation, updating, or "defaulting", will fail
- fleet-server, synthetics, and apm can only use same-cluster Elasticsearch output
When an output is deleted, any integrations that were specifically using it will "clear" their output configuration and revert back to either #2 or #3 in the above list
When an output is edited, all agent policies across all spaces that use it will be bumped to a new revision, this includes:
- Agent policies that have that output specifically set in their settings (existing behavior)
- Agent policies that contain integrations which specifically has that output set (new behavior)
When a proxy is edited, the same new revision bump above will apply for any outputs using that proxy

The final agent policy YAML that is generated will have:

outputs block that includes:
- Data and monitoring outputs set at the agent policy level (existing behavior)
- Any additional outputs set at the integration level, if they differ from the above
outputs_permissions block that includes permissions for each Elasticsearch output depending on which integrations and/or agent monitoring are assigned to it

Integration policies table now includes Output column. If the output is defaulting to agent policy-level output, or global setting output, a tooltip is shown:

Configuring an integration-level output is done under Advanced options in the policy editor. Setting to the blank value will "clear" the output configuration. The list of available outputs is filtered by what outputs are available for that integration (see above):

An example of failure: ES output cannot be changed to Kafka while there is an integration

TODO

Adjust side effects of editing/deleting output when policies use it across different spaces
Add API integration tests
Update OpenAPI spec
Create doc issue

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

… behind enterprise license check

…he package (i.e. APM can only have ES output)

Include package policies in search for agent policies to be bumped when output is updated

…egration

…on link

…egration

obltmachine · 2024-07-24T18:55:47Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

…egration

nimarezainia · 2024-07-31T04:16:37Z

@kilfoyle for your radar

…egration

…references

…bana into feat/output-per-integration

x-pack/plugins/fleet/common/constants/output.ts

nchaulet · 2024-08-12T14:14:42Z

...ions/agent_policy/create_package_policy_page/components/steps/step_define_package_policy.tsx

+                          />
+                        }
+                      >
+                        <EuiSelect


Should we use a super select like we do for the agent policy output selector? the default value looks more explicit to me
Currently

Agent policy output selector

I can address this in a follow up PR

nchaulet · 2024-08-12T14:21:03Z

...gins/fleet/public/applications/fleet/sections/settings/services/agent_and_policies_count.tsx

    page: 1,
    perPage: SO_SEARCH_LIMIT,
  });

  if (agentPolicies.error) {
    throw agentPolicies.error;
  }
-  const agentPolicyCount = agentPolicies.data?.items?.length ?? 0;
+
+  const packagePolicies = await sendGetPackagePolicies({


Not necessarily related to that PR but I have more and more concerns about retrieving all package policies client side, this will be a blocker to support a larger number of policies, and I am not sure of the value of showing an exact count of policies for the user.
I am wondering if this could be simplified to two scenarios, the output is not used by any agent policy or the output is used by at least one agent policy.

I agree with this, and I actually saw that on some of our other settings confirmation modals, we don't show the exact count like this. we just tell the user that this will agents enrolled into affected policies. I'll capture this in a follow up enhancement to remove these granular counts in modals

Could some one explain what this block is referring to? is it referring to the counts in the Fleet table?

@nimarezainia It is the count in the confirmation modal when user updates/deletes an output, like "this will update X policies used by X agents"

@nimarezainia @nchaulet I opened #190432

nchaulet · 2024-08-12T14:24:53Z

x-pack/plugins/fleet/server/routes/package_policy/handlers.ts

+      throw new PackagePolicyRequestError(canUseMultipleAgentPoliciesErrorMessage);
+    }
+
+    if (newPolicy.output_id && pkg) {


Should that validation happen at the package policy service level so it will also be validated for preconfigured package policies and in the future if other solution create package policy with output_id

Should we validate also for the update package policy method? (Maybe it's already done and I missed that)

yes, this should happen at the service level. I was copying what was done for multiple policy IDs but I don't think that was a good pattern. I pushed changes to move both multiple policy and integration output validation to the service level. they are executed together in a preflight function that is added to create, update, bulk create, and bulk update methods

thanks for catching this pattern!

…icy service level

…egration

kibana-ci · 2024-08-13T03:53:54Z

💚 Build Succeeded

Buildkite Build
Commit: 617c3d9

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1229	1234	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.8MB	1.8MB	+2.4KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	168.8KB	169.0KB	+198.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1351	1357	+6

History

💔 Build #227227 failed 0802091
💔 Build #226410 failed bd713b7
💛 Build #226366 was flaky 9019534
💔 Build #226173 failed 7ca5a4f
💔 Build #226159 failed eef7592

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jen-huang

nchaulet

Tested locally and work as expected 🚀

kilfoyle

LGTM for the docs! 🚢

## Summary Resolves elastic#143905. This PR adds support for integration-level outputs. This means that different integrations within the same agent policy can now be configured to send data to different locations. This feature is gated behind `enterprise` level subscription. For each input, the agent policy will configure sending data to the following outputs in decreasing order of priority: 1. Output set specifically on the integration policy 2. Output set specifically on the integration's parent agent policy (including the case where an integration policy belongs to multiple agent policies) 3. Global default data output set via Fleet Settings Integration-level outputs will respect the same rules as agent policy-level outputs: - Certain integrations are disallowed from using certain output types, attempting to add them to each other via creation, updating, or "defaulting", will fail - `fleet-server`, `synthetics`, and `apm` can only use same-cluster Elasticsearch output - When an output is deleted, any integrations that were specifically using it will "clear" their output configuration and revert back to either `elastic#2` or `elastic#3` in the above list - When an output is edited, all agent policies across all spaces that use it will be bumped to a new revision, this includes: - Agent policies that have that output specifically set in their settings (existing behavior) - Agent policies that contain integrations which specifically has that output set (new behavior) - When a proxy is edited, the same new revision bump above will apply for any outputs using that proxy The final agent policy YAML that is generated will have: - `outputs` block that includes: - Data and monitoring outputs set at the agent policy level (existing behavior) - Any additional outputs set at the integration level, if they differ from the above - `outputs_permissions` block that includes permissions for each Elasticsearch output depending on which integrations and/or agent monitoring are assigned to it Integration policies table now includes `Output` column. If the output is defaulting to agent policy-level output, or global setting output, a tooltip is shown: <img width="1392" alt="image" src="https://github.com/user-attachments/assets/5534716b-49b5-402a-aa4a-4ba6533e0ca8"> Configuring an integration-level output is done under Advanced options in the policy editor. Setting to the blank value will "clear" the output configuration. The list of available outputs is filtered by what outputs are available for that integration (see above): <img width="799" alt="image" src="https://github.com/user-attachments/assets/617af6f4-e8f8-40b1-b476-848f8ac96e76"> An example of failure: ES output cannot be changed to Kafka while there is an integration <img width="1289" alt="image" src="https://github.com/user-attachments/assets/11847eb5-fd5d-4271-8464-983d7ab39218"> ## TODO - [x] Adjust side effects of editing/deleting output when policies use it across different spaces - [x] Add API integration tests - [x] Update OpenAPI spec - [x] Create doc issue ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>

Closes #192339 ## Summary Display two additional columns with Outputs hosts in agent list table and agent details section - The two columns show monitoring output and the integrations output and link to the output flyout in settings - Display a badge that show the outputs set per integration introduced by #189125 - Same info displayed in agent details as well To achieve this, I added two new endpoints. 1. Endpoint that fetches all the outputs associated with a single agent policy (outputs defined on agent policy or default defined in global settings and if any, outputs per integration) ``` GET kbn:/api/fleet/agent_policies/<AGENT_POLICY_ID>/outputs ``` 2. Endpoint that fetches the outputs as above, for a defined set of agent policy ids ``` POST kbn:/api/fleet/agent_policies/outputs { "ids": ["policy_id1", "policy_id2", ...] } ``` The reason to pass an array of ids is to ensure that we fetch the info only for the policies displayed in the table at any given moment. ### Screenshots **Agent list** ![Screenshot 2024-10-16 at 17 51 57](https://github.com/user-attachments/assets/3ee08df1-9562-497f-9621-4a913b3dad74) ![Screenshot 2024-10-16 at 17 52 05](https://github.com/user-attachments/assets/72b9da7d-872a-45f8-b02d-29184ffb2179) **Agent details** ![Screenshot 2024-10-16 at 17 52 20](https://github.com/user-attachments/assets/b99aaf9e-14f1-44b8-9776-3e0136775af8) ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [ ] This will appear in the **Release Notes** and follow the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: kibanamachine <[email protected]>

jen-huang added 17 commits July 22, 2024 12:19

Update schemas to include output_id for package policies

f4c1aa7

Adjust create/edit package policy APIs to allow output_id and gate it…

04821c3

… behind enterprise license check

Adjust API gating to include check for allowed output type based on t…

d11b8d8

…he package (i.e. APM can only have ES output)

Compile output permissions correctly

06aba09

Fix update handler, add tests

3cf6b71

Remove output from package policies when it is deleted

4358b9b

Include package policies in search for agent policies to be bumped when output is updated

Merge remote-tracking branch 'upstream/main' into feat/output-per-int…

7739f10

…egration

Fix output preconfiguration test mock

a5fbc22

Fix truncation and wrapping issues with agent policy name + descripti…

bb776fa

…on link

Add Output column to integration policies table

9ce8a8c

Make output_id nullable in api schemas

ccc4702

Move license const

93bf7e5

Add Output select field to create/edit package policy form

d071843

Merge remote-tracking branch 'upstream/main' into feat/output-per-int…

9e2b219

…egration

Fix typos, clean up unused agent count code

5d15a12

Fix agent policy and agent count in edit output confirm modal

cca9d2f

Fix output id not being set on creation w/ non-simplified request

e964831

Fix query for retrieving an output's related agent policies

1a43c5c

This comment was marked as outdated.

Sign in to view

jen-huang added 2 commits July 26, 2024 12:35

Fix test failures

8c1f570

Merge remote-tracking branch 'upstream/main' into feat/output-per-int…

fc4ff97

…egration

This comment was marked as outdated.

Sign in to view

Merge remote-tracking branch 'upstream/main' into feat/output-per-int…

b760c92

…egration

Merge remote-tracking branch 'upstream/main' into feat/output-per-int…

2be5ddd

…egration

kilfoyle mentioned this pull request Aug 2, 2024

Docs for integration-level outputs elastic/ingest-docs#1217

Closed

jen-huang added 3 commits August 5, 2024 13:52

Merge remote-tracking branch 'upstream/main' into feat/output-per-int…

f97e138

…egration

Fix mocks

5be583a

Make outputs service search across all spaces when retrieving policy …

700e0b9

…references

jen-huang added release_note:enhancement v8.16.0 labels Aug 7, 2024

jen-huang added 2 commits August 7, 2024 14:46

Fix default output ID

f59c5ca

Merge branch 'feat/output-per-integration' of github.com:jen-huang/ki…

bd713b7

…bana into feat/output-per-integration

jloleysens approved these changes Aug 8, 2024

View reviewed changes

elena-shostak approved these changes Aug 8, 2024

View reviewed changes

nchaulet self-requested a review August 8, 2024 19:45

nchaulet reviewed Aug 12, 2024

View reviewed changes

x-pack/plugins/fleet/common/constants/output.ts Show resolved Hide resolved

nchaulet reviewed Aug 12, 2024

View reviewed changes

jen-huang added 5 commits August 12, 2024 15:08

Remove unnecessary cleanup

476971f

Move validation of multiple agent policies and outputs to package pol…

e2e7a99

…icy service level

Move tests

560b078

Merge remote-tracking branch 'upstream/main' into feat/output-per-int…

0b0dfa6

…egration

Fix bad merge

0802091

jen-huang requested a review from nchaulet August 12, 2024 23:58

Fix types

617c3d9

nchaulet approved these changes Aug 13, 2024

View reviewed changes

kilfoyle approved these changes Aug 13, 2024

View reviewed changes

jen-huang mentioned this pull request Aug 13, 2024

[UII] Consider removing or refactoring agent count in confirmation modals #190432

Open

jen-huang merged commit 8266415 into elastic:main Aug 13, 2024
21 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Aug 13, 2024

jen-huang deleted the feat/output-per-integration branch August 13, 2024 15:50

blakerouse mentioned this pull request Aug 28, 2024

Elastic Agent - Exceptions in the ‘Output for Integrations’ within a policy elastic/elastic-agent#5374

Closed

jen-huang mentioned this pull request Oct 11, 2024

[Fleet] Display outputs in agent list table and agent details #195801

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UII] Support integration-level outputs #189125

[UII] Support integration-level outputs #189125

jen-huang commented Jul 24, 2024 •

edited

Loading

obltmachine commented Jul 24, 2024

This comment was marked as outdated.

This comment was marked as outdated.

nimarezainia commented Jul 31, 2024

nchaulet Aug 12, 2024

jen-huang Aug 12, 2024

nchaulet Aug 12, 2024

jen-huang Aug 12, 2024

nimarezainia Aug 13, 2024

jen-huang Aug 13, 2024

jen-huang Aug 13, 2024

nchaulet Aug 12, 2024

jen-huang Aug 12, 2024 •

edited

Loading

kibana-ci commented Aug 13, 2024

API count

nchaulet left a comment

kilfoyle left a comment

[UII] Support integration-level outputs #189125

[UII] Support integration-level outputs #189125

Conversation

jen-huang commented Jul 24, 2024 • edited Loading

Summary

TODO

Checklist

obltmachine commented Jul 24, 2024

🤖 GitHub comments

This comment was marked as outdated.

This comment was marked as outdated.

nimarezainia commented Jul 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jen-huang Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

kibana-ci commented Aug 13, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

History

nchaulet left a comment

Choose a reason for hiding this comment

kilfoyle left a comment

Choose a reason for hiding this comment

jen-huang commented Jul 24, 2024 •

edited

Loading

jen-huang Aug 12, 2024 •

edited

Loading