[Fleet] Reusable integration policies #75867

ruflin · 2020-08-25T12:14:03Z

Currently each integration policy belongs to a specific Agent policy. It is not possible to reuse the same policy in multiple agent policies. Having support for this would be useful in a few cases:

Having a single system policy to collect data about the system and reuse it in Agent specific to Apache or MySQL servers
Have a single endpoint policy used for different Agent policies

As discussed in the design doc, we landed on a proposal to make all integration policies reusable with any number of agent policies.
Technically this means changing the 1:1 relationship of agent policy - integration policy to 1:N. Integration policies will store a list of agent policy ids to reference agent policies.
The full agent policy sent to the agents will contain the inputs from all integration policies that reference it.
The UI and API will be adjusted to support linking an integration policy to many agent policies.

Implementation plan

Give feedback

https://github.com/elastic/ingest-dev/issues/3262
https://github.com/elastic/ingest-dev/issues/3263
Support modifying agent policies when editing an integration #184394

QA:Needs Validation Team:Fleet
Show a Shared label in Agent policy page / Integrations tab #182219

QA:Needs Validation Team:Fleet
Show multiple agent policies on integration policies list UI #182111

QA:Needs Validation Team:Fleet
Add Manage agent policies action #182112

QA:Needs Validation Team:Fleet
Add warning when deleting integration policy to notify about being used by agent policies #182218

QA:Needs Validation Team:Fleet
When deleting an agent policy, only delete integration policies if no other agent policies use it #182222

QA:Needs Validation Team:Fleet
https://github.com/elastic/ingest-dev/issues/3300
[Fleet] Enable feature flag enableReusableIntegrationPolicies #186175

Team:Fleet
[Fleet]: User is navigated to Home page on editing integration added to multiple agent policies. #187336

QA:Validated Team:Fleet bug impact:medium
https://github.com/elastic/ingest-dev/issues/3464
Options

As a follow-up, we should support "orphaned" integration policies that don't belong to an agent policy.

Design solution

Reach out to @simosilvestri for implementation support.

Figma file / Prototype/UX issue

New UX Criteria

Add/Edit integration wizard

Ability to add multiple agent policies -> Prototype link

Ability to edit selected agent policies -> Prototype link
Confirmation dialog to show possible dependencies

Integration policies table > Agent policy tab -> Prototype link

Display the policy badge inline on the table based on the column width.
The number of policies trimmed in the column opens a popover with a full list of clickable agent policies.
Customers can manage agent policies that share the same integration.
Make the agent policy badges clickable, linking to the agent policy details page in Fleet.

Fleet > Agent policy > Integration tab

The integration name shows a shared label with an info tooltip to communicate to the user the number of policies shared with that integration.

Workflow

Select a reusable policy

When a user adds an integration to an Agent policy, if there are existing reusable policies for this integration, the user gets shown an option to select one. On selection a summary could be shown but the user cannot edit it.

Create reusable policy

As a reusable policy can only be created outside an Agent Policy, a new place and workflow is needed to create an manage these.

Editing reusable policy

A policy that is used in multiple agent policies should have a special indicator in the policy list. If the user wants to edit the policy, not the standard edit screen is shown but one that makes it obvious this policy is used in multiple places. It should be visible which other agent policies use the same policy. On save it must be shown how many other Agent policies this change effects.

Technical details

On the technical side, I expect that the package policy is referenced in the agent policy by id. This id can be used in multiple places. The policy itself is not tied to a config and can also exist without an assigned agent policy. If a user removes a reusable policy from an Agent Policy, the reusable policy will continue to exist. A reusable policy can only be deleted if 0 agent policies use it.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-08-25T12:14:13Z

Pinging @elastic/ingest-management (Team:Ingest Management)

ph · 2020-08-25T16:54:59Z

@ruflin the Reusable Integration Policy (RIP nice acronym) is linked in the agent policy? I think it makes sense when you edit and save the reusable integration policy that you can either modify it which will apply the changes to all the Agent Policy or
copy and only have the configuration applied to the current Agent policy.

This is similar to the behavior you have when you work with Symbols in Sketch.

ph · 2020-08-25T16:56:50Z

The above effectively allow you to change you mind have and have the copy behavior from #75868

ruflin · 2020-08-26T07:33:14Z

@ph Yes, the policy is referenced in the agent policy. For the option to also copy, don't see any problem with that but would say that this is an additional feature.

mostlyjason · 2020-09-04T14:31:56Z

We discussed something similar to this a few months ago called shared data sources. There is an old google doc discussing some details.

SHolzhauer · 2020-09-07T08:58:56Z

Would you be able to create a reusable policy with another reusable policy? Effectively creating the structure I am looking for in #76640 ?

ruflin · 2020-09-08T07:17:09Z

@SHolzhauer No, it would be more like the following:

- apache
  - linux system
  - webserver
  - apache

- tomcat
  - linux system
  - webserver
  - tomcat
  
- proxies
  - linux system
  - proxy

- linux
  - linux system

Each sub block is a reusable entry.

SHolzhauer · 2020-09-08T07:48:58Z

Trying to wrap my head around this, but I believe this would still solve the use-case. Going to close the other issue in favor of this one.

runejuhl · 2023-01-30T11:10:16Z

Sorry for the noise, but it's been 2.5 years since the last comment -- are there any plans to implement this feature?

Fabien4941 · 2023-03-01T15:52:53Z

We have exactly the same need.

Example of our policies:

Server type A:

System
Docker
Custom Logs

Server type B:

System
Docker

Server type C :

System

"System" integrations are strictly identical in each policy. When a modification is made to one of the policies, it must be manually transferred to all the others, it is not convenient. Especially when the number of policies increases.

renzedj · 2023-03-01T16:15:55Z

We have exactly the same need.
...
"System" integrations are strictly identical in each policy. When a modification is made to one of the policies, it must be manually transferred to all the others, it is not convenient. Especially when the number of policies increases.

Same here. We just identified a need to add something to our system policy across our entire environment. I'm going to have to apply this manually to approximately 60 policies, with all the potential for error and time commitment that implies.

SHolzhauer · 2023-11-28T16:48:35Z

This is something that is still blocking our migration to Agent!

dcarltonafs · 2023-11-30T15:45:09Z

Also following this thread for our implementation of fleet. I might be getting twisted around here but it seems each thread in 75867, 75868, and 76640 closes in a cyclical favor of one of the other two threads. Any idea if either integration policy copying or a single integration policy being applied to multiple agent-policies (preferred) is on the horizon? If not, could an explanation be put out on how to use API calls to create a new policy with the same configuration? Rather than using integration policy names the API seems to automatically assign uuid's so I don't see how we can use API calls to copy configurations right now.

nicpenning · 2024-04-16T16:55:39Z

Any progress being made here? Over time more and more policies with slight changes are occurring and starting to sprawl out of control.

zez3 · 2024-04-17T10:32:14Z

@nchaulet can you or @kpollich please update us on this ER?

kpollich · 2024-04-17T13:51:00Z

(Edit: I'm updating my comment to be a little less committal, as priorities can shift. See the edit history for my previous comment.)

This work is prioritized and the product definition is nearly complete. We'll be working on finalizing the technical implementation details in the coming weeks. We'll reuse this issue for the technical implementation details so look for an updated description that lays out the implementation in the near future.

juliaElastic · 2024-04-30T13:15:18Z

@kpollich Added the implementation plan to this issue, the last few tasks are WIP, asked a few clarifying questions on the doc from Nima.

kpollich · 2024-05-07T16:50:28Z

@juliaElastic - After talking with @nimarezainia, I moved the "orphaned" integration policy work out of the main implementation plan. We can tackle that work in a follow-up as it's a less common use case.

juliaElastic · 2024-05-08T08:22:57Z

@kpollich I think #182222 is still needed for the first phase, because by default all integration policies would be deleted when an agent policy is deleted.

cmacknz · 2024-06-04T16:17:03Z

We need to stress test this feature to determine how it impacts the maximum number of agent policies we support.

criamico · 2024-07-02T09:01:05Z

Found this bug while testing the new feature: #187331

## Summary Relates #75867 Change to `Agent policy` to `Agent policies` in a few places. <img width="1154" alt="image" src="https://github.com/elastic/kibana/assets/90178898/52ea19ea-6302-44b8-88b2-54abd6a4ec37"> <img width="1405" alt="image" src="https://github.com/elastic/kibana/assets/90178898/503844c5-d9ec-4bdf-a2d3-aeb4e9100764"> <img width="1409" alt="image" src="https://github.com/elastic/kibana/assets/90178898/9149bf6f-c9fa-4535-ab7b-c4ff4df35dca"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

nimarezainia · 2024-07-19T04:35:17Z

Available in 8.15 behind a feature flag. @kpollich can i close this? PRs for telemetry are available.

kpollich · 2024-07-19T12:11:57Z

Yes let's close this. The feature flag is enabled by default in 8.16 as of #186175.

Part of #75867 Depends on merging first elastic/telemetry#3759 ## Summary Define new telemetry hourly job for reusable policies; Example of data: ``` { total_integration_policies: 3, shared_integration_policies: 2, shared_integrations: { agents: 10, name: 'aws-1', pkg_name: 'aws', pkg_version: '1.0.0', shared_by_agent_policies: 3, } ``` ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Elastic Machine <[email protected]>

nicpenning · 2024-07-20T01:40:19Z

Wow, 4 years in the making. This is huge! 🥳

amolnater-qasource · 2024-09-09T09:07:19Z

Hi Team,

We have created 15 testcases under Testmo for this feature under Fleet test suite under below Section:

Reusable Integrations

Please let us know if any other scenario needs to be added from our end.

Thanks!

amolnater-qasource · 2024-10-17T10:08:53Z

Hi Team,
As per changes under #186357 we have created 03 more testcases for this feature under Fleet Test Suite at links:

Validate count of the agents displayed separately for each policy using the shared integration under Integration policies tab.
Validate Add agent button shows all policies used by the shared integration under Integration policies tab.
Validate upgrade integration button under Integration policies tab navigates to Upgrade integration page with all policies selected for shared integration.

Please let us know if anything else is required from our end.
Thanks!!

amolnater-qasource · 2024-10-25T08:16:02Z

Hi Team,

We have executed 18 testcases under the Feature test run for the 8.16.0 release at the link:

Reusable Integrations

Status:

PASS: 18

Build details:
VERSION: 8.16.0 BC2
BUILD: 79434
COMMIT: 59220e9

As the testing is completed on this feature, we are marking this as QA:Validated.

Please let us know if anything else is required from our end.
Thanks

ruflin added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 25, 2020

jen-huang changed the title ~~[Ingest Manager] Reusable policies~~ [Ingest Manager] Reusable integration policies Aug 25, 2020

ruflin mentioned this issue Sep 4, 2020

[Fleet] Allow combining agent configurations #76640

Closed

ruflin mentioned this issue Feb 25, 2021

[Question] Will policies stay 1:1 or will they be mergeable? elastic/integrations#743

Closed

jen-huang changed the title ~~[Ingest Manager] Reusable integration policies~~ [Fleet] Reusable integration policies Apr 27, 2021

jen-huang mentioned this issue Apr 13, 2022

[Fleet] Copy integration policy #75868

Open

juliaElastic mentioned this issue Jun 26, 2024

[Fleet] Enable feature flag enableReusableIntegrationPolicies #186175

Closed

juliaElastic mentioned this issue Jul 2, 2024

[Fleet] handle multiple policies in edit package policy extension view #187334

Merged

1 task

amolnater-qasource mentioned this issue Jul 2, 2024

[Fleet]: User is navigated to Home page on editing integration added to multiple agent policies. #187336

Closed

This was referenced Jul 15, 2024

[Fleet] Add telemetry for reusable policies #188319

Closed

[Fleet] Define new telemetry hourly job for reusable policies #188536

Merged

kpollich added the v8.15.0 label Jul 19, 2024

kpollich closed this as completed Jul 19, 2024

amolnater-qasource mentioned this issue Jul 24, 2024

[Fleet]: Policies added to the integration aren't visible under Manage agent policies callout message. #189035

Closed

muskangulati-qasource mentioned this issue Aug 29, 2024

[Fleet]: The agent count is not consistent while adding integration to multiple policies #191702

Closed

jlind23 mentioned this issue Sep 3, 2024

Reusable integration policies elastic/elastic-agent#2227

Closed

amolnater-qasource mentioned this issue Sep 9, 2024

Bad certificates error on running logstash elastic/logstash#16435

Closed

arvindersingh-qasource mentioned this issue Oct 3, 2024

[Fleet]: Count of Agent Policies on top right corner does not update until user refresh browser window #194824

Open

amolnater-qasource mentioned this issue Oct 4, 2024

[Fleet]: Shared integration is no longer visible under other spaces, if any of the policy loses access to another space. #194900

Closed

amolnater-qasource mentioned this issue Oct 16, 2024

[Fleet]: Add agent flyout is not opened on navigating to Integration policies tab on clicking "Add elastic agent to your hosts" while adding integration to the shared policies. #196487

Open

sukhwindersingh-qasource mentioned this issue Oct 16, 2024

[Fleet] Able to create agent policies from the add agent flyout for shared integration, which temporarily displays incorrect integrations for the selected policy #196504

Open

amolnater-qasource mentioned this issue Oct 25, 2024

[Fleet]: Remote ES not supported error is displayed on Editing shared Elastic Defend when one policy has Remote ES selected as Default output for integrations. #197752

Open

amolnater-qasource added the QA:Validated Issue has been validated by QA label Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Reusable integration policies #75867

[Fleet] Reusable integration policies #75867

ruflin commented Aug 25, 2020 •

edited by nimarezainia

Loading

Implementation plan

elasticmachine commented Aug 25, 2020

ph commented Aug 25, 2020 •

edited

Loading

ph commented Aug 25, 2020

ruflin commented Aug 26, 2020

mostlyjason commented Sep 4, 2020

SHolzhauer commented Sep 7, 2020

ruflin commented Sep 8, 2020

SHolzhauer commented Sep 8, 2020

runejuhl commented Jan 30, 2023

Fabien4941 commented Mar 1, 2023 •

edited

Loading

renzedj commented Mar 1, 2023

SHolzhauer commented Nov 28, 2023

dcarltonafs commented Nov 30, 2023 •

edited

Loading

nicpenning commented Apr 16, 2024

zez3 commented Apr 17, 2024 •

edited

Loading

kpollich commented Apr 17, 2024 •

edited

Loading

juliaElastic commented Apr 30, 2024

kpollich commented May 7, 2024

juliaElastic commented May 8, 2024

cmacknz commented Jun 4, 2024

criamico commented Jul 2, 2024

nimarezainia commented Jul 19, 2024

kpollich commented Jul 19, 2024 •

edited

Loading

nicpenning commented Jul 20, 2024

amolnater-qasource commented Sep 9, 2024

amolnater-qasource commented Oct 17, 2024

amolnater-qasource commented Oct 25, 2024

[Fleet] Reusable integration policies #75867

[Fleet] Reusable integration policies #75867

Comments

ruflin commented Aug 25, 2020 • edited by nimarezainia Loading

Implementation plan

Design solution

New UX Criteria

Workflow

Select a reusable policy

Create reusable policy

Editing reusable policy

Technical details

elasticmachine commented Aug 25, 2020

ph commented Aug 25, 2020 • edited Loading

ph commented Aug 25, 2020

ruflin commented Aug 26, 2020

mostlyjason commented Sep 4, 2020

SHolzhauer commented Sep 7, 2020

ruflin commented Sep 8, 2020

SHolzhauer commented Sep 8, 2020

runejuhl commented Jan 30, 2023

Fabien4941 commented Mar 1, 2023 • edited Loading

renzedj commented Mar 1, 2023

SHolzhauer commented Nov 28, 2023

dcarltonafs commented Nov 30, 2023 • edited Loading

nicpenning commented Apr 16, 2024

zez3 commented Apr 17, 2024 • edited Loading

kpollich commented Apr 17, 2024 • edited Loading

juliaElastic commented Apr 30, 2024

kpollich commented May 7, 2024

juliaElastic commented May 8, 2024

cmacknz commented Jun 4, 2024

criamico commented Jul 2, 2024

nimarezainia commented Jul 19, 2024

kpollich commented Jul 19, 2024 • edited Loading

nicpenning commented Jul 20, 2024

amolnater-qasource commented Sep 9, 2024

amolnater-qasource commented Oct 17, 2024

amolnater-qasource commented Oct 25, 2024

ruflin commented Aug 25, 2020 •

edited by nimarezainia

Loading

ph commented Aug 25, 2020 •

edited

Loading

Fabien4941 commented Mar 1, 2023 •

edited

Loading

dcarltonafs commented Nov 30, 2023 •

edited

Loading

zez3 commented Apr 17, 2024 •

edited

Loading

kpollich commented Apr 17, 2024 •

edited

Loading

kpollich commented Jul 19, 2024 •

edited

Loading