[Fleet] delete unenrolled agents task #195544

juliaElastic · 2024-10-09T08:35:29Z

Summary

Testing steps:

enable deleting unenrolled agents by adding xpack.fleet.enableDeleteUnenrolledAgents: true to kibana.dev.yml or turn it on on the UI
add some unenroll agents with the helper script

cd x-pack/plugins/fleet
node scripts/create_agents/index.js --status unenrolled --count 10

 info Creating 10 agents with statuses:
 info    unenrolled: 10
 info Batch complete, created 10 agent docs, took 0, errors: false
 info All batches complete. Created 10 agents in total. Goodbye!

restart kibana or wait for the task to run and verify that the unenrolled agents were deleted

[2024-10-08T16:14:45.152+02:00][DEBUG][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] Executed deletion of 10 unenrolled agents
[2024-10-08T16:14:45.153+02:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] runTask ended: success

Added to UI settings:

If the flag is preconfigured, disabled update on the UI with a tooltip:

The update is also prevented from the API:

Once the preconfiguration is removed, the UI update is allowed again.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

…te --fix'

juliaElastic · 2024-10-09T09:49:28Z

x-pack/plugins/fleet/server/tasks/delete_unenrolled_agents_task.ts

+export const VERSION = '1.0.0';
+const TITLE = 'Fleet Delete Unenrolled Agents Task';
+const SCOPE = ['fleet'];
+const INTERVAL = '1h';


Added 1h interval, it should be efficient to delete all unenrolled agents with deleteByQuery, no need to load them in memory.

juliaElastic · 2024-10-09T09:51:35Z

x-pack/plugins/fleet/server/services/preconfiguration/delete_unenrolled_agent_setting.ts

There is a caveat with enabling both preconfig and UI setting, it can happen that the preconfigured value gets out of sync when changed from the UI.
We could disable changing from UI when preconfigured if this is an issue.

… src/core/server/integration_tests/ci_checks'

elasticmachine · 2024-10-09T10:33:07Z

Pinging @elastic/fleet (Team:Fleet)

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/fleet --update'

…taging'

nchaulet · 2024-10-09T14:30:40Z

x-pack/plugins/fleet/server/services/setup.ts

@@ -195,6 +199,12 @@ async function createSetupSideEffects(
  logger.debug('Setting up Space settings');
  await ensureSpaceSettings(appContextService.getConfig()?.spaceSettings ?? []);

+  logger.debug('Setting up delete unenrolled agents setting');


I tried to enable it then disable it via the kibana config and it stay enabled, it is expected? in my opinion if that settings is managed via the kibana config it should probably be the only source of truth

xpack.fleet.enableDeleteUnenrolledAgents: false

Good catch, it was a bug, it is fixed now.

I agree that it's clearer to have one single source of truth. Are we ok to keep this setting in preconfiguration only, and remove from the UI @kpollich @nimarezainia ?

I think, similar to most of the configuration we should allow to configure it both from UI or kibana config, but if it's configured in preconfiguration this should become the only way to configure it (until it's removed from the config)

In that case I have to introduce an is_preconfigured flag to know if the delete flag was preconfigured.

ymao1 · 2024-10-09T14:30:42Z

x-pack/plugins/fleet/server/tasks/delete_unenrolled_agents_task.ts

+    this.logger.debug(`[DeleteUnenrolledAgentsTask] Fetching unenrolled agents`);
+
+    const response = await esClient.deleteByQuery({
+      index: AGENTS_INDEX,


should the abortcontroller be passed into this request so the request is cancelled if aborted?

thanks for raising, fixed

azasypkin · 2024-10-09T14:57:56Z

x-pack/plugins/fleet/server/saved_objects/index.ts

@@ -181,6 +182,16 @@ export const getSavedObjectTypes = (
            },
          ],
        },
+        3: {


question: @juliaElastic this SO type (GLOBAL_SETTINGS_SAVED_OBJECT_TYPE) isn't registered as encrypted saved object, right?

Right, it's not an encrypted SO.

azasypkin

LGTM, since the added migration is for the ingest_manager_settings SO type, which isn’t an encrypted saved object.

nchaulet

Tested locally and it worked as expected, LGTM 🚀

jloleysens

API and mappings changes LGTM

ymao1

Response Ops changes LGTM. Reviewed for new recurring task type that runs every hour with a timeout of 1 minute and respects cancellation signals.

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/fleet --update'

…taging'

elasticmachine · 2024-10-14T09:03:15Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 3d5c4a0

Failed CI Steps

FTR Configs #51

Test Failures

[job] [logs] FTR Configs #51 / discover/esql discover esql view switch modal should not show switch modal when switching to a data view while a saved search is open

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	1210	1211	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.7MB	1.7MB	+2.4KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	171.6KB	171.6KB	+38.0B

History

💔 Build #241427 failed 5e5bb64
💔 Build #241176 failed b7b7eec
💔 Build #241095 failed 8c25642
💔 Build #241018 failed 147c2d4
💛 Build #240983 was flaky 1af548d4d574a6ca4742e1a129baa91f921cf0df

cc @juliaElastic

kibanamachine · 2024-10-14T09:04:21Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11324350540

kibanamachine · 2024-10-14T09:09:15Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 195544

Questions ?

Please refer to the Backport tool documentation

Closes elastic#189506 Testing steps: - enable deleting unenrolled agents by adding `xpack.fleet.enableDeleteUnenrolledAgents: true` to `kibana.dev.yml` or turn it on on the UI - add some unenroll agents with the helper script ``` cd x-pack/plugins/fleet node scripts/create_agents/index.js --status unenrolled --count 10 info Creating 10 agents with statuses: info unenrolled: 10 info Batch complete, created 10 agent docs, took 0, errors: false info All batches complete. Created 10 agents in total. Goodbye! ``` - restart kibana or wait for the task to run and verify that the unenrolled agents were deleted ``` [2024-10-08T16:14:45.152+02:00][DEBUG][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] Executed deletion of 10 unenrolled agents [2024-10-08T16:14:45.153+02:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] runTask ended: success ``` Added to UI settings: <img width="1057" alt="image" src="https://github.com/user-attachments/assets/2c9279f9-86a8-4630-a6cd-5aaa42e05fe7"> If the flag is preconfigured, disabled update on the UI with a tooltip: <img width="1009" alt="image" src="https://github.com/user-attachments/assets/45041020-6447-4295-995e-6848f0238f88"> The update is also prevented from the API: <img width="2522" alt="image" src="https://github.com/user-attachments/assets/cfbc8e21-e062-4e7f-9d08-9767fa387752"> Once the preconfiguration is removed, the UI update is allowed again. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>

Backport #195544 to 8.x --------- Co-authored-by: kibanamachine <[email protected]>

delete unenrolled agents task

504da49

juliaElastic added release_note:feature Makes this part of the condensed release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Oct 9, 2024

juliaElastic self-assigned this Oct 9, 2024

juliaElastic and others added 3 commits October 9, 2024 10:35

Merge branch 'main' into cleanup-unenrolled

d8d81bd

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

4191916

…te --fix'

added delete flag to settings UI

0061716

juliaElastic commented Oct 9, 2024

View reviewed changes

Merge branch 'main' into cleanup-unenrolled

ed96ae5

juliaElastic marked this pull request as ready for review October 9, 2024 09:52

juliaElastic requested review from a team as code owners October 9, 2024 09:52

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

11148b0

… src/core/server/integration_tests/ci_checks'

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Oct 9, 2024

kibanamachine and others added 3 commits October 9, 2024 11:13

add test on task

76970c6

more tests, make prerelease_integrations_enabled optional

22d2d85

juliaElastic mentioned this pull request Oct 9, 2024

[REQUEST]: Document new setting to auto delete unenrolled agents elastic/ingest-docs#1376

Closed

juliaElastic and others added 2 commits October 9, 2024 14:17

fix types

8f20bca

[CI] Auto-commit changed files from 'make api-docs && make api-docs-s…

8fe491a

…taging'

nchaulet self-requested a review October 9, 2024 13:28

fix test

3c84092

juliaElastic requested a review from a team as a code owner October 9, 2024 14:21

Merge branch 'main' into cleanup-unenrolled

4a6a2dd

nchaulet reviewed Oct 9, 2024

View reviewed changes

ymao1 reviewed Oct 9, 2024

View reviewed changes

fix moving delete flag from true to false, add abortController

6fe1fc2

azasypkin reviewed Oct 9, 2024

View reviewed changes

juliaElastic requested a review from nchaulet October 10, 2024 07:07

juliaElastic force-pushed the cleanup-unenrolled branch from 1af548d to 6fe1fc2 Compare October 10, 2024 07:19

azasypkin approved these changes Oct 10, 2024

View reviewed changes

prevent delete flag to update from UI when preconfigured

147c2d4

nchaulet approved these changes Oct 10, 2024

View reviewed changes

jloleysens approved these changes Oct 10, 2024

View reviewed changes

fix checks

8c25642

juliaElastic force-pushed the cleanup-unenrolled branch from 80c15e0 to 8c25642 Compare October 10, 2024 11:23

ymao1 approved these changes Oct 10, 2024

View reviewed changes

juliaElastic and others added 6 commits October 10, 2024 14:06

update current_mappings

3c604cc

added finally block

42839cf

[CI] Auto-commit changed files from 'make api-docs && make api-docs-s…

b7b7eec

…taging'

Merge branch 'main' into cleanup-unenrolled

5e5bb64

Merge branch 'main' into cleanup-unenrolled

3d5c4a0

juliaElastic merged commit c53b2a8 into elastic:main Oct 14, 2024
28 checks passed

kibanamachine added the v9.0.0 label Oct 14, 2024

juliaElastic mentioned this pull request Oct 14, 2024

[8.x] [Fleet] delete unenrolled agents task #196072

Merged

juliaElastic added a commit that referenced this pull request Oct 14, 2024

[8.x] [Fleet] delete unenrolled agents task (#196072)

7595011

Backport #195544 to 8.x --------- Co-authored-by: kibanamachine <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] delete unenrolled agents task #195544

[Fleet] delete unenrolled agents task #195544

juliaElastic commented Oct 9, 2024 •

edited by kibanamachine

Loading

juliaElastic Oct 9, 2024

juliaElastic Oct 9, 2024

elasticmachine commented Oct 9, 2024

nchaulet Oct 9, 2024

juliaElastic Oct 9, 2024

juliaElastic Oct 9, 2024

nchaulet Oct 10, 2024

juliaElastic Oct 10, 2024

ymao1 Oct 9, 2024

juliaElastic Oct 9, 2024

azasypkin Oct 9, 2024

juliaElastic Oct 10, 2024

azasypkin left a comment

nchaulet left a comment

jloleysens left a comment

ymao1 left a comment

elasticmachine commented Oct 14, 2024

kibanamachine commented Oct 14, 2024

kibanamachine commented Oct 14, 2024

@@ @@ -181,6 +182,16 @@ export const getSavedObjectTypes = ( @@
                           },
                         ],
                       },
+: {

[Fleet] delete unenrolled agents task #195544

[Fleet] delete unenrolled agents task #195544

Conversation

juliaElastic commented Oct 9, 2024 • edited by kibanamachine Loading

Summary

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Oct 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

azasypkin left a comment

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

jloleysens left a comment

Choose a reason for hiding this comment

ymao1 left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 14, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

kibanamachine commented Oct 14, 2024

kibanamachine commented Oct 14, 2024

💔 All backports failed

Manual backport

Questions ?

juliaElastic commented Oct 9, 2024 •

edited by kibanamachine

Loading