Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Clean up documents in .fleet* indices related to agents that were already unenrolled #189506

Closed
3 tasks
criamico opened this issue Jul 30, 2024 · 5 comments · Fixed by #195544
Closed
3 tasks

[Fleet] Clean up documents in .fleet* indices related to agents that were already unenrolled #189506

criamico opened this issue Jul 30, 2024 · 5 comments · Fixed by #195544
Assignees
@juliaElastic
Labels
QA:Validated Issue has been validated by QA Team:Fleet Team label for Observability Data Collection Fleet team v8.16.0

Comments

@criamico
Copy link
Contributor

criamico commented Jul 30, 2024 •
edited
Loading

Follow up of #179399

After unenrolling agents, we leave some documents related to these agents that can be a problem for some users, as they take up space.
However this constitutes a breaking change as we've never deleted those docs and in some cases they can be useful for recoverability purposes. So we should find a way to delete those documents only if the users want to.

  • Create a Kibana task that cleans up the documents after some time that agents where unenrolled.
  • Have an opt-in option (false by default) in advanced section of Fleet settings and in preconfiguration that schedules this task only when the user has selected it. Frequency of the task TBD.
    Note that it should be a global option, i.e. it should apply to all the unenrolled agents, regardless of the inactive timeout being selected or not.
  • Update docs to clearly explain what this option does.
@criamico criamico added the Team:Fleet Team label for Observability Data Collection Fleet team label Jul 30, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@criamico criamico mentioned this issue Jul 30, 2024
Closed
13 tasks
@nimarezainia
Copy link
Contributor

thanks @criamico - can this be an option in the advanced section of the Fleet settings? I think it's a great idea but it would need to be an opt-in as mentioned, and it won't apply necessarily at an Agent Policy level. Globally however, I think we would want the user to make that decision about any agent that has un-enrolled (regardless of the timeout being configured or not).

@criamico
Copy link
Contributor Author

criamico commented Jul 31, 2024 •
edited
Loading

can this be an option in the advanced section of the Fleet settings?

@nimarezainia yes I think it would be a good place to add it.

I think it's a great idea but it would need to be an opt-in as mentioned, and it won't apply necessarily at an Agent Policy level. Globally however, I think we would want the user to make that decision about any agent that has un-enrolled (regardless of the timeout being configured or not).

I too think that a global option would be more consistent and also more clear for the user. The text description and the docs should explain clearly what this option does, since it can be disruptive in some cases. I'll update the ticket description to clarify those points.

@juliaElastic juliaElastic self-assigned this Sep 12, 2024
@kpollich kpollich added QA:Needs Validation Issue needs to be validated by QA v8.16.0 labels Sep 18, 2024
@kpollich
Copy link
Member

FYI @amolnater-qasource I have added this as a QA issue for 8.16.0

This was referenced Oct 9, 2024
Merged
Closed
juliaElastic added a commit to juliaElastic/kibana that referenced this issue Oct 14, 2024
@juliaElastic @kibanamachine
[Fleet] delete unenrolled agents task (elastic#195544)
643bcad 
Closes elastic#189506

Testing steps:
- enable deleting unenrolled agents by adding
`xpack.fleet.enableDeleteUnenrolledAgents: true` to `kibana.dev.yml` or
turn it on on the UI
- add some unenroll agents with the helper script
```
cd x-pack/plugins/fleet
node scripts/create_agents/index.js --status unenrolled --count 10

 info Creating 10 agents with statuses:
 info    unenrolled: 10
 info Batch complete, created 10 agent docs, took 0, errors: false
 info All batches complete. Created 10 agents in total. Goodbye!
```
- restart kibana or wait for the task to run and verify that the
unenrolled agents were deleted
```
[2024-10-08T16:14:45.152+02:00][DEBUG][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] Executed deletion of 10 unenrolled agents
[2024-10-08T16:14:45.153+02:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:0.0.5] [DeleteUnenrolledAgentsTask] runTask ended: success
```

Added to UI settings:
<img width="1057" alt="image"
src="https://github.com/user-attachments/assets/2c9279f9-86a8-4630-a6cd-5aaa42e05fe7">

If the flag is preconfigured, disabled update on the UI with a tooltip:
<img width="1009" alt="image"
src="https://github.com/user-attachments/assets/45041020-6447-4295-995e-6848f0238f88">

The update is also prevented from the API:
<img width="2522" alt="image"
src="https://github.com/user-attachments/assets/cfbc8e21-e062-4e7f-9d08-9767fa387752">

Once the preconfiguration is removed, the UI update is allowed again.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <[email protected]>
@amolnater-qasource
Copy link

amolnater-qasource commented Nov 11, 2024 •
edited
Loading

Hi Team,
We have revalidated this on latest 8.16.0 BC5 kibana cloud environment and found it working fine.

Observations:

  • Unenrolled agents are removed from kibana on restart on enabling Delete unenrolled agents.
  • On adding xpack.fleet.enableDeleteUnenrolledAgents: true Delete unenrolled agents toggle is already enabled.
  • Unenrolled agents are successfully deleted from UI.

Build details:
VERSION: 8.16.0 BC6
BUILD: 79644
COMMIT: a8a07df

Screenshots:
Image
Image
Image

Hence we are marking this ticket as QA:Validated.

Please let us know if anything else is required from our end.
Thanks!

@amolnater-qasource amolnater-qasource added QA:Validated Issue has been validated by QA and removed QA:Needs Validation Issue needs to be validated by QA labels Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@juliaElastic juliaElastic

Labels
QA:Validated Issue has been validated by QA Team:Fleet Team label for Observability Data Collection Fleet team v8.16.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fleet] delete unenrolled agents task juliaElastic/kibana
6 participants
@kpollich @nimarezainia @elasticmachine @criamico @amolnater-qasource @juliaElastic