[Agent] "Metricbeat" service crashed on restarting elastic-agent, want improved beat side logging to inform user

[amolnater-qasource]

Kibana version: 7.13.0 BC-7 Kibana self-managed environment

Host OS and Browser version: Windows 10 x64, All

Build Details:

 Artifact link used: https://staging.elastic.co/7.13.0-8eb98cbf/summary-7.13.0.html
 BUILD: 40864
 COMMIT: 6ce6847436ff9bef0ad91268b6585e0f9339c9fd

Preconditions:

1. 7.13.0 BC-7 self-managed Kibana environment should be available.
2. Windows 10 x64 Fleet Server agent must be installed using Default Fleet server policy having only Fleet Server integration.

Steps to reproduce:

1. Login to Kibana environment.
2. Assign "Agent" to Default policy having System, Endpoint and Fleet Server Policy.
3. Restart elastic-agent from services.
4. Navigate to agent "Logs" tab and observe Metricbeat service stuck in "Starting-Restarting-Crashed" loop.

Expected Result:
[Self managed]: "Metricbeat" service should be in "Running" state on restarting elastic-agent.

Logs:
logs.zip

Screenshot:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@dikshachauhan-qasource Please review.

[dikshachauhan-qasource]

Reviewed and assigned to @EricDavisX

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[EricDavisX]

@ph @ruflin this seems like the root cause to the issue I just pinged you on. It also sounds familiar and pretty severe. Is this not a blocker for self-managed where we expect the stack / agents to go on/off line quite a bit?

[michalpristas]

cannot reproduce, nor darwin nor windows, used same version.

could i get content of data/elastic-agent-{hash}/logs directory? this would help

[michalpristas]

also if possible maybe whole Program Files/Elastic/Agent would be good. not sure if github allows to send that large file, if not please DM me on slack

[amolnater-qasource]

Sure @michalpristas as per request I will share all the required details on slack.
Thanks

[michalpristas]

there was an orphaned metricbeat process blocking data path, when agent was stopped it was not stopped with agent and continue running eating a lot of RAM (2.5gb)

when i killed this process issue went away.
the problem with restart was that the orphaned metricbeat was using same data path as the one we were trying to start. the new one found out path is already locked and exited with error.

we should find out why orphaned metricbeat was even created and could possibly isolate data paths more. but i think it's good to know you have such a process by seeing agent complaining rather that have multiple beats running and eating resources

[EricDavisX]

@michalpristas thanks for the thoughts. do you think we can check for the data path being in use at start up and put more specific error logging into place to help guide users / devs about this particular undesired occurrence? we could use this ticket for that for now. @amolnater-qasource if you were doing regular operations / test steps, lets put some time to figuring out which specific steps caused this and log a new ticket, i expect the description would be 'after doing xyz Agent is shutdown but metricbeat is orphaned and still running'

[michalpristas]

this is internal logic of beats and i would like to avoid having checks like this in agent.
we could check periodically or at start for processes running from our path before starting anything and raising it as ERROR WARNING. but i guess this is up to a discussion

cc @urso

[amolnater-qasource]

Hi @EricDavisX
As per your above feedback we have reported a ticket with required steps at this #25829

Thanks
QAS

[EricDavisX]

I've updated the short desc and modified the labels, it is not urgent for 7.13 to improve the logging - the other ticket logged, noted above is high priority for self-managed usage and is tagged for 7.13.1 review (usual urgent issues review process)

[amolnater-qasource]

Hi @EricDavisX
We have revalidated this issue on 7.14.0 self-managed Snapshot Kibana and found it fixed.

Build Details:

Artifact link: https://snapshots.elastic.co/7.14.0-28665d9b/downloads/beats/elastic-agent/elastic-agent-7.14.0-SNAPSHOT-windows-x86_64.zip
Build: 41559
Commit: 9838db392e7fcfc12f004b68fb1b09739f131148

Steps followed:

1. Login to self-managed Kibana environment.
2. Install Agent with Default Fleet Server Policy.
3. Assign "Agent" to Default policy having System, Endpoint and Fleet Server Integration.
4. Restart elastic-agent from services.

Observations:
We observed Filebeat and Metricbeat in running state after restart under agent Logs tab.

Screenshot:

Hence closing this out.

Thanks
QAS

[amolnater-qasource]

Hi @michalpristas
We have observed an inconsistent Metricbeat Starting, Restarting and Crashed loop just after installing agent on 7.14.0 Snapshot Kibana cloud environment.

• However it gets resolved itself after few minutes, agent gets offline and then comeback online.
• All the binaries comeback in RUNNING state.

Build details:

Build: 42366
Commit: 22dee04008b9936be37225b97a6456e750d559a7
Artifact Link: https://snapshots.elastic.co/7.14.0-ef1f955b/downloads/beats/elastic-agent/elastic-agent-7.14.0-SNAPSHOT-windows-x86_64.zip

Logs:
logs.zip

Please let us know if anything else is required from our end.
cc: @EricDavisX
Thanks
QAS