[Inconsistent]: Agent showing inconsistent behaviour on changing log level to debug when installed with a policy having System and Endpoint Security

[amolnater-qasource]

Kibana version: 7.13.1 BC-2 Kibana cloud environment

Host OS and Browser version: Windows 10 x64, All

Build Details:

 Artifact link used: https://staging.elastic.co/7.13.1-acc5da1f/downloads/beats/elastic-agent/elastic-agent-7.13.1-windows-x86_64.zip
 Build: 40903
 Commit: 9b177ae95979d96cf172517a3c6e469bbbda07ed

Preconditions:

1. 7.13.1 Kibana cloud environment should be available.
2. Hosted Fleet Server agent must be available.
3. Windows 10 x64 agent must be installed with Default Policy having System and Endpoint Integration.

Steps to reproduce:

1. Login to Kibana environment.
2. Navigate to logs tab and set logging level to Debug
3. Observe Metricbeat Starting-Restarting logs or if Metricbeat is running Agent goes to offline state.

Expected Result:
Agent should remain Healthy after changing log level to debug.

Agent Logs for Metric-beat issue :
logs.zip

Agent Logs for offline issue :
logs2.zip

Screenshots:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Reviewed & Assigned to @EricDavisX

[EricDavisX]

@michalpristas is this related to the other 2 metricbeat crashing / restart loop problems? It may be a false positive and go away when the other issues are fixed.

[michalpristas]

may be related yes, let's retest once fix make it to snapshot

[EricDavisX]

just confirming, the duplicate issue is: #26041

• let's track that and re-test this when it is confirmed in the build and has been backported. i think that will be soon.

[amolnater-qasource]

Hi @EricDavisX

We reported this issue as this was reproduced on a different Kibana version and with different steps. Also, Issue here is related to Agent status going offline after changing logging level. Logs are highlighted for help as per observation made in whole scenario.

Further, we will retest this issue whenever Metricbeat issue will be fixed to confirm the behavior.

Thanks
QAS

[amolnater-qasource]

Hi @EricDavisX
We have revalidated this issue on 7.14.0 Snapshot Kibana Cloud environment and found it still reproducible on 7.14.0 Snapshot.

Observations:

• On changing log level, Metricbeat is stuck in Starting-Restarting-Crashed loop under agent Logs tab.

Logs:
logs.zip

Build details:

Build: 41559
Commit: 9838db392e7fcfc12f004b68fb1b09739f131148
Artifact Link: https://snapshots.elastic.co/7.14.0-28665d9b/downloads/beats/elastic-agent/elastic-agent-7.14.0-SNAPSHOT-windows-x86_64.zip

Please let us know if anything else is required from our end.

Thanks
QAS

[EricDavisX]

@michalpristas it has been re-tested... do we need newer data? Maybe not. Keep in touch if you need anything or want to bring in other team mates to help triage.

[michalpristas]

fix was not part of the tested version above.
this should be already adressed and needs to be retested. hopefully closed as well :-)

[ph]

@michalpristas can you add a date / commit when this was fixed so we can validate that we have correctly fixed the problem.

[michalpristas]

commit in 7.x: e73b271
pushed in Jun 11, 2021

[michel-laterman]

@amolnater-qasource, I'm not able to reproduce this with agent v7.13.3. Can you also retry?

[amolnater-qasource]

Hi @michel-laterman
We have validated this issue on 7.13.3 BC-2 and we observed it was fixed there.
However we observed this issue reproducing on 7.14.0 Snapshot and shared our observations at #26034 (comment)

We kept this open just to keep a track for 7.14.0 as we don't have any stable BC build for 7.14.0.
We will revalidate this issue once 7.14.0 BC-2 will be available.

Thanks
QAS

[michel-laterman]

@amolnater-qasource ok, i'll reopen the issue, please let me know if it occurs again.

[amolnater-qasource]

Hi @michel-laterman
We have tested this issue on latest 7.14.0 Snapshot environment and found this issue not reproducible anymore.

Screenshot:

Build details:

Build: 42479
Commit: 7f950fbe0d37a576158ad19f5c7c8244628d67e6
Artifact Link: https://snapshots.elastic.co/7.14.0-9b23ec55/downloads/beats/elastic-agent/elastic-agent-7.14.0-SNAPSHOT-windows-x86_64.zip

Hence, we are closing this.

Thanks
QAS