Elastic Agent: Error: fail to enroll: fail to execute request to Kibana: EOF

[mtojek]

We noticed some flakiness with enrollment:

Attaching to elastic-package-stack_elastic-agent_1
�[36melastic-agent_1              |�[0m The Elastic Agent is currently in BETA and should not be used in production
�[36melastic-agent_1              |�[0m 
�[36melastic-agent_1              |�[0m Error: fail to enroll: fail to execute request to Kibana: EOF
�[36melastic-agent_1              |�[0m Error: enrollment failed: exit status 1

Source: https://beats-ci.elastic.co/job/Ingest-manager/job/integrations/job/master/227/artifact/build/elastic-stack-dump/latest/barracuda/logs/elastic-agent.log

We aren't using the recommended healthcheck due to the other issue: #24956

So far it's https://github.com/elastic/elastic-package/blob/master/internal/install/static_snapshot_yml.go#L105.

Do you have any idea why was there the EOF issue?

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[mtojek]

Another occurrence:

https://beats-ci.elastic.co/job/Ingest-manager/job/integrations/job/master/229/artifact/build/elastic-stack-dump/latest/system/logs/elastic-agent.log

@blakerouse @michalpristas any idea what can be the root cause here?

[michalpristas]

havent seen this so far, any special circumstances?
is this master?

[michalpristas]

looking at the code this is most likely coming from http.Client itself, other errors are decorated with messages.
are we sure kibana is up and running?
other reason may be closing the connection and tring to reuse that one after rate limiting, but for that i would expect some warnings in agent logfile. do we have these available?

[mtojek]

is this master?

7.13.0-SNAPSHOT

but for that i would expect some warnings in agent logfile. do we have these available?

All logs are present here:
https://beats-ci.elastic.co/job/Ingest-manager/job/integrations/job/master/229/artifact/build/elastic-stack-dump/latest/system/logs/

When a similar issue happened in the past I suspected that something restarts in background (fleet-server?).

[michalpristas]

i was hoping for a log file inside /opt/Elastic/Agent

[mtojek]

Unfortunately we don't have it. If there is a way to route the output of that file to stdout/stderr we can grab it with docker logs.

[blakerouse]

@michalpristas 7.13+ docker of Elastic Agent routes the elastic-agent.log now to stdout so that it can be picked up with docker logs. Unless it is ran with the LOGS_PATH environment variable.

[michalpristas]

havent seen this in last 12 builds, will keep an eye but maybe it was fixed as a side effect of other enroll changes.
will keep an eye on builds for next week and if i dont see it, i will close the issue

[mtojek]

We suspected that it may be related to the healthcheck issue, but unfortunately we can't switch to the correct one yet. For reference: elastic/elastic-package#329 (comment)

[ph]

I am going to close this, its april 13.