Make enrollment errors more accurate

[michalpristas]

errors when enroll fails is not accurate enough
with agent running using

sudo /root/elastic-agent/elastic-agent enroll -f --insecure \
  --url=https://192.168.117.101:8220 \
  --fleet-server-es=https://192.168.117.101:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzA3NDA3NTM5ODQ6OTFkRm1FdXBRbXlwLWk4R3Nyd1p0UQ \
  --fleet-server-policy=0d8b9ba0-0ce5-11ec-8978-2bc18b6d481d \
  --certificate-authorities=/etc/elasticsearch/other/ca/ca.crt \
  --fleet-server-es-ca=/etc/elasticsearch/other/elasticsearch-ca.crt \
  --fleet-server-cert=/etc/elasticsearch/other/fleet-server/fleet-server.crt \
  --fleet-server-cert-key=/etc/elasticsearch/other/fleet-server/fleet-server.key

this output is retrieved

2021-09-04T18:56:37.214+1000    INFO    cmd/enroll_cmd.go:508   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-09-04T18:56:37.337+1000    INFO    application/application.go:66   Detecting execution mode
2021-09-04T18:56:37.337+1000    INFO    application/application.go:87   Agent is in Fleet Server bootstrap mode
2021-09-04T18:56:37.581+1000    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-09-04T18:56:37.581+1000    INFO    application/fleet_server_bootstrap.go:124       Agent is starting
...
<redacted flees server restarts>
...
2021-09-04T18:57:43.276+1000    INFO    cmd/enroll_cmd.go:664   Fleet Server - Missed last check-in
2021-09-04T18:57:43.276+1000    WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-09-04T18:57:43.838+1000    INFO    cmd/enroll_cmd.go:396   Starting enrollment to URL: https://192.168.117.101:8220/
2021-09-04T18:57:43.939+1000    INFO    cmd/run.go:189  Shutting down Elastic Agent and sending last events...
2021-09-04T18:57:43.939+1000    INFO    operation/operator.go:192       waiting for installer of pipeline 'default' to finish
2021-09-04T18:57:43.939+1000    INFO    process/app.go:176      Signaling application to stop because of shutdown: fleet-server--7.14.1
2021-09-04T18:58:14.978+1000    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-09-04T18:58:14.978+1000    INFO    cmd/run.go:197  Shutting down completed.
2021-09-04T18:58:14.978+1000    INFO    log/reporter.go:40      2021-09-04T18:58:14+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-04T18:58:14.978+1000    INFO    [api]   api/server.go:66        Stats endpoint (/root/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /root/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection
Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred:
        * missing enrollment api key

it looks like api key is missing. But if we look closely to fleet server logs we see

{"log.level":"info","service.name":"fleet-server","version":"7.14.1","commit":"834362b","pid":2618391,"ppid":2618352,"exe":"/root/elastic-agent/data/elastic-agent-703d58/install/fleet-server-7.14.1-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///root/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/root/elastic-agent/data/elastic-agent-703d58/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/root/elastic-agent/data/elastic-agent-703d58/run/default/fleet-server--7.14.1"],"@timestamp":"2021-09-04T08:56:38.262Z","message":"boot"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"error","service.name":"fleet-server","error.message":"1 error: file is not a certificate adding /etc/elasticsearch/other/elasticsearch-ca.crt to the list of known CAs accessing 'output.elasticsearch'","@timestamp":"2021-09-04T08:56:38.768Z","message":"Exiting"}

the root cause of failed enroll is Fleet Server failing to read invalid certificate.

This might be misleading and is definitely wrong.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!