[Fleet] Elastic-Agent doesn't start up with .rpm / .deb usage (with enroll cmd)

[EricDavisX]

I need to put details in... I can grab what logs do exist. But during the weekly sync we realized this wasn't logged so I'm dropping a ticket in. Tested with 8.0

The e2e-testing work is finding this bug:

edavis$ OP_LOG_LEVEL=TRACE DEVELOPER_MODE=true ELASTIC_AGENT_USE_CI_SNAPSHOTS=false ELASTIC_AGENT_VERSION=8.0.0-SNAPSHOT godog -t "endpoint-policy-check"

manual testing finds the same. linux .tar.gz is working fully.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

I wouldnt be surprised that debian is also broken. The package generation is quite similar between the two using fpm.

[EricDavisX]

here is what I did to get started:

from history output:
30 sudo rpm -vi elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm
32 sudo elastic-agent enroll https://8040186313ec4a228ecc22ebee9ed2c4.us-central1.gcp.foundit.no:443 YW9mUm9YUUJWMUZuVTV1TlJNeTQ6WjBTQ3R6VFZUb0toeTBEQ05CcGVlQQ==
34 sudo systemctl enable elastic-agent
35 sudo systemctl start elastic-agent

Here is what systemctl says after doing the install and enroll.

[zeus@mainqa-atlcolo-10-0-6-45 ~]$ ls
elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm leia-1.0.0-py2.7.egg
[zeus@mainqa-atlcolo-10-0-6-45 ~]$ sudo systemctl status elastic-agent
● elastic-agent.service - Agent manages other beats based on configuration provided.
Loaded: loaded (/usr/lib/systemd/system/elastic-agent.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2020-09-18 15:51:35 EDT; 2 days ago
Docs: https://www.elastic.co/products/beats/elastic-agent
Process: 1971 ExecStart=/usr/share/elastic-agent/bin/elastic-agent --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=203/EXEC)
Main PID: 1971 (code=exited, status=203/EXEC)

Sep 18 15:51:34 mainqa-atlcolo-10-0-6-45.eng.endgames.local systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
Sep 18 15:51:35 mainqa-atlcolo-10-0-6-45.eng.endgames.local systemd[1]: elastic-agent.service: Service RestartSec=100ms expired, scheduling restart.
Sep 18 15:51:35 mainqa-atlcolo-10-0-6-45.eng.endgames.local systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 5.
Sep 18 15:51:35 mainqa-atlcolo-10-0-6-45.eng.endgames.local systemd[1]: Stopped Agent manages other beats based on configuration provided..
Sep 18 15:51:35 mainqa-atlcolo-10-0-6-45.eng.endgames.local systemd[1]: elastic-agent.service: Start request repeated too quickly.
Sep 18 15:51:35 mainqa-atlcolo-10-0-6-45.eng.endgames.local systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
Sep 18 15:51:35 mainqa-atlcolo-10-0-6-45.eng.endgames.local systemd[1]: Failed to start Agent manages other beats based on configuration provided..
[zeus@mainqa-atlcolo-10-0-6-45 ~]$

here is a ps call that returns nothing:
[zeus@mainqa-atlcolo-10-0-6-45 ~]$ ps ax | grep elastic
4612 pts/0 S+ 0:00 grep --color=auto elastic
[zeus@mainqa-atlcolo-10-0-6-45 ~]$

here is the log after the enroll call, just the one line:
[zeus@mainqa-atlcolo-10-0-6-45 ~]$ sudo cat /var/lib/elastic-agent/data/elastic-agent-9e4ba2/logs/elastic-agent-json.log
{"log.level":"debug","@timestamp":"2020-09-18T15:51:07.391-0400","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/enroll","ecs.version":"1.5.0"}
[zeus@mainqa-atlcolo-10-0-6-45 ~]$

and there is no action store file at:
/var/lib/elastic-agent/data/elastic-agent-9e4ba2/action_store.yml

And I tried running the .tar.gz on a host and it worked, and then I un-installed it and used the .rpm and started it up and find the same as the above (so I know the host should be ok).

[ph]

@blakerouse is this still an issue?

[EricDavisX]

I can hit this today to verify on 7.10 BC2 build.

[EricDavisX]

And I can confirm - with the QAS team's help, the 7.10 BC2 is indeed still in need of some fixing for the .rpm / .deb installs

[EricDavisX]

this is awesome, thanks Blake.

@rahulgupta-qasource I'll pass a note in the daily email to discuss testing. Thanks!

[ghost]

Bug Conversion:

Below 01 Testcase already exists for this ticket:
https://elastic.testrail.io/index.php?/cases/view/33960