[Ingest Manager] Agent remains in Enrolling state on executing the enroll command to move it to a different policy after install command

[ghost]

Kibana version:
Kibana: 7.10.0 BC1 cloud environment with commit 7a9f9132fffd8567ce3b985dde8bea6529e8fa3c

Elasticsearch version:
Elasticsearch: 7.10.0 BC1 cloud environment

Host OS and Browser version:
Windows , All

Original install method (e.g. download page, yum, from source, etc.):
7.10.0 BC1 cloud environment deployed from Staging cloud platform https://staging.found.no

Description
Agent remains in Enrolling state on executing the enroll command to move it to a different policy after install command

Preconditions

1. 7.10.0 BC1 cloud environment should be available(with commit 7a9f9132fffd8567ce3b985dde8bea6529e8fa3c).
2. Login to the above environment with elastic superuser.
3. 7.10.0 BC1 agent downloaded from https://staging.elastic.co/7.10.0-116336aa/downloads/beats/elastic-agent/elastic-agent-7.10.0-windows-x86_64.zip
4. A policy should exist.

Steps to Reproduce

1. RDP to the endpoint and place the agent in C:/ProgramFiles directory
2. Unarchive the agent
3. Open powershell as admin and Execute ./elastic-agent install with default policy
4. execute the enroll command on the host in powershell to move it to a different policy:
   .\elastic-agent enroll adfasdfasdf.asdf:9243 adsfasdfsdf==
5. Observe that agent remains in Enrolling state with no activity logs.
6. Execute .\elastic-agent run and observe error in powershell.

Test data
N/A

Impacted Test case id
N/A

Actual Result
Agent remains in Enrolling state on executing the enroll command to move it to a different policy after install command

Expected Result
Agent should be Online state with new policy and activity logs.

What's working

• N/A

What's not working

• N/A

Screenshot

1. On executing enroll command :
   

2. On executing run command:
   

Logs
N/A

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ghost]

Please review the bug @vipulsrivastava-qasource

[ghost]

Reviewed and assigned to @EricDavisX

[EricDavisX]

I like that you tried the elastic-agent.exe run command, but that shouldn't be done in general as part of the test, I expect with a running agent that should error. ITs a good error handling test. :)

Not sure if this is the same bug as is already found, but I'll raise it to the team now - thanks!

[EricDavisX]

@EricDavisX may not be critical for 7.10 but its good to identify the test cases we do want to eventually support (if not now) with the new 'install' command and what should work after it is executed. @blakerouse ?

[blakerouse]

@EricDavisX You can still run from the extracted directory, with enroll followed by install. The unable to trigger restart is not an error, its just showing you a warning. Its not a problem.

But you cannot have another Agent already running on the system, and that seems to be the issue.

[EricDavisX]

Thank you for confirming, we will not focus on that use case or error message - the steps they intended indicate they did not intentionally have a separate Agent running. so we will request a clean test environment, with confirmation that no service is installed and no Agent or Endpoint running, then we can assess the below:

• Unarchive the agent
• Open powershell as admin and Execute ./elastic-agent install with default policy
• execute the enroll command on the host in powershell to move it to a different policy:
  .\elastic-agent enroll adfasdfasdf.asdf:9243 adsfasdfsdf==
• assess Agent state

... in this case, should Agent require any 'run' command after the enroll to make Agent pick up the changes or does it automatically?

@rahulgupta-qasource I am learning that (I think) 'install' command is supported to be done multiple times in a row, and this is because for the most part, it just takes the param/values sent and runs the enroll / restart commands that exist. so, lets add a case for this:

• on clean environment with no Agent
• Unarchive the agent
• Open powershell as admin and Execute ./elastic-agent install with default policy
• assess Agent state, wait for it to come on-line
• still in powershell, execute './elastic-agent install' with a new policy (different token)
• assess Agent state and see it change in UI

If either of the above is still broken in current build we can capture logs and post details. Thanks

[ghost]

Hi @EricDavisX

Thank you for sharing the feedback.

We have validated the above scenario on 7.10.0 BC3 Kibana cloud environment with commit ed66f41a8a60ad03426beff65ed270a743c46ac4 and 7.10.0 BC3 elastic-agent downloaded from below location: https://staging.elastic.co/7.10.0-aea04452/downloads/beats/elastic-agent/elastic-agent-7.10.0-windows-x86_64.zip

Steps followed:

1. On clean VM(say win x64) with no Agent

2. Unarchive the agent

3. Open powershell as admin and Execute './elastic-agent install' with Default policy

4. Agent is displayed in Online state with running filebeat and metricbeat logs.
   Screenshot:
   

5. Do not uninstall the above agent and execute './elastic-agent install' for installing with new policy (different token)

6. Observe that already installed error is displayed:
   Screenshot:
   

7. Navigate to location C:\Program Files\Elastic\Agent and uninstall the agent using the command './elastic-agent uninstall' and observe that agent goes to Offline status.

8. Navigate to location C:\Program Files\Elastic-Agent and execute './elastic-agent install' command with a new policy say 'new_policy' (different token):

9. Observe that agent is installed with new_policy, is in Online state and running filebeat, metricbeat.
   Screenshot:
   

We have created the testcase C34910 for the same.

Please let us know if anything is missing from our end.

[blakerouse]

@rahulgupta-qasource Based on your latest comment, I believe this to be resolved then? Can this issue be closed?

[EricDavisX]

@rahulgupta-qasource thanks for the follow thru. I'd like to run one more test before closing it. In short-hand:
install Agent with default
then use 'enroll' command to change to a different policy.

Unless we have that covered already elsewhere. If all relating cases are working we can indeed close this

[dikshachauhan-qasource]

Hi @EricDavisX

We have performed scenario as per steps mentioned in above comment (#21794 (comment)) on 7.10.0 BC3 Kibana cloud environment with commit ed66f41a8a60ad03426beff65ed270a743c46ac4 and 7.10.0 BC3 elastic-agent downloaded from below location: https://staging.elastic.co/7.10.0-aea04452/summary-7.10.0.html/elastic-agent-7.10.0-windows-x86_64.zip

Steps followed:

1. On clean VM(say win x64) with no Agent

2. Unarchive the agent

3. Open powershell as admin and Execute './elastic-agent install' with Default policy

4. Agent is displayed in Online state with running filebeat and metricbeat logs.
   Screenshot:
   

5. Navigate back to location C:\Program Files\Elastic-Agent and execute '.\elastic-agent enroll https://0f31d12e79aa443383db22cb4a607a0f.us-central1.gcp.foundit.no:9243 SDB2eWFIVUJuOW9ER2NaWWlndk46U1FJZVVNMmhTMGVkY0tuYTNrTk5pQQ==' command using new policy's enrollment token.
   Screenshot:
   

Observation: - At UI, agent remains in enrolling state.
- Agent not moves under different policy when attempted to enroll using enroll command, after install.
Screenshot:

[blakerouse]

Looking at the screenshot of the PowerShell, this looks to be a user error. What you are doing is installing Elastic Agent and enrolling it (that is the Online) agent shown in Kibana. Then you are enrolling the agent again in the current directory, not the installed agent. You are never starting this agent so it stays in the Enrolling state in Kibana.

This is all expected behavour. You should not be executing the final enroll command in your testing. If you want to re-enroll the installed Elastic Agent you need to be sure to execute from the correct directory.

C:\Program Files\Elastic\Agent\elastic-agent.exe enroll

In your screenshot your doing it from the extracted directory of:

C:\Program Files\Elastic-Agent\elastic-agent.exe enroll

Which is incorrect.

[EricDavisX]

Thank you @blakerouse - this is a nuance I think that had not been updated in the test cases we had created, and I may have done a poor job of passing the specs out as we are learning the feature.

@rahulgupta-qasource @dikshachauhan-qasource thanks for the continued testing, can you update this in test content please and re-test it as noted?

Also, I'll request that we do a full review of test content that relates to 'install' or 'enrollment' and make sure we have the correct cited path (the installed Agent) listed for tests to be more confident moving forward. For reference, you can re-read thru the original issue with many good details, here: #21019

[dikshachauhan-qasource]

Hi @EricDavisX @blakerouse,

Thanks for the clarification of enroll command execution. We have updated the required information in our testcase.

https://elastic.testrail.io/index.php?/cases/view/34391

Also, we have retested it on 7.10.0 BC3 Kibana cloud environment with commit ed66f41a8a60ad03426beff65ed270a743c46ac4 and 7.10.0 BC3 elastic-agent downloaded from below location: https://staging.elastic.co/7.10.0-aea04452/summary-7.10.0.html/elastic-agent-7.10.0-windows-x86_64.zip

Steps followed:

1. Access an existing VM(say win x64) with no Agent

2. Unarchive the agent

3. Open powershell as admin and Execute './elastic-agent install' with Default policy

4. Agent is displayed in Online state with running filebeat and metricbeat logs.
   Screenshot:
   

5. Navigate to C:\Program Files\Elastic\Agent\ location in powershell and run enroll command using enrollment token of different policy.
   ./elastic-agent enroll https://b2c34fab897f48b4b44c1037069e8db5.europe-west1.gcp.cloud.es.io:9243 OU5WeWMzVUJ0aUZaelp5ZmNKRkc6eHdqUWhmaV9UcWlRRVplRXhBS3dkQQ==
   Screenshot:
   

Observation: Agent has been enrolled successfully with a new policy after install command.

Hence, we are closing this ticket.