No activity logs are displayed on installing agent on upgraded 7.10.0 BC2 cloud environment with 7.9.2 BC2 policy

[ghost]

Kibana version:
Kibana: Upgraded 7.10.0 Kibana BC2 cloud environment with commit f1c0bdd

Elasticsearch version:
Elasticsearch: Upgraded 7.10.0 BC2 cloud environment

Host OS and Browser version:
Windows , All

Original install method (e.g. download page, yum, from source, etc.):
7.9.2 Kibana BC2 cloud environment upgrade to 7.10.0 Kibana BC2 cloud environment from https://staging.found.no

Description
No activity logs are displayed on installing agent on upgraded 7.10.0 BC2 cloud environment with 7.9.2 BC2 policy

Preconditions

1. 7.9.2 BC2 cloud environment should be available.
2. Login into the above Kibana environment with 'elastic' superuser.
3. Agent configuration with name 'Rahul' and 'Rahul' should be created under "Configurations" tab in 7.9.2 BC2 Kibana cloud environment
4. 7.10.0 BC2 agent download from location on Host:
   https://staging.elastic.co/7.10.0-174bd372/downloads/beats/elastic-agent/elastic-agent-7.10.0-windows-x86_64.zip

Steps to Reproduce

1. Upgrade 7.9.2 BC2 cloud environment to 7.10.0 BC2 Kibana cloud environment
2. Click on "Fleet" link from L.H.S menu section and navigate to "Fleet" page.
3. Navigate to Policies tab
4. Install agent with ./elastic-agent install on Host using prerequisite 'Rahul' policy enrollment token.
5. Observe that no activity logs are displayed for above host under Agents->Host in Kibana.

Test data
N/A

Impacted Test case id
https://elastic.testrail.io/index.php?/cases/view/34209

Actual Result
No activity logs are displayed on installing agent on upgraded 7.10.0 BC2 Kibana cloud environment with 7.9.2 BC2 policy

Expected Result
Agent activity logs should be displayed on installing agent on upgraded 7.10.0 BC2 Kibana cloud environment with 7.9.2 BC2 policy

What's working

• Agent is in Online status

What's not working

• N/A

Screenshot

Logs
elastic-agent logs.zip

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ghost]

@vipulsrivastava-qasource Please review the bug

[ghost]

Reviewed and Assigned @EricDavisX

[EricDavisX]

@rahulgupta-qasource can you do an adjacent tests to give us more data on this please:

• if you create a new policy in 7.10 after upgrading does it work as expected on a similar host?

@nchaulet hi - any reason we can think of why 7.9 enrollment tokens would not work after upgrading the env to 7.10 and installing on a new agent? I think during a Beta we could get away with it... but for production use folks will def expect to be able to upgrade and continue to deploy with same policy / tokens right?

[nchaulet]

Normally 7.9 enrollment tokens should works in 7.10, looks like there is no POLICY distributed to the agent here I am investigating and able to have a repro locally

[nchaulet]

will be fixed by https://github.com/elastic/beats/issues/21904

[EricDavisX]

ok thanks. i'm moving this to Kibana repo then. cheers, and thanks for the quick fix.

[EricDavisX]

Nicolas notes it is fixed by:
[Fleet] Fix creation of POLICY_CHANGE action during 7.9 => 7.10 migration
#81041

[ghost]

Hi Eric

On creating a new policy(say Rahul_new) on upgraded 7.10.0 BC2 build(upgrade from 7.9.2 BC2 to 7.10.0 BC2) and installing agent with ./elastic-agent install on Host using new 'Rahul_new' policy enrollment token.

Observation:
we observed that only POLICY CHANGE acknowledged activity logs are displayed.

Screenshot:

Elastic -agent logs:
elastic-agent logs.zip

Please let us know if anything is missing from our end.

[EricDavisX]

the fix is merged, but not built into a BC yet. It went in to 7.10 line yesterday (Tuesday) at 4:40 PM
#81229

The 7.10 builds are here: https://internal-ci.elastic.co/view/Unified%20Release%20-%20snapshot/job/elastic+unified-release+master+snapshot-build-7.10/

As far as BC3 or beyond goes, we will hear when it arrives, it may not be in that BC either, so we may have to wait.

[EricDavisX]

BC3 is out @rahulgupta-qasource it would be ok to try this out, would be nice to close it if we can. I never heard or checked if it made it in for sure tho, sorry.

[ghost]

Hi @EricDavisX

We have validated the above scenario after upgrade from 7.9.2 BC2 to 7.10.0 BC3 Kibana cloud environment(with commit ed66f41) and 7.10.0 BC3 elastic-agent downloaded from below location: https://staging.elastic.co/7.10.0-aea04452/downloads/beats/elastic-agent/elastic-agent-7.10.0-windows-x86_64.zip

Observed that :

1. Activity logs are successfully displayed after installing agent on host with prerequisite 7.9 'Rahul' policy enrollment token.
   Screenshot:
   

2. Activity logs are successfully displayed after installing agent on host with new policy created on upgraded 7.10.0 BC3 Kibana cloud environment
   Screenshot:
   

Hence, closing the ticket.

[ghost]

Bug Conversion:

Created 01 Testcase for this ticket:
https://elastic.testrail.io/index.php?/cases/view/35145