"elastic_agent.apm" data events not available on Logs tab after enabling logs collection settings at cloud deployment.

[dikshachauhan-qasource]

Kibana version:
7.13 BC4 Kibana Cloud Staging environment

Host OS and Browser version:
Managed host on cloud, All

Preconditions

1. 7.13 BC4 Kibana Cloud Staging environment should be available.

Kibana version: 7.13 BC4
BUILD 40749
COMMIT 5a6bad454ffe263aafed54cbd3f764253694bf37
Artifact link: https://staging.elastic.co/7.13.0-5c4bc719/summary-7.13.0.html

2. Enable logs and metrics setting at cloud deployment page.

Steps to reproduce:

1. Login to Kibana cloud environment.
2. Go to Stack monitoring tab.
3. Observe, APM server instance data is available.

4. Go to logs tab, and look for metrics collection for apm server using event.dataset filter.

Actual Result:

• "elastic_agent.apm" data events not available on Logs tab after enabling logs collection settings at cloud deployment.
• Data stream for apmserver is also not available on Fleet>Data stream tab.

Screenshots:

[elasticmachine]

Pinging @elastic/fleet (Feature:Fleet)

[dikshachauhan-qasource]

@manishgupta-qasource Please review

[manishgupta-qasource]

Reviewed & Assigned to @EricDavisX

[EricDavisX]

@simitt @mostlyjason @ruflin I think this was in response to slack discussions about the feature not working. please do review and help push it on.

[simitt]

Not seeing any apm data in datastreams is expected, as apm is still running standalone in the setup (not an elastic-agent integration).

Enabling metrics & logs collection in the cloud UI triggers sending logs and metrics to the users cluster. For APM and Elastic Agent & Fleet Server following should be shown.

• Logs: navigate to the log app and query for event.dataset : "apm.log" and event.dataset : "agent.log" and ensure you see logs. (note: for apm log level error is set, while for agent+fleet-server log level info is set). The agent logs should contain logs from the agent and the fleet-server.

• Metrics: Navigate to the Stack Monitoring tab and ensure that there is an APM & Fleet Server section, in the details of this section the APM & Fleet Server - Resource Usage should be shown (note: no event data are expected as long as you don't ingest any apm events).

  Note: this currrently needs to be tested with the latest daily snapshot and will only be contained in the next Build Candidate (2021-05-11).

[jen-huang]

I'm closing this based on @simitt's notes above (thanks Silvia!), please re-open if needed.

[dikshachauhan-qasource]

Hi @simitt

Thanks for the details and feedback. We will revalidate it with the latest BC available now and will re-open if required.

Thanks
QAS

[dikshachauhan-qasource]

Hi @EricDavisX

We have revalidated the above scenario on latest 7.13 BC7 and found it working fine for below agent logs:

• agent.log
• elastic_agent
• elastic_agent.fleet-server

Steps performed:

• enabled logs and metrics at cloud deployment.
• Validated data sync at apm # fleet overview resource usage under stack monitering.

@simitt for apm.log today, we observed that these are not available on Kibana restart. We attempted 2-3 times. Please let us know if required to raise a ticket for this. Or any pre-condition is required to generate them.

Build details:

BUILD 40864
COMMIT 6ce6847436ff9bef0ad91268b6585e0f9339c9fd

Thanks
QAS

[simitt]

@simitt for apm.log today, we observed that these are not available on Kibana restart. We attempted 2-3 times. Please let us know if required to raise a ticket for this. Or any pre-condition is required to generate them.

@dikshachauhan-qasource the APM log level is set to error. Have you tried to manually trigger an error for APM, for example by running curl -i <apm-url>/config/v1/agents? (Should return an authorization error). After this, APM logs should be available. If they are not, please raise a ticket.

[dikshachauhan-qasource]

@simitt Thanks for the feedback.

We have reported required issue at #1000324 and linked with it.

cc @EricDavisX