Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[APM & Fleet]: 'apm.log' logs NOT available after enabling Logs and metrics and restarting Kibana from Cloud deployment page. #134559

Open
ghost opened this issue Jun 16, 2022 · 19 comments
Open

[APM & Fleet]: 'apm.log' logs NOT available after enabling Logs and metrics and restarting Kibana from Cloud deployment page. #134559

ghost opened this issue Jun 16, 2022 · 19 comments
Labels
bug Fixes for quality problems that affect the customer experience impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@ghost
Copy link

ghost commented Jun 16, 2022

Description:
'apm.log' logs NOT available after enabling Logs and metrics and restarting Kibana from Cloud deployment page.

Build Details:
VERSION: 8.3.0-BC4
BUILD: 53413
COMMIT: 875ea18
ARTIFACT LINK: https://staging.elastic.co/8.3.0-fcbaa721/summary-8.3.0.html

Preconditions:

  1. Elastic 8.3.0-BC4 environment should be deployed.

Steps to Reproduce:

  1. Enable logs and metrics at cloud deployment.
  2. Wait for 1-2 mins for action to complete.
  3. Got to stack monitoring Tab in kibana.
  4. Validate data sync at apm & fleet overview resource usage.
  5. Restart Kibana.
  6. Go to logs tab.
  7. Apply event.dataset filter for below: "apm.log"
  8. Observe that 'apm.log' logs NOT available after enabling Logs and metrics and restarting Kibana from Cloud deployment page.

Actual Result:
"elastic_agent.apm" data events not available on Logs tab after enabling logs collection settings and restarting Kibana at cloud deployment.

Screenshot:
Screenshot (550)
Screenshot (551)
@ghost ghost added bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team labels Jun 16, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@ghost
Copy link
Author

ghost commented Jun 16, 2022

@manishgupta-qasource Please review.

@ghost ghost added the impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. label Jun 16, 2022
@manishgupta-qasource
Copy link

Secondary review for this ticket is Done

@jen-huang
Copy link
Contributor

Not sure that this is a Fleet issue. I do see elastic_agent.apm_server in the screenshot so it seems to be instrumenting correctly. @samratbhadra-qasource Do you have a ticket reference for expecting to see apm.log?

@ghost
Copy link
Author

ghost commented Jun 17, 2022

Hi @jen-huang
Please find below the reference ticket for the issue:

Please do let us know if anything else is required from our end.

cc: @simitt
Thanks!

@jen-huang
Copy link
Contributor

Digging through the related issues, it might be due to the APM logging level set to "error" and the lack of logs simply indicates that no errors came in. @samratbhadra-qasource Could you re-validate using the instructions here from Silvia?: #100324 (comment)

@ghost
Copy link
Author

ghost commented Jun 30, 2022

Hi @jen-huang
We have revalidated this issue on the 8.3.0 release build and observed that the issue is still reproducible. Please find below the testing details:

Screenshot:
Screenshot (607)
Screenshot (605)

Build Details:
VERSION: 8.3.0
BUILD: 53518
COMMIT: 8a54c80

Please do let us know if anything else is required from our end.

Thanks!

@jen-huang jen-huang added Team:APM All issues that need APM UI Team support and removed Team:Fleet Team label for Observability Data Collection Fleet team labels Jun 30, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/apm-ui (Team:apm)

@jen-huang
Copy link
Contributor

@simitt / @axw Any insights for the above observations?

@ghost
Copy link

ghost commented Oct 13, 2022

Hi @jen-huang

  • We have revalidated this issue on the 8.5 BC3 Kibana Cloud environment and observed that the issue is still reproducible.

Build Details:
Build: 56932
Commit: 1bb0d05

Screenshot:

image

image
image

Please do let us know if anything else is required from our end.
Thanks

@ghost
Copy link

ghost commented Dec 2, 2022

Hi @simitt

  • We have revalidated this issue on the latest 8.6 BC4 Kibana Cloud environment and observed that the issue is still reproducible.

Build Details:
BUILD: 58612
COMMIT: 218162f

Screenshot:
image
image

CC: @jen-huang Please let us know if we are missing anything.
Thanks

@simitt
Copy link
Contributor

simitt commented Dec 5, 2022

When I check an 8.6 deployment, I also do not see any fleet-server logs. My assumption is that something might be off with the log collection for the Elastic Agent subprocesses. Probably worth pulling in the Elastic Agent team for this.

@kpollich
Copy link
Member

kpollich commented Dec 5, 2022

This looks like a result of elastic/elastic-agent#1814. Looks like a fix may have landed in elastic/elastic-agent#1845 but probably wasn't in the BC used for testing here.

@jen-huang
Copy link
Contributor

@simranvaseer-qasource Could you revalidate this with the latest 8.6 BC?

@amolnater-qasource amolnater-qasource added the QA:Ready for Testing Code is merged and ready for QA to validate label Dec 22, 2022
@ghost
Copy link

ghost commented Dec 29, 2022

Hi @jen-huang

  • We have revalidated this issue on the latest 8.6 BC9 Kibana Cloud environment and observed that the issue is still reproducible.

  • "elastic_agent.apm" data events not available on Logs tab after enabling logs collection settings and restarting Kibana at cloud deployment.

Build Details:

Version: 8.6.0 BC9
Build: 58832
Commit: 93183bddac40f8a7ee8e566d1651f9f3b586a520

Screenshot:
Untitled
Untitled

Please let us know if we are missing anything.
Thanks

@jen-huang
Copy link
Contributor

Looks like elastic/elastic-agent#1814 is still outstanding so that may lead to this issue still not being fixed, @kpollich do you agree?

@kpollich
Copy link
Member

kpollich commented Jan 5, 2023

@jen-huang - Yes that appears to be the case here.

@gbamparop
Copy link
Contributor

Removing the Team:APM label as it seems that the issue is not related to the APM UI in Kibana. Feel free to ping us if that's not the case.

@gbamparop gbamparop removed the Team:APM All issues that need APM UI Team support label Jan 30, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Jan 30, 2023
@jen-huang jen-huang added Team:Fleet Team label for Observability Data Collection Fleet team and removed QA:Ready for Testing Code is merged and ready for QA to validate needs-team Issues missing a team label labels Feb 1, 2023
@amolnater-qasource
Copy link

Hi Team,

While executing regression for 8.7.0 release, we have found this issue still reproducible at our end.

Build details:
BUILD: 61017
COMMIT: e70452f

Screenshots:
image
8

Impacted Testcase:
Validate 'apm.log' logs with logging level as ERROR are available after enabling Logs and metrics and restarting Kibana from Cloud deployment page.

Please let us know if anything else is required from our end.
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Fixes for quality problems that affect the customer experience impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jen-huang @simitt @gbamparop @kpollich @elasticmachine @manishgupta-qasource @amolnater-qasource