Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terminate Beyla if we can't open a configured Prometheus port #1111

Merged
merged 3 commits into from
Aug 28, 2024

Conversation

grcevski
Copy link
Contributor

Beyla can fail to register the HTTP Prometheus metrics endpoint because of number of reasons. It can be that the port is already opened or we don't have permission to listen on a specific port.

Currently, when we fail to open the Prometheus port the Beyla process prints an error message, but then successfully runs. Successful run can be misleading, since for the end users there is no indication that the process is unhealthy and the error message is buried in the very first few lines of the logs. The symptom is metrics are not scraped, but Beyla is up and running.

This PR causes Beyla to terminate as soon as we fail to register the port. We can't add early validation for this error scenario, because we can't predict all of the possible reasons we may not be able to open the Prometheus port.

@codecov-commenter
Copy link

codecov-commenter commented Aug 27, 2024

Codecov Report

Attention: Patch coverage is 60.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 81.98%. Comparing base (a3ffb2f) to head (b5a83f5).
Report is 1 commits behind head on main.

Files Patch % Lines
pkg/internal/connector/prommgr.go 60.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1111      +/-   ##
==========================================
- Coverage   82.03%   81.98%   -0.06%     
==========================================
  Files         140      140              
  Lines       11514    11517       +3     
==========================================
- Hits         9446     9442       -4     
- Misses       1548     1551       +3     
- Partials      520      524       +4     
Flag Coverage Δ
integration-test 57.10% <20.00%> (-0.17%) ⬇️
k8s-integration-test 59.17% <20.00%> (-0.11%) ⬇️
oats-test 37.06% <0.00%> (-0.01%) ⬇️
unittests 52.33% <60.00%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@mariomac mariomac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch 🚀

@grcevski grcevski merged commit 1fb43b8 into grafana:main Aug 28, 2024
6 checks passed
@grcevski grcevski deleted the end_beyla_on_bad_prom_port branch August 28, 2024 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants