Refactor application health monitoring #82
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Application health is needed for two distinct reasons: 1. Sending alerts to a monitoring webhook for setup errors. Relatedly, sending heartbeat messages to the webhook when healthy. 2. The HTTP health probe. Needed so that the orchestration environment (kubernetes or whatever) can kill the pod when unhealthy Several Snowplow apps each implement their own logic of when to toggle the health probe and when to send an alert. This PR consolidates that logic into one place. After this PR, the application code just needs to call methods on the AppHealth class. This lib then manages webhook events and health probe, based on the current status of the AppHealth.
istreeter
force-pushed
the
feature/application-health-monitoring
branch
from
4b3a25f to
b29df23
Compare
pondzix
reviewed
Aug 14, 2024
modules/runtime-common/src/main/scala/com/snowplowanalytics/snowplow/runtime/AppHealth.scala
Show resolved
Hide resolved
pondzix
reviewed
Aug 14, 2024
modules/runtime-common/src/main/scala/com/snowplowanalytics/snowplow/runtime/AppHealth.scala
Show resolved
Hide resolved
pondzix
reviewed
Aug 14, 2024
|
|
||
| private[runtime] sealed trait SetupStatus[+Alert] | ||
| private[runtime] object SetupStatus { | ||
| case object AwaitingHealth extends SetupStatus[Nothing] |
There was a problem hiding this comment.
It only affects what is printed in logs, right? In health probe it still means 503 unhealthy?
There was a problem hiding this comment.
- When we first transition to
Healthythen we send aheartbeatevent to the webhook - When we first transition to
Unhealthythen we send aalertevent to the webhook
Therefore we need a type which is neither Healthy nor Unhealthy so we detect the transition. Hence AwaitingHealth.
pondzix
reviewed
Aug 14, 2024
modules/runtime-common/src/main/scala/com/snowplowanalytics/snowplow/runtime/HealthProbe.scala
Outdated
Show resolved
Hide resolved
pondzix
reviewed
Aug 14, 2024
modules/runtime-common/src/main/scala/com/snowplowanalytics/snowplow/runtime/Webhook.scala
Show resolved
Hide resolved
pondzix
reviewed
Aug 14, 2024
modules/runtime-common/src/main/scala/com/snowplowanalytics/snowplow/runtime/Webhook.scala
Show resolved
Hide resolved
istreeter
force-pushed
the
feature/application-health-monitoring
branch
from
2011180 to
f261713
Compare
istreeter
force-pushed
the
feature/application-health-monitoring
branch
from
4b492d9 to
0e72774
Compare
istreeter
force-pushed
the
feature/application-health-monitoring
branch
from
1135ccf to
8d43c85
Compare
istreeter
force-pushed
the
feature/application-health-monitoring
branch
from
8d43c85 to
a139a53
Compare
This was referenced Aug 21, 2024
Merged
Merged
pondzix
approved these changes
Aug 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Application health is needed for two distinct reasons:
Several Snowplow apps each implement their own logic of when to toggle the health probe and when to send an alert. This PR consolidates that logic into one place.
After this PR, the application code just needs to call methods on the AppHealth class. This lib then manages webhook events and health probe, based on the current status of the AppHealth.