Skip to content

Commit

Permalink
Threshold alarms are low severity if not anomalous
Browse files Browse the repository at this point in the history
  • Loading branch information
stacimc committed Sep 26, 2023
1 parent 4af521f commit febaf89
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 12 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,14 @@ Alarm link:

## Severity Guide

Confirm that there is not a total outage of the service. If not, the severity is
likely low. Check for a recent deployment that may have introduced the problem,
and rollback to the previous version. If not, check the request count and
general network activity. If abnormally high, refer to the [traffic analysis run
book][traffic_runbook] to identify and block any malicious traffic.

If the avg response time is not [anomalously high][anomaly_alarm], the severity
is likely low. Check for a recent deployment that may have introduced the
problem, and rollback to the previous version. If not, check the request count
and general network activity. If abnormally high, refer to the [traffic analysis
run book][traffic_runbook] to identify and block any malicious traffic.

[anomaly_alarm]:
https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#alarmsV2:alarm/API+Thumbnails+Production+Average+Response+Time+anomalously+high
[traffic_runbook]:
/meta/monitoring/traffic/runbooks/identifying-and-blocking-traffic-anomalies.md

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,14 @@ Alarm link:

## Severity Guide

Confirm that there is not a total outage of the service. If not, the severity is
likely low. Check for a recent deployment that may have introduced the problem,
and rollback to the previous version. If not, check the request count and
general network activity. If abnormally high, refer to the [traffic analysis run
book][traffic_runbook] to identify and block any malicious traffic.

If the P99 response time is not [anomalously high][anomaly_alarm], the severity
is likely low. Check for a recent deployment that may have introduced the
problem, and rollback to the previous version. If not, check the request count
and general network activity. If abnormally high, refer to the [traffic analysis
run book][traffic_runbook] to identify and block any malicious traffic.

[anomaly_alarm]:
https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#alarmsV2:alarm/API+Thumbnails+Production+P99+Response+Time+anomalously+high
[traffic_runbook]:
/meta/monitoring/traffic/runbooks/identifying-and-blocking-traffic-anomalies.md

Expand Down

0 comments on commit febaf89

Please sign in to comment.