Investigate high priority OpenSearch alerts being sent to pager duty

[tom-j-smith]

Background

Currently, we do not have a way of sending high-priority alerts from OpenSearh to PagerDuty. This issue is to investigate and create a repeatable way for us to set it so that high-priority OpenSearch alerts trigger PagerDuty when alerting.

This guide was being used to try and set up the connection but encountered an issue when trying to create a webhook channel in OpenSearch using terraform as it does not appear that you cannot configure webhook headers through json in terraform.
It may be possible to pass the Integration Key as routing_key in the payload, but then we will not be able to use some feature like alert testing to ensure the channel is configured.

A service has been created on PagerDuty for high priority OpenSearch alerts - https://moj-digital-tools.pagerduty.com/service-directory/PG9BY5U/integrations
The service will also require someone with Slack workspace admin privileges to set up the slack extensions/add-on for the service so messages will come through to the high-priority-alarms channel

The OpenSearch alert module will need to be updated to allow pre-configured channels to be passed in so other HighPriority alerts can use the same PagerDuty route rather than setting up and configuring a new channel for each alert.

[tom-j-smith]

One potential solution would be to configure the channel manually in OpenSearch to ensure the webhook headers are configured, test the channel connects to the PagerDuty service, then update the OpenSearch alert module to take the channel created for the alert.
The module will also need to be altered to allow a different payload to be passed to ensure PagerDuty triggers correctly.