Route SNS notifications through a Lambda function

[hannes-ucsc]

In the past few months, the need to filter and route notifications published to the shared monitoring SNS topic has become evident. To solve #5236, for example, we want to suppress certain events originating from personal deployments. For #5043 we will need to filter/suppress AccessDenied events caused by malicious parties scanning the DB CloudFront distributions.

To satisfy these needs, we need to route all notifications through a Lambda function that we can then implement custom filtering logic in. For every event that is not filtered out, the Lambda function will then send an email to the the group. This will remove the need for confirming the subscription of the group email to the topic, which required manual intervention. However, it is not trivial to programmatically and reliably send email due to the various ant-spam protections in place these days.

We should use Amazon SES for sending email. SES requires that the email address in the from field of sent emails is verified or that the domain part of that email address is verified. We should use the Azul domain name of the main deployment as the domain of the sender. Terraform has a resource type for SES-verifying a domain and can provision the required TXT record in the domain's hosted zone.

This ticket is simply about replacing the built-in SNS email subscription with a Lambda function in the indexer app that 1) is called for every notification to the SNS topic and that 2) forwards that notification as an email to the group via SES.

[bvizzier-ucsc]

@hannes-ucsc: "We have managed to whittle down the number of false alarms where things like permissions denied errors to a level where we can remove the threshold (from 12 to 0). The only type of alarm that we are still ignoring is vulnerabilities detected by AWS Inspector (it just sends out too many redundant notifications). We may be missing important critical vulnerabilities until we deal with them every two weeks. To mitigate that we watch out for wide spread critical vulnerabilities through other channels. Because of that we can icebox this feature."