Investigate Per-Service Monitoring Progress

[dfitchett]

User story

As a VRO engineer, I want understand the purpose and current monitoring capabilities of each app so that I can track metrics to see when it is behaving as expected and be alerted to when it's not behaving as expected.

Background / What You Need to Know

Related ticket: #3472

AC

• Silent Failure mural board has been updated for each service regarding the following topics (to show that they each have been completed):
  • End Point error handling and reporting;
  • Health Check monitoring;
  • Alerting and Notifications;
  • Monitoring and Escalation Policy (Incident Response Policy);
  • Datadog Metric Tagging Standards;
  • Creating Remediation / Prevention tickets
• Ticket/s for remediation has/have been created:
  • Alerting rabbitmq metrics in datadog  #3482
  • Application Alerting for BIP #3017
  • Add monitor tags to svc-bip-api #3528
  • Fix bug in BGS returning wrong values on downstream BGS exception #3526
  • svc-bgs-api: Failed Datadog requests delay responses being sent to requestor #3536
  • Final Fix to svc-bgs-api K8s readiness probe (2417) #2539
  • Application Alerting for BGS #3018
  • svc-bip-api: Detect dropped RabbitMQ connection #3509
  • svc-bgs-api: Detect dropped RabbitMQ connection #3530
• Updated mural board is shared with the team
• Work with Product / Delivery if additional support is needed to create ticket/s
• Can pair with other team members who want to work on this

Resources

VRO Dependency Diagram (link)

Silent Failures Mural board (link)

Claim Status Tool Evidence Submissions Silent Failures Diagram

working draft for VRO

Enable Observability Epic:
#2335

[dfitchett]

Only need to do this for rabbitmq, svc-bip-api, and svc-bgs-api