Update our Incident Response process and Slack workflow

[dfitchett]

Description

As the on-call engineer for the VRO platform team, I need as many tasks automated as possible to reduce the risk of missing manual follow-ups. Specifically, I want to update our existing Incident Response process and Slack workflow to automatically incorporate the new step of reaching out to partner teams to assess how our downed services impacted their processes and, ultimately, the veterans we serve. This will ensure that we are documenting the effects of failures on the VRO platform and reduce the potential for silent failures.

AC

• add a new task in the Incident Report Slack workflow to gather impact metrics from partner teams when a high severity incident (as part of or new message sent to the #benefits-vro-on-call channel)
  • specify in the instructions that the step should be done for SEV 1 and SEV 2 incidents
  • specify a timebox or SLA for completing the task
• update Incident Response wiki page with the new step clearly stated
  • add any other updates based on latest changes regarding silent failures and deployment workflow (optional/as time allows)
• create an impact metric table on the incident response page where these metrics can be populated and shared easily
• share with VRO team
• inform partners of the change so they know what to expect - send an update message #benefits-vro-support in the thread of the Incident Report workflow launch- talk to Derek about what alerts already exist for BIP and BGS

Resources

Reference Incidents Epic

[gabezurita]

Note: I'm on this issue as part of my onboarding to help learn about the VRO on-call process 😄

[gabezurita]

Some docs:
https://sre.google/workbook/incident-response/
https://sre.google/sre-book/managing-incidents/
https://sre.google/sre-book/incident-document/
https://github.com/department-of-veterans-affairs/abd-vro/wiki/On-Call-Responsibilities

[bianca-rivera]

made more significant updates and formatting changes; added new AC bullet to share with VRO team before sending message to partner teams of latest update regarding impact metrics and include highlighting any other substantial changes

[gabezurita]

Excellent job, @bianca-rivera !!!