Add thumbnail repsonse time runbooks

[stacimc]

Fixes

Related to #2502 by @sarayourfriend
Related to https://github.com/WordPress/openverse-infrastructure/pull/619

Description

Adds runbooks for thumbnail response time alarms.

Question for reviewers: the content for all of these is identical. I could make a single runbook and update the links in the alarms. The reason I didn't is that it seems possible (although perhaps unlikely) that false positives could occur separately for each of them 🤔

Testing Instructions

Make sure they look alright in the preview!

Checklist

• My pull request has a descriptive title (not a vague title likeUpdate index.md).
• My pull request targets the default branch of the repository (main) or a parent feature branch.
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no visible errors.
• I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[sarayourfriend]

LGTM. Regarding sharing them: I think we could share the anomaly ones, maybe, but I do think it's good to have separate instructions for debugging average vs p99. For example, for average response time, you probably don't need to do anything very special to find "the slow requests", because overall things are slow. For p99, however, as long as average is still okay, it is revealing some kind of edge case where we'd need to query the nginx logs for target response time. I don't know whether that needs to be in the runbook, because it is a different in principle that applies to all average and p99 response time debugging, but it is a notable difference between the two. But yeah, not sure whether that's a difference in the runbook or a difference in some other resource.

The anomaly and threshold alarms also have a difference, I think. Like the threshold alarm runbooks could probably have an instruction to say "check whether it is a one-off spike or persistent" and guide severity along those lines. If we spike to 10 seconds average response time, that's not good, but it also doesn't mean we need to drop everything to debug it if things have gone back down.

That's also just a general principle that could apply to almost every alarm, so I don't know whether that needs to be in the runbooks or, like above, in some other document that they link to. I think this relates somewhat to the level of confidence that something persistently bad is happening based on the alarm, which does also relate to the false alarms you mentioned. Ideally anomaly alarms would be higher confidence that something bad is persistently happening, once we've tuned them, and threshold alarms are kind of more up in the air, could be really bad or could just be a one-off spike that we should look into, but not necessarily with urgency.

Anyway, all of that is just my thoughts on this. I don't have strong recommendations either way, but whatever we come up with, ideally we can apply it consistently between the alarms. It looks like there are some important differences but also important similarities in the principles of these alarms and what they can indicate, I just don't know what the correct or most flexible way of sharing and differentiating between those is.

[github-actions]

Full-stack documentation: https://docs.openverse.org/_preview/3053

Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again.

You can check the GitHub pages deployment action list to see the current status of the deployments.

New files ➕:

• https://docs.openverse.org/_preview/3053/meta/monitoring/runbooks/api_thumbnails_avg_response_time_above_threshold.html
• https://docs.openverse.org/_preview/3053/meta/monitoring/runbooks/api_thumbnails_avg_response_time_anomaly.html
• https://docs.openverse.org/_preview/3053/meta/monitoring/runbooks/api_thumbnails_p99_response_time_above_threshold.html
• https://docs.openverse.org/_preview/3053/meta/monitoring/runbooks/api_thumbnails_p99_response_time_anomaly.html

Changed files 🔄:

• https://docs.openverse.org/_preview/3053/meta/monitoring/runbooks/index.html