Validate shortened doc links

[debadair]

The Elastic Health API returns information that includes links to Troubleshooting topics in the documentation. They're using the link shortening service for these links so that they are easier to manage. There are other places where we are embedding links to the docs (deprecation warnings) and using shortened links. We need to routinely test these to ensure the links we're embedding in output don't 404.

[gtback]

Conceptually, I think this comes in three steps:

1. Get a list of short-links we need to test (technically, we could do skip this step and go straight to step 3 if they're already links to the documentation)
2. Look up the original link for each shortened link (we can do this by just making the web request, or we could directly query the Elastic cluster that stores this data).
3. Ensure all "docs-looking" links are valid.

Step 3 can use the existing patterns for Kibana links.
Step 2 is probably up to me (I'll need to think about whether making the requests would slow down the builds too much, or skew metrics, or anything like that; @JoshMock, do you have any thoughts here?)
I'll need help with Step 1: more information about where the links are saved in code, and how we can extract them.

[JoshMock]

As far as hitting the shortener service, HEAD requests work: curl --head https://ela.st/infra-aon

If you get a 200-300 level response and a valid Location header (which you could then follow if you so chose to ensure the final destination actually exists) you know the link is good.

The only other thing I'll say is that recursive link validation tools sound easy to build, but there be dragons there. I wrote one at my previous job and saw a surprising amount of false negatives because some servers deny HEAD requests, or behave in other non-standard ways. Sometimes this is intentional bot-blocking, and sometimes it's just accidental bugs, but it's always annoying. 😄

[gtback]

Thanks, @JoshMock !

In this case, I was imagining just finding all ela.st links, and making a request and getting the Location header for the redirect. Then, if the URL starts with https://www.elastic.co/guide/, take the part after that and ensure it exists under: https://github.com/elastic/built-docs/tree/master/html (this is how all of the existing link checking works). My plan was to not make any requests for anything under www.elastic.co (or indeed, any domains other than ela.st).

I should have thought to use a HEAD request, which means you can filter out a "click" like this based on http.request.method, as well as possibly using a custom User Agent.