Reduce chaos with better end-to-end testing

[emmatyping]

As Sentry grows, having automated tests catch issues between self-hosted and other components becomes more important.

We do already have minimal testing to catch issues so that we can ship a great self-hosted product and not block other teams, but we recently found that our end-to-end (e2e) testing of self-hosted is lacking in some areas.

1. First, each repo that runs e2e tests (sentry, snuba, relay) has its own configuration, which means changes to self-hosted e2e tests need to made in all three repositories. We recently broke builds because our e2e tests passed in self-hosted, but failed in other repos. Therefore, we should standardize this configuration somewhere so we only have one source of truth.

2. In addition, our e2e tests are currently the slowest in getsentry/sentry, which is slowing down CI for many people. The devinfra team has pointed out this is a pain point for Sentry development velocity. We can accelerate the installations speed of self-hosted in our e2e tests (and for our users!) by reducing the number of containers we spin up.

3. Finally, a snuba change recently broke self-hosted because we weren't testing enough of the Sentry pipeline in our tests. We have several ideas of how to improve this, including testing parts of the error pipeline that further test kafka and making sure that the test instance is healthy after we run tests.

4. Simplify e2e tests. Currently there are a lot of scripts and moving parts in the e2e tests. We should simplify them if possible.

To Do

• Standardize e2e testing self-hosted#1756
• Make e2e tests more thorough self-hosted#1761
• Check health of containers in e2e tests self-hosted#1762
• Announce on Vanguard
• Reduce number of containers in use self-hosted#1763 - Stretch - we are not doing

┆Issue is synchronized with this Jira Epic by Unito

[chadwhitacre]

Moving here from #40 (comment):

I can provide some historical context on self-hosted e2e tests:

1. Around that time there were many changes happening with database migrations and event ingestion so it was very easy to break something with reliance on a new service or just changing how events were ingested. This may not be the case anymore.
2. The releases are named nightly but they are actually per SHA (I know this threw you off multiple times @chadwhitacre 😅 ) to ensure we can get a release out at any time we want. If you disable the entire job for front-end only changes, you also lose the ability to ship those changes which is probably quite less than ideal. What you can do instead would be to build the docker image in a non-blocking job and push it without the setup phase.
3. The most time consuming parts of the build were the actual building part (with the frontend build taking the majority of that time) and then the initial setup with the database migrations. If you can:
   1. Find a way to re-use built front-end packages across SaaS builds and self-hosted, you'd save that time
   2. Compact database migrations again with a hard-stop or have a DB snapshot volume container so you don't have to go through those steps every time, unless there's a DB migration you'd save A TON of time
4. You should probably add more coverage for self-hosted regarding symbolication (I wanted to do this way long ago) and performance events
5. There's this one piece of front-end that is not well-tested: self-hosted welcome screen. With proper e2e testing (selenium?) that can be less of a concern
6. Ideally, at least single-tenant would be using the same images or image base with self-hosted to save some time and resources there. There were efforts around this and it was close but then the work stalled.

[emmatyping]

I might want to tack on a stretch goal of dealing with some self-hosted integration test instability, e.g. getsentry/self-hosted#1777 and several times in getsentry/self-hosted#1775 the integration tests flake.

[chadwhitacre]

Mostly done. Cutting bait to focus on higher priorities for the new quarter.