Analytics Database and Build Time Processor

[jjnesbitt]

This PR adds an analytics database, with which arbitrary data can be stored and queried from, as well as a build timing processor, which processes succeeded jobs and their respective phase timings. To accomplish this, several components have been added/configured:

1. A terraform configuration to create the database
2. A very small Django app (no website or server) to be used for model/table migration and management, located at analytics
3. A build timing processor webhook listener, which mimics the structure of the gitlab-error-processor. This http server listens for received webhooks from gitlab, and dispatches a kubernetes job to run the upload_build_timings analytics management command.
4. A staging kustomization to support this configuration in the staging cluster
5. A github workflow for checking the django app migrations (fails if migrations need to be run).

Note: The Terraform changes are already applied to production and running correctly

[jjnesbitt]

Okay, I've backed out all of the changes involving running migrations from the github workflow (will address in a follow up), and have just retained the very simple workflow of checking the migrations. So for the time being, if any migrations are made, they will need to be applied manually.

I've also backed out any changes that existed solely for testing, so this PR is ready to go.

[scottwittenburg]

This is working for me, so I'm approving 😆 But it might make sense for @mvandenburgh to take a look from a kube/terraform perspective as well.

One question, well maybe with two parts: Will this just work when we merge Ryan's timing statistics PR and set up the webhook from spack/spack? Will there be a way to keep triggering it and testing from scott/pipeline-experiments at that point as well?

[jjnesbitt]

One question, well maybe with two parts: Will this just work when we merge Ryan's timing statistics PR and set up the webhook from spack/spack? Will there be a way to keep triggering it and testing from scott/pipeline-experiments at that point as well?

I believe the answer to both is yes. Flux should deploy the required services (might hiccup at first as the images will need to be built), and the webhook makes no distinction as to which project is invoking it.

[mvandenburgh]

I've been disconnected from the timing info stuff so I'll defer to Scott on the specifies of the DB schema and build processor server. But the Terraform for the database and k8s resources look good to me.