3.0 Docker Build Improvements

[humphd]

I want to start a discussion about how to improve our Docker infrastructure. We have a very basic setup at the moment, and there are bunch of ways that we could make it better.

First, I'd like to see us build and push our Docker images during CI runs. Whenever we pass lint, tests, etc. we should build the Docker images and then push those to a registry. We already do part of this: our e2e tests require us to build the containers so that we can run the microservices. However, after we build them, we aren't caching anything (i.e., we don't push the built images to a registry). We should be doing that so that subsequent updates to the PR, or other PRs that use parts of the system that haven't changed, can re-use the existing cached versions (i.e., download from the registry vs. rebuild).

Second, we should be smarter about what we build. A PR will only change some of the monorepo, not everything. It's going to be pretty rare for us to need to rebuild the entire world for a PR. We should be able to use existing images that we get from cache/download from a registry. Then we can just build the parts that have changed, and upload those to the registry when we're done. I am hoping that Turbo Repo or Rushjs (see #1778) can help us figure this out.

Third, our local development, gitpod development, and staging/production should all leverage the fact that we'll have built versions of the system already sitting in a registry. We probably don't need to build anything, and can just pull the images and run them. We'll obviously have times where we want to build a local version of a service, while we make changes to it and test. But the rest of the services could be run from existing images pulled from a repo. This should make gitpod go much faster, since it only needs to download images vs. build them from scratch.

Fourth, we need to do a better job of reducing the size of our containers. We should be using multi-layer builds in all Dockerfiles, and do things like separate our devDeps from our production deps, use smaller base images.

Fifth, we need to define some targets to make it easier to debug our containers. A dev type target layer could run our containers in debug mode, and expose a debugger, use pretty logging, etc. to make it easier to work with a single container via the debugger.

Sixth, we should figure out which registry to use for our needs. We could put things in the GitHub Container Registry (see #2264), or use Docker Hub, or even run our own builders on our staging/production box and host our own private registry, see https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners. It might be cool to do all these builds on our own infrastructure! It might also be more work and pain than we need, I'm not sure.

I'm sure there are lots more things we can do, but before I started to file issues and set people loose, I wanted to get some initial feedback on our approach.

cc @raygervais, @manekenpix