Epic: [Self-Hosted] Towards an independent installation job

[corneliusludmann]

Current State

In the current installation process, users configuring the Gitpod instance in the KOT UI. KOTS starts a Kubernetes job based on the installer image that runs a bash script where the values of the KOTS UI configuration are replaced by KOTS directly in the bash script. The job prepares the installer config and runs the installer to install Gitpod.

We have several dedicated images specified in KOTS to run preflight checks and gather support bundle data. These are not designed to run independently of KOTS.

This approach has the following drawbacks:

• The bash script of the installer (as job description YAML value) is rather poorly maintainable and testable.
• With the installation job and the preflight checks, we are pretty heavily dependent on KOTS. For example, we cannot re-run the installer when we detect a secret change, etc.

Objectives

• Bringing more responsibilities back to the installer image. Reduce dependence on KOTS.
• Making the installer job and the preflight checks better maintainable and testable (e.g think about having the KOTS user config as input and the resulting Gitpod config as golden file).
• Having more control over the installation / upgrade process (e.g. migration path from internal db/storage/registry to external db/storage/registry).
• Pave the path to other config options besides the KOTS UI.

Proposed Solution and Tasks

• Generate installer config inside KOTS #12152

• Make the remaining bash script part of the installer image instead of the KOTS job template. #12196

• Move Gitpod-related preflight checks to the installer image so that they can be executed independently of KOTS.
  Provide an preflight check entry point that allows running a single preflight check or all preflight checks. Use the installer image for all preflight checks in KOTS.

• Move Gitpod-related support bundle functions to the installer image and call them from KOTS.

• Add post-installation tests to the installer image.
  These can be executed by customers to verify an installation and can also be executed by the installer job after the installation. Maybe we have different checks like quick and extensive.

• Move the installer job to a dedicated and KOTS-independent container following the operator pattern.
  KOTS updates the Gitpod config and the Gitpod operator picks up the config on changes.

• Minimize the KOTS config to the minimum needed for the initial installation of Gitpod (mainly ingress like domain, certs, etc.). Build an admin dashboard that allows configuring/migrating everything else afterwards.
  This will allow us to counter the limitations we currently experience with the KOTS UI. With an admin dashboard, we have more freedom to build the config surface we need. (see also [1, 2, 3], all internal)

• For every step, we need to maintain backward compatibility or have a path to migrate customers.

• Remark: Instead of having everything in the installer image, we could also think about having a dedicated installer, preflight, post-installation check, and support bundle image.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.