[installer]: deprecate static message bus password and replace with secret

[mrsimonemms]

Description

The experimental config had a staticMessagebusPassword key which violates our rules on configuration, namely "sensitive data must be stored in a secret".

To that end, this deprecates the old behaviour and replaces with a new messageBus object in the config. I took the decision to put it in a new object because, as a dependency, it's plausible that we'll need to have additional configuration options in the future (similar to registry/db/storage).

The complexity with this ticket is that RabbitMQ load definitions secret which requires the password. As there's no documentation on how to achieve that, the load definition secret stores the password as %PASSWORD%, I've created an init container which reads the load definition and password secrets and replaces %PASSWORD% with the actual password and then saves it to the emptyDir which is what's actually used by the RabbitMQ resource.

If no password is given or the old experimental config is used, this stores the given password in a secret so RabbitMQ can read it.

There is no MapValue for the experimental config that can be done because it's too complex to create a secret for a time-limited deprecated function. The deprecated message doesn't display the password because I don't want to write credentials into logs, even if the password is easily readable.

Related Issue(s)

Fixes #15899

How to test

Install a cluster with the following config. To check that the new secret has been promulgated, run kubectl exec -it -n gitpod deployments/server -- env | grep MESSAGEBUS_PASSWORD

1. No password specified (default)

No special config required

2. Deprecated experimental config

experimental:
  common:
    staticMessagebusPassword: some-password

3. New password secret

kubectl create secret generic -n gitpod --from-literal=rabbitmq-password=secret-password message-bus-password

messageBus:
  credentials:
    kind: secret
    name: message-bus-password

Release Notes

[installer]: deprecate static message bus password and replace with secret 

Documentation

Build Options:

• /werft with-github-actions
  Experimental feature to run the build with GitHub Actions (and not in Werft).
• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh

[werft-gitpod-dev-com]

started the job as gitpod-build-sje-deprecate-messagebus-pass.8 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-sje-deprecate-messagebus-pass.9 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-sje-deprecate-messagebus-pass.10 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-sje-deprecate-messagebus-pass.11 because the annotations in the pull request description changed
(with .werft/ from main)

[mrsimonemms]

/hold so I can rebase the golden files before final merge

[mrsimonemms]

/werft run

👍 started the job as gitpod-build-sje-deprecate-messagebus-pass.13
(with .werft/ from main)

[Pothulapati]

LGTM apart from the above comment! Init container does seem to be a bit complicated but seems to be the only way without changing too much logic. Hoping we can move to using extraConfiguration option in the future so that we don't have to pass the password at two different places 🤔

/hold Just in case

[geropl]

I tried to review the code, but it was a bit too large for me to judge with confidence.
Now looking into comparing the rendered diff... 👀

[mrsimonemms]

@geropl yes, it's a bigger than I'd have hoped. Happy to sync with you if you need it

[geropl]

Reviewed the diff, and chatted with Simon.
LGTM!

[mrsimonemms]

/unhold

Rebased from main and updated golden files

[mrsimonemms]

/hold

[mrsimonemms]

/unhold