Epic: Get rid of OTS (One-Time Secret)

[csweichel]

Summary

The one-time secret (OTS) mechanism is used to deliver secrets to the workspace cluster. It's not location-aware which breaks workspace startup across regions.

Context

The one-time secret (OTS) mechanism is used to deliver secrets to the workspace cluster. During workspace startup, server will create up to three OTS:

• one for the SCM token
• one for the Gitpod token
• potentially one for environment variables

A one-time secret is stored in the database and identified by a UUID. Using this UUID it can be downloaded once, after which it's removed from the database. When the OTS is created, server produces a URL from which the OTS can be downloaded. This URL is not location-specific, but uses the load balancer (i.e. gitpod.io/...).

Because the different regions use different databases which are synchronised using db-sync, and because the OTS URL is not region-aware, a workspace created in another region will race db-sync. This can lead to workspace startup failure.

Value

Removing OTS will

• reduce failure modes (see Failed to download OTS in US cluster (possibly happens for prebuilds, only) #8096)
• enable cross-region prebuilds (see Run prebuilds in the storage-bucket's region #6650)
• reduce complexity in webapp

Acceptance Criteria

This work is complete when

• there's a secure way to keep secrets on the workspace side (Kubernetes secrets qualify here)
• the OTS mechanism is no longer in use for shipping secrets
• the OTS mechanism has been removed from the code-base

Measurement

We are successful here when there's no loss of functionality, and no more need for the OTS mechansim.

Tasks

• Add "secret" support to ws-manager, where a StartWorkspace request can carry named secrets
• Ship the SCM token as named secret and pass it to ws-daemon during InitWorkspace
• Ship the Gitpod token as named secret and pass it as environment variable to supervisor
• Ship the user's environment variables as named secrets and pass as environment variables to the workspace
• Unsupported initializer error opening snapshot in GItpod (internal) preview environment (breaks without OTS) #12554
• [Get rid of OTS] Add an integration test for protected_secrets feature flag #11318
• [Get rid of OTS] Load testing #13490
• [installer] Support enabling protected secrets #13484
• https://github.com/gitpod-io/ops/issues/5608
• Enable protected secrets by default in the installer for self-hosted #13632
• Enable protected secrets by default in the saas without using configcat #13633
• Remove old OTS code #13634

[atduarte]

Added to workspace teams' project, even though it's shared work between WebApp and Workspace, to avoid this being forgotten.

[geropl]

@atduarte Added for Team WebApp as well! Please notify when you have the interface changes (w/o impl, maybe). Happy to jump on the changes on our end.

[geropl]

@kylos101 Should we sync on the current status here? I feel we're pretty close. 👍

[csweichel]

Next step IMHO is to connect the protected_secrets workspace feature flag to ConfigCat and roll this out slowly.

[atduarte]

@csweichel what about the integration test?

[kylos101]

Added: #12554 to the epic

[kylos101]

@atduarte this is In-Validation now.

@Furisto I see that the protected_secrets flag still exists in ConfigCat. Can we remove now?

[easyCZ]

There still seem to be OTS implementation on Server, and it's also being called/used - metrics

[Furisto]

@easyCZ @kylos101 I believe @geropl wanted to remove it.

[easyCZ]

From an internal conversation it looks like the other client which is generating traffic against the OTS endpoints are Desktop IDEs.

@loujaybee @akosyakov Would you be able to validate if this is indeed the case and the Desktop IDEs use the OTS? What would be the effort to remove OTS from them so that we can remove the endpoints entirely?

[akosyakov]

@kylos101 @csweichel Does it mean we can remove SUPERVISOR_ENVVAR_OTS now?

Also

gitpod/components/supervisor/pkg/supervisor/config.go

Lines 348 to 365 in 09642ab

	if tks[i].TokenOTS == "" {
	continue
	}
	resp, err := client.Get(tks[i].TokenOTS)
	if err != nil {
	return nil, xerrors.Errorf("cannot download token OTS: %w", err)
	}
	if resp.StatusCode != http.StatusOK {
	return nil, xerrors.Errorf("cannot download token OTS: %d (%s)", resp.StatusCode, resp.Status)
	}
	tkn, err := io.ReadAll(resp.Body)
	resp.Body.Close()
	if err != nil {
	return nil, xerrors.Errorf("cannot download token OTS: %w", err)
	}
	tks[i].Token = string(tkn)
	}

is not used anymore? i.e 34e5648 seems to remove it

[kylos101]

@akosyakov maybe? I added a conversation here, socializing that it would help to get more data about the client.

[kylos101]

@akosyakov @easyCZ inboxed this epic to IDE and WebApp, as there appears to be some additional cleanup needed to fully remove OTS, and set the status for workspace back to In-Validation.

[geropl]

@akosyakov @kylos101 @atduarte FYI: I thought we can close this issue, but it turns out here is one thing left: AdditionalContentContext. 😬

@kylos101 Let's sync how we want to go about this one.

[geropl]

After discussing wiht @kylos101 : Let's close this one and continue the tiny bit left here: #16481.