[ws-manager] Wait for workspace pod to be ready

[sagor999]

Description

Make sure to wait for container to start up, in case of out of memory or other errors.

Related Issue(s)

Fixes #8253

How to test

Spin up new cluster in workspace preview env:
./new-vm.sh -v aledbf-wait.10 -z us-west1-c
then try to start up workspaces in it.

Release Notes

Improve handling of "Out of Memory" error when starting up workspaces

Documentation

[sagor999]

/werft with-clean-slate-deployment

👎 unknown command: with-clean-slate-deployment
Use /werft help to list the available commands

[sagor999]

/werft help

👍 You can interact with werft using: /werft command <args>.
Available commands are:

• /werft run [annotation=value] which starts a new werft job from this context.
  You can optionally pass multiple whitespace-separated annotations.
• /werft help displays this help

[sagor999]

/werft run

👍 started the job as gitpod-build-aledbf-wait.10

[csweichel]

This would help with the problem we're trying to solve, but most likely deteriorate the feedback mechanism towards the user by all but removing the Pending phase. I.e. now, users would see a very long "preparing workspace" phase until they finally get redirected.

How much of a concern that is is something we'd see in practice.
Certainly this is another bit of motivation to move towards a CRD based solution.

[sagor999]

I reduced exponential backoff factor, as otherwise in worst case scenario it would have been waiting for hours there to retry.

[kylos101]

Would it make sense to leverage the cap property for backoff? I believe it's intent is to ensure the duration does't exceed a certain time limit.

[sagor999]

Yeah, we could do that as well. What would be a reasonable cap? 5 min?

[kylos101]

I think 5 minutes will do.

We can always bring it down if we need, but this way it's a predictable experience.

W/o accounting for sleep (and jitter), that would get users to iteration 8 (but not beyond) here: https://go.dev/play/p/dtUxS6TwTrw

[sagor999]

Continue working in new PR (due to werft issues in this one): #8289