[PVC] investigate we could avoid the cannot mount workspace error

[jenting]

Bug description

The user hits the cannot mount workspace error. Not sure if it's an intermediate state or not.
If it's an intermediate state, we could ignore it just like we ignore the error here.
If not, see if we could recreate the PVC and remount rather than report the error to the user.

It causes the cannot mount workspace error comes first and then makes the issue happens.

https://gitpod.slack.com/archives/C0201TXMV54/p1666253166702419

Steps to reproduce

We don't know.

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

#7901

[kylos101]

@sagor999 this error happened in us72 and eu71.

Logs, edit: this error was logged for 3 workspace instances only.

However, what's really weird about this is, one of the instances that landed in us72 also landed (with the same ID) in us71.

Logs for one instance

Looping in @geropl , because I believe this is not supposed to happen.

[sagor999]

However, what's really weird about this is, one of the instances that landed in us72 also landed (with the same ID) in us71.

maybe because each StartWorkspace request selects cluster prior to making the call? And due to traffic shift in the process, it landed on a different cluster.

[kylos101]

@geropl I'll create a separate issue to socialize the below:

I would have expected startWorkspace requests to different clusters to have used different instance IDs. I could also be wrong.

I recall that ws-manager-api retrys, it seems like it is initialized to zero retry attempts, though in the logs it looks like it is retrying.

For example, the first one was to us71, and 7 seconds later the second one was to us72. 🤔

There are some logged warnings for server where it cannot connect and retries.

[svenefftinge]

FWIW I just had the "cannot mount workspace" error when starting gitpodio-gitpod-zjkqtz6c188.
Giving it another try worked.

[jenting]

Two options:

• We could consider treating the"cannot mount workspace" error as an intermediate state in ws-manager, just like what we did in PR Skip "MountVolume.MountDevice failed for volume" #13760
• We could delete the workspace pod and recreate a new workspace pod once we hit the "cannot mount workspace" error, just like what we did in PR [ws-manager] Handle Pending pod due to Unschedulable reason #11690

[sagor999]

@jenting lets try with first option first (We could consider treating the"cannot mount workspace" error as an intermediate state in ws-manager) and if that not going to fix it, then we can go with option two of recreating the pod.
Since it is hard to repro and verify this issue, we would have to deploy option 1 first and see if that would resolve this problem.