workspaces which cannot be scheduled to a cluster get stuck in Pending

[kylos101]

Bug description

Workspaces which cannot be scheduled to a cluster get stuck in Pending.

Steps to reproduce

In a preview environment, kubectl cordon the node. Try starting a workspace. It'll stay stuck in Pending for-ev-er!

Workspace affected

n/a

Expected behavior

The workspace should fail, and land in a permanent Failed phase

Example repository

n/a

Anything else?

This becomes a problem when you're trying to delete a cluster, because ws-manager-bridge reports that there are still active workspaces in the database for the cluster, even though there are no workspace pods in the governed workspace cluster.

This prevented the delete of us53 (which was empty, and had no workspace clusters), until we ran the following queries:

-- find pending workspaces that aren't really pending
SELECT dbwi.id
FROM gitpod.d_b_workspace dbw  
inner join gitpod.d_b_workspace_instance dbwi 
on dbw.id = dbwi.workspaceId  
WHERE 
dbwi.phasePersisted not in ('stopped','failed') 
# change to match a region you're interested in
and dbwi.region = 'us53'
ORDER by dbwi.creationTime asc
;

-- force stop the pending workspaces
UPDATE gitpod.d_b_workspace_instance AS wsi
SET
			wsi.stoppedTime = IF(wsi.startedTime = '', wsi.creationTime, wsi.startedTime),
			wsi.stoppingTime = IF(wsi.startedTime = '', wsi.creationTime, wsi.startedTime),
			wsi.phasePersisted = 'stopped',			
			wsi.status = '{"phase": "stopped", "conditions": {}}'
WHERE wsi.id IN ('fb52614b-927b-4412-b583-dd3cc7bc3087',
'04eedb5b-1f35-4e4b-a8b3-56ea13c0f2dd',
'9b3d3928-59dd-46f0-9876-cc2bab905a2e',
'261bbde5-100f-4a19-8085-125ef82a0f0a',
'7878cc2a-544e-4f24-82ee-62a24c4f9c34',
'5dda17bf-1fe9-4cc6-b5ae-82aa6603049c',
'ad40bce6-a74c-47a9-b9a4-d9490afbfd41',
'028d826e-49b2-45ce-b829-8b4f693e93d3',
'99b74f2b-0a49-45c6-9c59-c4339a6dcb48',
'3695365a-0b4c-4df2-96f3-700598fe3baf')
;

• Delete this page once resolved

[kylos101]

note: this may not be a problem with ws-manager, in hindsight, it might be a problem on the webapp side, but needs work to confirm.

[jenting]

Similar to #11673.

[utam0k]

@jenting Has this issue already been solved by #11673?

Similar to #11673.

[jenting]

@jenting Has this issue already been solved by #11673?

Similar to #11673.

No, the #11673 is to make a retry within 30 mins (we wait for the workspace pod to be running with 7 mins timeout).

The good news is the workspace pod be removed after 30 mins, and the bad news is the dashboard displays the workspace in a Pending state.

[kylos101]

I believe Pending is a phase that indicates webapp has not selected a cluster (yet) for the workspace to land on. @geropl reassigning for now. wdyt?

[kylos101]

👋 @geropl hey, we had to do some workspace clean-up in the database last week when decommissioning to gen61, as we shifted to gen62. May we ask for your help in scheduling this for the next couple weeks?

[geropl]

@kylos101 Done, set to next week.

[geropl]

Linked to: #6770