[prebuilds] Failed Prebuilds without Snapshot are marked as 'available'

[svenefftinge]

There are many failing prebuilds without a snapshot that are still marked as available and therefore used for new workspace starts.

SELECT * FROM d_b_prebuilt_workspace WHERE snapshot='' and state = 'available' order by creationTime DESC

• We need to figure out why these workspaces fail (many have an error message (Out of Memory) but some don't).
• Make sure they are marked as failed, so they are not considered when starting a workspace.

[svenefftinge]

I'm looking into filtering those prebuilds out.

[geropl]

Discovered yesterday here.

@svenefftinge I don#t think it's possible. Quick sync?

[geropl]

We figured out there are two issues here:

• @svenefftinge figured that we're failing to check the prebuild.error state when trying to determine whether a snapshot is available or not
• there is a problem with failing prebuilds overriding their failed state (Failed to download OTS in US cluster (possibly happens for prebuilds, only) #8096)

[geropl]

Removing the workspace label as the interpretation is webapps responsibility.