[bridge] Ensure we do not override "failed" instance states (affects prebuilds!)

[geropl]

Currently we're seeing that failed prebuild instances' status is override by newer instance updates lagging that state (ref).

While this is a bug, we should be failsafe here, and basically:

• never overwrite the failed condition
• add additional logging for unexpected updates
• accept and forward ws-manager version field

[geropl]

@easyCZ This was the issue I tried to refer to: it's relevant for prebuild reliability, but also for other workspaces

The version field won't help with this concrete problem (because ws-manager happily gives out a higher version for earlier, already gone-thorough phases). But it's nice to forward anyway, and should help with reconnect issues in ws-manager-bridge (and potentially, all downstream channels).

[easyCZ]

Thanks! I've also cut #9107.

The version field won't help with this concrete problem

Are you referring to the status_version field or is this some other version field?

[geropl]

Are you referring to the status_version field or is this some other version field?

No, that's the field I meant! 💯

[easyCZ]

The prebuild part of this issue has now landed in #9116. I am going to first validate the effects of this change on Prebuilds before making the corresponding change for Workspace Instance updates.

Leaving this open as we still need to make the same change for WorkspaceInstanceStatus updates.

[easyCZ]

Using prior work to detect stale events, we can inspect if stale events are a problem for WorkspaceInstance updates also. In practice, we haven't observed stale event updates for prebuilds. The code-flow is as follows:

1. Receive a workspace instance update
2. Process it, and re-map to a db-record so that we can communicate the WorkspaceInstance status
3. Invoke a controller loop for prebuilds. If the update is not for a Prebuild workspace instance, we ignore it.

This means that prebuild updates are a reasonable (but complete) baseline for "stale updates".

[geropl]

We're fixing this in two steps:

1. roll out [bridge] Add log.error in case we are about to override a previous "failed" condition #10900 and observe logs for ~1 week to see if how often we're hitting that situation
2. based on the data we gathered, decide how to proceed with this: Ideally by ignoring destructive updates to the "failed" condition 👍

[geropl]

It turns out we cannot simply disable this, because there is a host of different cases where we seem to rely on it. We have to investigate the different root causes on workspace side before we can move forward here.

How to investigate:

1. pull ws-manager-bridge logs with this query:

"We received an empty \"failed\" condition overriding an existing one"
resource.labels.container_name="ws-manager-bridge"

2. Download as CSV or similar, and turn into a list of instanceIds
3. Use this DB query:

    SELECT wsi.id, wsi.workspaceId, ws.type, pws.state, pws.error, wsi.phasePersisted, wsi.status->"$.conditions.failed", wsi.status->"$.conditions.headlessTaskFailed", wsi.status->"$.conditions.timeout"
	FROM d_b_workspace AS ws
	JOIN d_b_workspace_instance AS wsi
		ON wsi.workspaceId = ws.id
	LEFT JOIN d_b_prebuilt_workspace AS pws
		ON ws.id = pws.buildWorkspaceId
	WHERE wsi.id IN ( <INSTANCE_IDS>);

I'm closing this one for now.