DevWorkspace start occasionally fails due to FailedMount

[amisevsk]

Description

Occasionally, starting what should be a valid DevWorkspace will fail due to the controller detecting a FailedMount event. Workspace startup is failed with message

Detected unrecoverable event FailedMount: Unable to attach or mount volumes: unmounted volumes=[claim-devworkspace], unattached volumes=[]: timed out waiting for condition

How To Reproduce

Reproduces rarely when starting DevWorkspaces. No way to consistently trigger

Expected behavior

DWO should not fail workspaces for transient failure events. More fine-grained detection may be needed.

Additional context

[dkwon17]

I have also had a similar issue when starting a DevWorkspace, but I'm not sure how to reproduce the issue.

Right after I got the error, I tried starting the DevWorkspace 2-3 more times, but still had the Detected unrecoverable event FailedMount: error.

Output from the Logs tab: https://gist.github.com/dkwon17/a05ea0864036759e1a78083240af9516#file-failed_mount_logs

But fortunately I was able to start the DevWorkspace when I tried again after about 20 minutes.

[amisevsk]

Closing this issue as #795 is merged. If we continue to encounter these events, we can reopen this issue.

Note I still haven't been able to verify the fix in #795, as this issue reproduces very sporadically.