Cannot generate a workspace snapshot

[shaal]

Bug description

After opening DrupalPod in Gitpod, I open the hamburger menu on top-left corner, and choose Gitpod: Share Workspace Snapshot

I see Capturing workspace snapshot dialog -

After a few minutes, it displays an error -

Errors in console:

(Using Gitpod's VScode v1.59)

Steps to reproduce

1. Open DrupalPod in gitpod:
   https://gitpod.io/#https://github.com/shaal/DrupalPod
2. Open the hamburger menu on top-left corner
3. Choose Gitpod: Share Workspace Snapshot

Expected behavior

I was able to take snapshots of DrupalPod workspace in the past, but somehow I can no longer able to.

Example repository

https://github.com/shaal/DrupalPod

Anything else?

I am able to create workspace snapshot on 'simpler' repos, such as - https://gitpod.io/#https://github.com/gitpod-io/template-typescript-node

[shaal]

I tested this also by opening https://github.com/gitpod-io/gitpod in Gitpod, and coose from the menu: Gitpod: Share Workspace Snapshot, same error.

[pawlean]

I could not replicate this. 🤔 This is what I see:

What version of VS Code are you on?

[shaal]

Vscode 1.59
I tried various browsers - Chrome, edge, Firefox. Still see the same error on both Gitpod repo and drupalpod repo.

[shaal]

I updated the description of this issue, to clarify this bug is happening only on some complex repos (DrupalPod, Gitpod, etc.), while on simpler repo (typescript demo workspace) I am able to create a snapshot.

[akosyakov]

I updated the description of this issue, to clarify this bug is happening only on some complex repos (DrupalPod, Gitpod, etc.), while on simpler repo (typescript demo workspace) I am able to create a snapshot.

GLB drops web socket connection each 10mins. Maybe it is bad luck that it happens in between.

[shaal]

@akosyakov I tested this multiple times, one after the other, different repos, etc.
Are you able to take a snapshot of Gitpod or drupalpod repos?

[akosyakov]

takeSnapshot server api hangs for very long time and eventually it gets dropped by GCP via closing the web socket in 10 mins. I think meta team should look why it is. cc @JanKoehnlein @gtsiolis

If it cannot be optimised then we should reconsider api, with notification from server instead of long running RPC call.

[gtsiolis]

I could reproduce the issue. Thanks @shaal for reporting and @akosyakov for the ping. 🏓

[csweichel]

/schedule

[roboquat]

@csweichel: Issue scheduled in the meta team (WIP: 0)

In response to this:

/schedule

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[geropl]

I'm not sure this will be an issue anymore with the recent changes to websocket (to be deployed). The MessageConnection stays intact depsite the underlying transport (websocket) is reconnecting, so we should still receive the response after reconnect and have the long-running call return. Will try to test and verify.

Update: Nope, still broken 😬

[zuernBernhard]

Experiencing the same with a Lando/Drupal Project in Gitpod and a larger Database. Can we help with (for example Debugging-Output / information) ?

[geropl]

Thx @zuernBernhard . We can reliably reproduce, and are currently investigating ways to mitigate this.

The trigger is indeed the size of the repo, so a possible mitigation/workaround would be to (remporarily) reduce that.

[geropl]

@zuernBernhard Could you share a link to the project you have these problems with?

[shaal]

I created a minimal repo to replicate this bug.
When /workspace storage space is 4GB or larger - snapshot fails.
When /workspace storage space is 3GB or smaller - snapshot successful.
https://github.com/shaal/gitpod-test-failing-snapshot

[geropl]

@akosyakov We have multiple options how to solve this; none of which I find super-nice because they're rather involving. Happy to re-iterate in sync, but here's the problem:

We have to somehow make server.takeSnapshot re-entrant, so clients don't have to rely on the websocket connection to stay alive for the entire upload. To do so, we could:

1. immediately return an ID (snapshotId from DB?)
2. offer a 2nd API call server.waitForSnaphot(snapshotId) which is re-entrant and blocks until the snapshot is there
3. (also needed: make wsDaemon.takeSnapshot return early)

This leaves us with the question how we detect the state of a snapshot in server, and how we update said state:

• introduce content-service API to detect state of an snapshot and poll that (:grimacing: )
• somehow stream snapshot state from ws-manager and feed into the DB in parallel to instance updates
• make snapshots part of the instance update
• ...

@csweichel Would be interested in your thoughts as well.

[geropl]

Just had a discussion with @csweichel . Results:

• introduce snapshot.state: 'pending' | 'available' | 'error'
• drive that from
  1. server.waitForSnapshot
  2. a fallback mechanism with timeout (to convert to error eventually)
• extend content-service to "read" the state of the snapshot

[geropl]

No, roboquat! 🙃

[JanKoehnlein]

@geropl Which state is this issue in now?

[geropl]

in progress. Waiting 1 day before merging the IDE PR, and then waiting for it being deployed: gitpod-io/openvscode-server#169

lessons for me after this issue: try hard to have 1 issue per PR; everything else is cumbersome.

[geropl]

IDE part got merged: gitpod-io/openvscode-server#169 (review)
Available on "insiders" per tomorrow.

[shaal]

@geropl Thank you! cannot wait to test it out!

[gtsiolis]

Closing this as the fix has landed in production. Cc @shaal @zuernBernhard

1. Make TakeSnapshot more reliable #6144
2. [gitpod-shared] Use "waitForSnapshot" to avoid disconnects openvscode-server#169

Thanks, everyone!

Cc @geropl

[shaal]

Thank you for fixing this!
I can confirm I am able to make snapshots again.

[geropl]

Thx @gtsiolis , missed the deployment.