Workspace stopping while I'm working with it

[michaelcollabai]

Bug description

This is one example for an issue I've seen a couple of times this week.
I'm working with VS Code Desktop on a workspace and at some point notice that my files aren't saving anymore. The gitpod.io tab sill shows the workspace as running, but when I refresh after a while the workspace is "Stopping". It usually stays in that state far longer than if I stop the workspace myself.
After a while the workspace is stopped, I can restart it and everything works again.

Steps to reproduce

Don't know if there any specific actions that trigger it, but it has happened ~once a day this week

Workspace affected

collabai-collabai-po75ubar8fj

Expected behavior

No response

Example repository

No response

Anything else?

No response

[jkaye2012]

This has started happening to me as well this week. Pods will stop (not crash, it seems like a graceful stop was triggered) while work is on-going. I have not lost work due to these stops (files seem to be saved when it occurs), but each time this happens costs me something like 10 minutes to wait for the workspace to stop then bring it back up.

Generally, I would expect Pods to lean towards not stopping. Stopping prematurely is a huge pain for users and each time it happens it gets more frustrating.

@phimae fyi, this is impacting our team considerably.

[jkaye2012]

Note that when this happens, the Pod says it timed out:

But I made changes to this Pod approximately two minutes before it stopped itself.

[jkaye2012]

Pod id: belvederetrading-tribe-e8owwqr3eju

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jkaye2012]

Is anyone looking at this? I don't think it's been resolved.

[atduarte]

@jkaye2012 sorry for taking so long to get to this.

@loujaybee @akosyakov could it be related with IDE integration or heart beating? Doesn't seem workspace related.

[akosyakov]

@jkaye2012 Could you share workspace ids and timestamps?

[jkaye2012]

They are shared above - that's all the information I have unfortunately (considering this was two months ago)

[akosyakov]

@jkaye2012 We reworked heartbeating in July [1] for VS Code Desktop. Do you have issues now? Provided workspace id is for June.

[jkaye2012]

I don't think I've had any issues this week. The original issue was in September, so well after that rework.

[Mosaab-Emam]

This is happening to me right now, its so frequent that it occurs at least once an hour! Sometimes the workspace stops 1 minute after I start it.

I'm using VS Code desktop. Here are my version details:

[jkaye2012]

Definitely still happening, I just had a pod close after less than 10 minutes after opening

[akosyakov]

Any additional info would be helpful, at least workspace id and time stamp.

[jkaye2012]

Sure:

workspace id: belvederetrading-tribe-1dlbhi0lnj9
time: started around 7:59am Mountain time, stopped around 8:08am

I was actively writing code in VSCode when the workspace stopped.

[kendallroth]

Experiencing consistent timeouts within minutes of opening (or creating) a workspace. I nuked my workspaces in an attempt to fix (and do not have their IDs) but do have the ID of the currently broken one: lims-lims-zevlyz86ynx (not sure how to get timestamp). As was mentioned above, I was actively working in VS Code when the workspace stopped.

[Phxntxm]

Hopefully more evidence'll help... I don't have exact timestamps for mine because it was in the past, but approximate:

Workspace ID: lims-lims-o0z0c1hehgc
Timeout times:

• Around 6:32 PST about 5 minutes after it started
• Around 6:45 PST just about 2 minutes after it started
• Around 7:59 PST... this one wasn't right after it started, but timeout is supposed to be "from last working in it" right? In which case I had just stopped working in it 9 minutes prior

Unsure if it's relevant, since it seems to be the same EFFECT as originally submitted, but we've been experiencing this issue for months, just not that frequently. Today has been MUCH MUCH more frequent though. Could be the same effect but a different cause though 🤷

[pkramarczyk]

Today has been MUCH MUCH more frequent though. Could be the same effect but a different cause though 🤷

Totally agree with that. It wasn't as noticeable in the past, but it happened at least twice today to a few people I work with.

[jacobparis]

I'm hitting this today and yesterday fairly often, jacobparis-monorepo-lz1ac5j37xf in eu90

[jkaye2012]

In case it helps debugging, the issue does not occur to me if I leave the browser tab open. If I close the browser tab, my workspace closes almost immediately (within 5 minutes generally)

[Foxhoundn]

Happened twice in the past hour literally while typing - nothing abnormal to report :)

[jkaye2012]

This is still happening as well: #9629 (comment)

[mustard-mh]

@epompeii @Assaf1978-boop @jkaye2012 Could you share your workspaceID and gitpod vscode extension version ? 🙏

We released 0.0.87 which brings error telemetry and retry notification when heartbeat manager failed to init

[jkaye2012]

belvederetrading-tribe-5vy6yu86g3i

My experience was yesterday - I'm not 100% sure which version I was on at the time

[mustard-mh]

@jkaye2012 Thank you, but I find no logs related to your workspaceID (which may means you extension version is not great than 0.0.85), it would be good if you could upgrade Gitpod VSCode Extension to 0.0.87 it should mitigate with retry dialog 🙏 And I'll keep this issue open

cc @Assaf1978-boop @epompeii

[epompeii]

@epompeii @Assaf1978-boop @jkaye2012 Could you share your workspaceID and gitpod vscode extension version ? 🙏

We released 0.0.87 which brings error telemetry and retry notification when heartbeat manager failed to init

I've been using this workspace: bencherdev-bencher-qea4vq8ub8b

My GitPod plugin version is: v0.0.87

[mustard-mh]

I've been using this workspace: bencherdev-bencher-qea4vq8ub8b

My GitPod plugin version is: v0.0.87

@epompeii I checked that your current workspace is working well. Telemetry shows nothing with your previous workspace instance 😞 Maybe this telemetry problem is related to the network.

We will appreciate if you could Export all Logs and send it to us ([email protected]) next you hit it, to help debug with it 🙏

[kylos101]

👋 hey there, we're rolling out a new generation of clusters in EU and US regions that should help resolve this issue. If you start a workspace on cluster us91 or eu91 and continue to experience this issue, let us know? 🙏

[sansari]

Hi @loujaybee, this issue just started happening again this morning (i.e. I open a workspace, start working with it in VS Code desktop, and then it times out while I'm using it). My coworkers on the same team are having the same issue.

The Gitpod extension in VS Code shows v0.0.93, so it seems well ahead of the recommended upgrade version to fix the issue.

[jeanp413]

Hi @sansari, sorry we we running some experiments as we weren't sure about the root cause last time, just released a new version 0.0.94 please update again 🙏
Let me know if that doesn't solve it

[sansari]

@jeanp413 Thanks, I've upgraded to 0.0.94, will update here if the issue persists.

[Assaf1978-boop]

General .

[kylos101]

👋 @laushinka @jeanp413 I removed workspace team for now, let us know if you need to readd?

[lilnasy]

I've been running into this issue consistently, most recently in us98. I ssh into my workspaces and I have a 6h timeout set, but it stops anywhere from 5-25 minutes since starting.

[Assaf1978-boop]

Seems,nice!

[akosyakov]

@lilnasy @Assaf1978-boop Which version of Gitpod extension do you have running? Please update to the latest and try again.

[Assaf1978-boop]

Proggarming.-and,Computing.נשלח מה-iPhone שלי‫ב-5 ביוני 2023, בשעה 8:53, ‏‏Anton Kosyakov ***@***.***> כתב/ה:‬ @lilnasy @Assaf1978-boop Which version of Gitpod extension do you have running? Please update to the latest and try again. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[lilnasy]

@akosyakov I am not using the gitpod extension or VSCode at all. I ssh in and work using a terminal editor.

[akosyakov]

Thank you for the clarification, @lilnasy. It's helpful to know that SSH connections are only kept alive for pty sessions and that the editor timeout can come into play under certain circumstances. If the editor timeout is causing problems, you can disable it by extending the inactivity timeout as described in this section of the Gitpod documentation.

If you're comfortable with it, could you please share the workspace/instance IDs that you've been having issues with? You can find these by running the gp info command in your workspace or by checking the workspace URL.

With these IDs, I can look more closely at the specifics of what might be going wrong with your SSH connections.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.