Workspace cannot be restarted when /worksapce is full.

[kylos101]

Is your feature request related to a problem? Please describe

Once disk usage in /workspace approaches or hits a limit (30G), our ability to restore workspaces becomes unstable and files can no longer be added to /workspace.

Describe the behaviour you'd like

1. How reliably do we restore /workspace once "almost full", say 29.5GB? With one large file? With many small files? How long do the backup and restore take?
2. Run a test, with a /workspace that has its limit set to 50GB in a XL cluster. With one large file? With many small files? How long do the backup and restore take?

Additional context

Given the below test, we were able to backup the workspace, but not restore it.

gitpod /workspace/template-python-flask $ dd if=/dev/zero of=upload_test bs=1M count=32000
dd: error writing 'upload_test': No space left on device
30535+0 records in
30534+0 records out
32017219584 bytes (32 GB, 30 GiB) copied, 19.5889 s, 1.6 GB/s

gitpod /workspace/template-python-flask $ touch foo
touch: cannot touch 'foo': No space left on device

gitpod /workspace/template-python-flask $ du -hs
30G  

gitpod /workspace/template-python-flask $ ls -ltrh
total 30G
-rw-r--r-- 1 gitpod gitpod  950 Mar  2 14:27 README.md
-rw-r--r-- 1 gitpod gitpod  122 Mar  2 14:27 MANIFEST.in
-rw-r--r-- 1 gitpod gitpod 1.5K Mar  2 14:27 LICENSE.rst
drwxr-xr-x 2 gitpod gitpod  122 Mar  2 14:27 tests
-rw-r--r-- 1 gitpod gitpod   38 Mar  2 14:27 setup.py
-rw-r--r-- 1 gitpod gitpod  535 Mar  2 14:27 setup.cfg
drwxr-xr-x 2 gitpod gitpod  110 Mar  2 14:27 flaskr.egg-info
drwxr-xr-x 5 gitpod gitpod  136 Mar  2 14:27 flaskr
drwxr-xr-x 2 gitpod gitpod   27 Mar  2 14:27 instance
-rw-r--r-- 1 gitpod gitpod  30G Mar  2 14:28 upload_test

On restore, the error we received was cannot initialize workspace: cannot write workspace ready file content: open /dst/.gitpod/ready.tmp: no space left on device.

[axonasif]

Hey @kylos101, I have a bunch of friends who would like to switch to Gitpod if they could have at least a 100GB container image and about 200GB+ total workspace-session storage for android system development, currently, they use a VPS server and opens in vscode via ssh and the flow and consistency isn't as great as in Gitpod if you know what I mean. If this would fall under this and if possible then it'd be nice! 😄

[kylos101]

Hi @axonasif thank you for sharing. The purpose of this issue is to test our current capabilities, and see the impact of stretching them - as an experiment. This is not describing a new feature.

[kylos101]

Testing our existing 30 GiB workspaces

Given setup like this:

dd if=/dev/zero of=upload_test bs=1M count=29000

gitpod /workspace $ ls -ltrh
total 29G
drwxr-x--- 7 gitpod gitpod 209 Mar 11 14:02 template-python-flask
-rw-r--r-- 1 gitpod gitpod 29G Mar 11 14:36 upload_test

gitpod /workspace $ du -cs
29907652        .
29907652        total

From one workspace https://gitpodio-templatepythonf-sl0kzid2f5p.ws-us34xl.gitpod.io/, started and stopped many times.

Number	Test file	Seconds	Notes
1	Backup 30 GiB file	~240
2	Restore 30 GiB file	83
3	Backup with new empty file	236
4	Restore with new empty file	180
5	Backup another empty file	~240
6	Restore another empty file	300	The workspace never started, and went to stopping after the elapsed time
7	Backup yet another empty file	~240
8	Restore yet another empty file	95

From https://gitpodio-templatepythonf-3ggexai3nb4.ws-ephemeral50.gitpod.io/ (gen35).

Number	Test file	Seconds	Notes
1	Backup 30 GiB file	~240
2	Restore 30 GiB file	120
3	Backup with new empty file	224
4	Restore with new empty file	115

Summary

• Backups are consistently around 240 seconds given this small set
• Restores vary (83-300 seconds), and may timeout, stopping the workspace. There may be a 5 minute context we'll want to increase, especially if we bump workspace storage up.

Next steps

• Use atop on the node to observe performance for backups and restores.
• Check for a related 5 minute context for the restore process
• Run the same experiment on the XL cluster with expanded /workspace

[kylos101]

This is no longer relevant. Why? We're switching to use PVCs, and we've added metrics to help measure content restore and content backup.

[utam0k]

fixed by #10395