Use in-memory file system for virtual documents (if available)

[krassowski]

What are you trying to do?

Improve the performance by storing the virtual documents on ramdisk. I've used ramdisk before from Python, but this time looking at pyfilesystem's memoryfs hoping it will give us cheap cross-platform support:

How is it done today, and what are the limits of current practice?

We are writing and reading a lot form the disk. For those who have SSD disks the speed is ok, but it must be a pain for others; even for those who have SSD, it would help them as the longevity of the SSDs will increase.

This might require that the virtual file system is symlinked to the project (or maybe not - we will see).

Related to slow autocompletion (#272) and other improvements to virtual documents #353 , #336

[bollwyvl]

As the complexity of this goes up, something that starts bringing in _lots_ of dependencies might need to become managed as its own package (e.g. jupyter-lsp-vdocs) which depends on jupyter-lsp-core... and/or we might need to collaborate more closely with the debugger folks. I can't imagine _two_ things try to do this at the same time on one computer is any fun. the proxy kernel approach certainly complicates things, as to when this work should be done, but might improve this situation, as at least one wouldn't have to bounce the server to try out a new feature...

[krassowski]

Right, it would be useful to disable this workaround if one only uses LSP servers/features that do not require a file on disk.

Had a look into pyfilesystem and unfortunately is not fit for the task as it does not give the on-disk path for the memory filesystem (i.e. it seems it uses a custom implementation rather than standard ramdisk which can be mounted to any directory) so we cannot give a valid path to the LSP servers.

It seems that using Linux ramdisk and ImDisk/SoftPerfect RAM Disk for Windows would be the way forward. This all should be wrapped in a huge try-catch and fallback to a static directory or tmpdir.

[bollwyvl]

Maybe it's worth looking at what pygls does? https://github.com/openlawlibrary/pygls/blob/master/pygls/workspace.py Again, I wouldn't be sad if we took that as a dependency, if it provides value... while it _also_ doesn't write to disk, it might be a better scaffolding than doing it ourselves/differently...