Switch from browserfs to OPFS #14263

robertjndw · 2024-10-07T21:02:38Z

What it does

This PR builds on top of the work that was done in PR #12853.
As browserfs was deprecated in March 24 (see original repo), it is beneficial to switch to an alternative.
As discussed in #14125, the direct successor of browserfs is zenfs. However, during the research I also stumbled across OPFS which is a new native Browser API which covers a lot of the features necessary for this use case.

How to test

Build and run the 'browser-only' Theia example.

Similarly to the previously mentioned PR #12853.

For debugging purposes of the OPFS, I can really recommend this extension for the Chrome DevTools: https://github.com/tomayac/opfs-explorer

Follow-ups

The watch functionality is currently missing. Since OPFS does not provide a native API for this, it has to be manually tracked and implemented

Review checklist

As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

As a reviewer, I agree to behave in accordance with the review guidelines

sdirix

Thanks for the work ❤️ ! For me it mostly works ;)

Please have a look at my comments and make sure that all automated CI checks go through.

packages/filesystem/package.json

examples/api-samples/src/browser-only/filesystem/example-filesystem-initialization.ts

sdirix · 2024-10-09T06:19:20Z

packages/filesystem/src/browser-only/opfs-filesystem-initialization.ts

+    createMountableFileSystem(): Promise<FileSystemDirectoryHandle>
+    initializeFS: (fs: FileSystemDirectoryHandle, provider: OPFSFileSystemProvider) => Promise<void>;


We are now exclusively using OPFS, therefore the whole initialization could be adapted, for example like this

Suggested change

createMountableFileSystem(): Promise<FileSystemDirectoryHandle>

initializeFS: (fs: FileSystemDirectoryHandle, provider: OPFSFileSystemProvider) => Promise<void>;

getRootDirectory(): Promise<FileSystemDirectoryHandle>

The initialization can then just be implemented in this one method. What do you think?

I see your point about simplifying the initialization into one method now that we're exclusively using OPFS. However, I think keeping the two methods still makes sense in this context.
One focuses on returning the root directory, while the other is responsible for preparing the file system, which is especially useful when we want to create files and folders in advance for the example. What are your thoughts on this approach?

packages/filesystem/src/browser-only/opfs-filesystem-provider.ts

james-pre · 2024-10-10T22:48:42Z

Could a maintainer please link this to #13520 so #13520 will be auto-closed on merge? Thanks!

robertjndw · 2024-10-16T15:00:21Z

@sdirix Can you maybe take a look at the CI/CD error? It seems that with the addition of dom.asynciterable to the tsconfig something broke 🤔

sdirix · 2024-10-16T15:33:59Z

@sdirix Can you maybe take a look at the CI/CD error? It seems that with the addition of dom.asynciterable to the tsconfig something broke 🤔

@theia/ffmpeg: $ eslint --cache=true --no-error-on-unmatched-pattern=true "{src,test}/**/*.{ts,tsx}"
@theia/re-exports:
@theia/re-exports: /home/runner/work/theia/theia/dev-packages/private-re-exports/src/bin-package-re-exports-from-package.ts
@theia/re-exports: 0:0 error Parsing error: Invalid value for lib provided: dom.asynciterable

Seems like the Typescript Eslint integration does not like the dom.asynciterable lib. I'm not sure why this is the case as the Typescript we are using is definitely compatible to it.

Anyway, you can override the lib settings in configs/base.eslintrc.json and there specify the parserOptions.lib to the previous values, i.e.

    "lib": [
      "ES2019",
      "ES2020.Promise",
      "DOM"
    ]

That should fix the issue

james-pre · 2024-11-05T18:24:30Z

LGTM! @robertjndw any updates on this?

sdirix

I updated to latest master.

I have some more comments before we can merge.

sdirix · 2024-11-08T11:53:07Z

packages/filesystem/src/browser-only/opfs-filesystem-provider.ts

+    private directoryHandle: FileSystemDirectoryHandle;
+    private initialized: Promise<true>;
+
+    // MARK: constructor


these marks should be removed

sdirix · 2024-11-08T11:57:57Z

packages/filesystem/src/browser-only/opfs-filesystem-provider.ts

+                };
+            }
+
+            throw createFileSystemProviderError('File does not exist', FileSystemProviderErrorCode.FileNotFound);


The message does not fit. According to the type system this error should never be thrown. So I would rather throw an "unknown file handle type error" with FileSystemProviderErrorCode.Unknown instead.

sdirix · 2024-11-08T11:58:25Z

packages/filesystem/src/browser-only/opfs-filesystem-provider.ts

+            throw createFileSystemProviderError('File does not exist', FileSystemProviderErrorCode.FileNotFound);
+
+        } catch (error) {
+            throw createFileSystemProviderError('File does not exist', FileSystemProviderErrorCode.FileNotFound);


At this point we are not sure that this is the cause of the error, right? I think the error handling here needs to be more generic

sdirix · 2024-11-08T12:00:22Z

packages/filesystem/src/browser-only/opfs-filesystem-provider.ts

+                    // Ignore errors for individual entries, log them and continue
+                    // console.error(error);


Are any actual errors thrown here? What does this try-catch protect against?

sdirix · 2024-11-08T12:04:41Z

examples/api-samples/src/browser-only/filesystem/example-filesystem-initialization.ts

        } catch (e) {
-            console.error('An error occurred while initializing the demo workspaces', e);
+            console.error('An error occurred while reading the demo workspaces', e);


We should check whether the error is a file not found error. Only then we should create files. Also we should not log the error as this is an expected error for any first time user.

Only other errors than the expected ones should be logged.

Even better would be to offer some kind of "exists" method so that we don't need to go through try-catch here.

robertjndw and others added 6 commits September 28, 2024 22:22

Initial setup and experiments with OPFS

70e7153

Add missing methods and event emitter

a28a085

Remove references to browserfs and replace it with OPFS

6578c25

Improve coding style and remove logging

e928eec

Merge branch 'eclipse-theia:master' into feat/switch-to-opfs

28d70de

Update CHANGELOG.md

05e26ad

robertjndw changed the title ~~Feat/switch to opfs~~ Switch from browserfs to OPFS Oct 7, 2024

JonasHelming requested a review from sdirix October 8, 2024 06:41

msujew mentioned this pull request Oct 8, 2024

Switch from browserfs to zenfs #14125

Closed

1 task

sdirix requested changes Oct 9, 2024

View reviewed changes

sdirix mentioned this pull request Oct 9, 2024

BrowserFS Deprecated #13520

Open

sdirix linked an issue Oct 11, 2024 that may be closed by this pull request

BrowserFS Deprecated #13520

Open

robertjndw added 2 commits October 16, 2024 15:33

Add yarn.lock file and AsyncIterable to base tsconfig

4fa4158

Implement feedback from PR and fix directory renaming bug

263815e

robertjndw added 2 commits October 17, 2024 11:18

Adapt eslintrc config

e24978c

Fix linting issues

8fa511b

sdirix added 2 commits November 8, 2024 12:44

Merge remote-tracking branch 'upstream/master' into feat/switch-to-opfs

ad7c56e

adapt changelog entry

3a630c6

sdirix reviewed Nov 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch from browserfs to OPFS #14263

Switch from browserfs to OPFS #14263

robertjndw commented Oct 7, 2024 •

edited

Loading

sdirix left a comment

sdirix Oct 9, 2024

robertjndw Oct 16, 2024

james-pre commented Oct 10, 2024

robertjndw commented Oct 16, 2024

sdirix commented Oct 16, 2024

james-pre commented Nov 5, 2024

sdirix left a comment

sdirix Nov 8, 2024

sdirix Nov 8, 2024

sdirix Nov 8, 2024

sdirix Nov 8, 2024

sdirix Nov 8, 2024

sdirix Nov 8, 2024

		createMountableFileSystem(): Promise<FileSystemDirectoryHandle>
		initializeFS: (fs: FileSystemDirectoryHandle, provider: OPFSFileSystemProvider) => Promise<void>;

	createMountableFileSystem(): Promise<FileSystemDirectoryHandle>
	initializeFS: (fs: FileSystemDirectoryHandle, provider: OPFSFileSystemProvider) => Promise<void>;
	getRootDirectory(): Promise<FileSystemDirectoryHandle>

		// Ignore errors for individual entries, log them and continue
		// console.error(error);

Switch from browserfs to OPFS #14263

Are you sure you want to change the base?

Switch from browserfs to OPFS #14263

Conversation

robertjndw commented Oct 7, 2024 • edited Loading

What it does

How to test

Follow-ups

Review checklist

Reminder for reviewers

sdirix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

james-pre commented Oct 10, 2024

robertjndw commented Oct 16, 2024

sdirix commented Oct 16, 2024

james-pre commented Nov 5, 2024

sdirix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertjndw commented Oct 7, 2024 •

edited

Loading