Persist filesystem changes after page refresh

[adamziel]

What problem is this issue trying to solve?

At the moment, all database changes and uploads are gone once the page is refreshed. Preserving them would be useful for courses, technical demos, even sharing a link to your changes.

What solution does this issue propose?

A few ideas:

• OPFS. Related Emscripten PR: Add support for read/seek on files in the native file system
• localStorage could store the binary contents of the .sqlite file every few seconds
• An "audit trail" of all SQL writes could be stored and re-applied on refresh
• (in the future) The sqlite file could reside in the persistent Filesystem API
• https://github.com/jlongster/absurd-sql can persist SQLite database in IndexedDB See https://sqlite.org/wasm/doc/tip/about.md

[adamziel]

Relevant: https://sqlite.org/wasm/doc/tip/about.md

Specifically:

James Long's aptly-named absurd-sql demonstrates persistent browser-side sqlite3 by storing the databases in IndexedDB storage. We experimented with that approach but it's... well, absurd 😉. It's too slow for anything beyond trivial databases and it's easy to get a database in an inconsistent state by visiting the page from two tabs.

[eliot-akira]

About persisting changes to the database, I recently learned about SQLite's Session extension.

The session extension provide a mechanism for recording changes to some or all of the rowid tables in an SQLite database, and packaging those changes into a "changeset" or "patchset" file that can later be used to apply the same set of changes to another database with the same schema and compatible starting data. A "changeset" may also be inverted and used to "undo" a session.

I read about it in an article about someone developing an iOS app with a local SQLite database (full copy of the user data in the app itself).

The session extension lets you start a session on an SQLite connection. All changes made to the database through that connection are bundled into a patchset blob. The extension also provides method for applying the generated patchset to a table.

This can be used to build a very simple client-sync system. Collect the changes made in a client, periodically bundle them up into a changeset and upload it to the server where it is applied to a backup copy of the database.

Perhaps this could be a lighter-weight approach to saving SQLite state in browser local storage, by storing (and restoring) the session history instead of a snapshot of the whole database.

[swissspidy]

Related: https://developer.chrome.com/blog/sqlite-wasm-in-the-browser-backed-by-the-origin-private-file-system/

[adamziel]

IndexedDB is way too slow for persistence backend.

Emscripten supports storing files in IndexedDb via IDBFS. I got a demo to work. If you run it and refresh the page, the initial console.log() will show the test.txt file:

	// In loadPHPRuntime():
 
	PHPRuntime.FS.mkdir('/offline');
	PHPRuntime.FS.mount(PHPRuntime.FS.filesystems.IDBFS, {}, '/offline');
	console.log(PHPRuntime.FS.readdir('/offline'));

	const syncFs = (populate:boolean) => new Promise((resolve, reject) => {
		console.log('Syncing FS...');
		PHPRuntime.FS.syncfs(populate, function (err: Error) {
			console.log('done', err);
			if (err) {
				reject(err);
			} else {
				resolve(true);
			}
		});
	});

	// Populate /offline from IndexedDb:
	await syncFs(true);

	PHPRuntime.FS.writeFile(
		'/offline/test.txt',
		new TextEncoder().encode('Hello world!')
	);

	// Write files from /offline to IndexedDb:
	await syncFs(false);

However, if you do that with WordPress files, each sync will take minutes:

	PHPRuntime.FS.mkdir('/wordpress');
	PHPRuntime.FS.mount(PHPRuntime.FS.filesystems.IDBFS, {}, '/wordpress');
	//await syncFs(true);
	console.log(PHPRuntime.FS.readdir('/wordpress'));
	for (const { default: loadDataModule } of dataDependenciesModules) {
		loadDataModule(PHPRuntime);
	}
	if (!dataDependenciesModules.length) {
		resolveDepsReady();
	}

	await depsReady;
	await phpReady;
	console.log(PHPRuntime.FS.readdir('/wordpress'));
	await syncFs(false);
	console.log(PHPRuntime.FS.readdir('/wordpress'));

Perhaps storing just diffs would work better, but it sounds like a lot of work.

Here's another idea

Exporting and importing a zipped WordPress site works in a blink of an eye. Let's reuse it for persistence.

[adamziel]

More thoughts:

The easiest way to implement this is through explicit buttons like:

• Save - exports the site, writes the zip to localStorage
• Restore - imports the last zip written to localStorage

These could be integrated with the version switchers and the beforeunload event to give an illusion of a continuous synchronisation. On top of it, we could have an autosave that runs this sync once a minute or so, if anything in the filesystem changed.

In fact, we could retain the zip file in memory, keep track of all VFS writes, and update only specific files inside the archive.

A few more ideas:

• Write the VFS object directly to localStorage without going through zip. Synchronize all the writes as diffs and flatten everything every X writes.
• Observe writes and perform them again in an Origin Private Filesystem. It is pretty well supported. After page refresh, initialize from OPFS. Perhaps we could just switch to OPFS as a backend?

[jimmywarting]

OPFS is a grate choice.
together with asyncify -2 and JSPI the WebAssembly runtime can be suspended while the js glue awaits a read/write operation.

you should look into switching to OPFS and JSPI

[adamziel]

@jimmywarting both are on the roadmap! JSPI will need to stabilize first - right now it’s chrome-only and limited to arm64 and x64. Also, we’ll need to build PHP with pthreads support as I think Emscripten can only support OPFS via threaded WASMFS (although I’d love to be wrong here).

[adamziel]

Done in #196