New IndexedDbVFS.js

[rhashimoto]

I reimplemented the IndexedDB VFS sample. Yes, again. Here are the main changes:

Different SQLite file types are handled differently

A filesystem usually handles I/O the same way for all types of files. But SQLite has very idiosyncratic access pattern for its particular file types, which can benefit us in a couple ways. First, we can optimize the design for those patterns, and second, we don't even need to write code for paths SQLite doesn't use. For example, SQLite never requests a database file read or write that crosses a page boundary so all the code that handles that is gone.

The new VFS has a specialized class to handle I/O for the main database. Right now all SQLite temporary files (which includes journals) are stored in memory, but I am considering adding other classes for other types.

Web Locks API

The Web Locks API is available in Safari Technology Preview, which hopefully will soon end the last holdout among the major evergreen browsers. The new VFS uses Web Locks, if available, to synchronize multiple reader / single writer access on the same site origin (if not available then no locking is done and concurrent access is unsafe).

Simplified transaction reuse

IndexedDB transactions are slow, and recent work has underlined how slow by demonstrating dramatic performance gains by removing as many as possible. The previous VFS iteration applied some of those concepts to improve performance.

Unfortunately, this added a fair bit of complication to the code, which is not ideal for a sample and makes it difficult to experiment with other approaches. For example, it had some negative interactions with concurrency. The new VFS has much simpler and less aggressive transaction reuse (e.g. no more dummy requests to extend transaction lifetimes). This trades back some performance for more understandable and better behaved code.

Unlimited write cache

The new VFS buffers all database writes until the end of the current SQLite transaction, at which point it stores them in a single IndexedDB transaction. This is good and bad. It's good for concurrency because it minimizes exclusive write locks. It's good for keeping the database file consistent - it doesn't need a persistent journal to guarantee consistency. But it's bad for memory consumption if you have big transactions, and furthermore if you use a journal anyway that might be doubling the memory use.

You can reduce memory use with PRAGMA journal_mode=off as long as you don't need ROLLBACK. Storing the journal in a separate IndexedDB store is an option. I also had another off-the-wall idea, a hack really, of not storing rollback journal writes and instead just clearing the database write cache when a rollback is detected - it's tricky and I haven't had success with it yet. Another memory-saving option is to spill the write cache to a separate IndexedDB store.

---

The IndexedDB schema has changed. The VFS contains code to update to the new version, but if you have trouble, e.g. with the demo page, you should first make sure you don't have any other tabs open on an old version and as a last resort clear your browser local state for the problem site.

[rhashimoto]

I made two of the proposed improvements:

Cache spill

The IndexedDB VFS buffers up to 16 MB of transaction changes in memory. After that it spills data to a separate object store.

"Zero" storage journal

Because the database isn't written until a transaction is committed, most of a rollback journal's contents are also available in IndexedDB, and so is redundant. The journal file implementation now only saves in memory the header and which pages are being written to the journal ("zero" storage means not saving any page data). If the journal file is read back, i.e. for rollback, the returned data is reconstituted on-the-fly using the database store.

---

These additions effectively remove any memory constraints on transaction sizes, and the low overhead of journalling means there is little advantage to change the default journal_mode unless user ROLLBACK is very frequent.