Rust implementation of IDBBatchAtomicVFS?

[abhay-agarwal]

Curious if there is any existing work around having a set of native rust bindings to wasm sqlite that can still leverage the IndexedDB-backed VFS.

We have a hybrid application where we have a common rust codebase that cross compiles to wasm for web, with wa-sqlite as a sqlite implementation, and then a native toolchain with a desktop binary of sqlite for our Tauri application.

It would be great to not have to call back out to js in wasm, but just to handle the operations entirely in rust, with the native c bindings. Happy to try and work on this, but just wanted to confirm if I have the right mental model here, and if there are any extreme gotchas. Things we also need:

1. SharedWorker support -- we have a common database for desktop, want to have a common database for web.
2. WebLock support -- I think web-sys can handle this part just fine, though there might be some fun async issues here that I am not aware of.

If anyone has some pointers to prior art, would love to jump on that!

[rhashimoto]

I don't really understand your question, but I also don't know anything about Rust. On first reading, it sounds like you want a way to use IndexedDB for SQLite storage "entirely in rust" without involving any JavaScript. I don't think that's possible as I don't know of any direct WebAssembly bindings for IndexedDB.

If you're instead asking whether you can use the SQLite C API directly in WebAssembly instead of the wa-sqlite wrappers, while still using the IDBBatchAtomicVFS, I think that can be done. I don't think you can use wa-sqlite as-is; I think you'll want to steal pieces of it and incorporate them into your application build.

Maybe someone else here will have a better understanding of your goals and better advice.

[abhay-agarwal]

There are no direct WebAssembly bindings for web-native APIs at the moment (to my knowledge) -- instead, there is a rust crate called web-sys that essentially provides a FFI-like set of interops that call into the native APIs from your wasm binary (essentially serializing and deserializing, but you can also provide direct memory mapped access as long as the data is bytes and not, e.g. strings.). IDB might be able to accept a direct ArrayBuffer, though I am not sure. In any case, the data needs to be copied to make it into IDB at some point.

A rust-centric build of sqlite-on-IDB would contain rust wrappers over C interfaces. These could be compiled via emscripten alongside sqlite itself.

The major change from wa-sqlite to a rust implementation would be that IDBBatchAtomicVFS would likely want to be re-written in Rust to take advantage of better typing, perhaps direct pointer access to SQLite, and maybe stuff like pre-compiled SQL queries, etc.

Not saying that any of this is anyone's responsibility to build, but it does seem like a natural next step for the evolution of sqlite-in-the-browser, if I'm not mistaken about anything.

[abhay-agarwal]

Upon further reflection on this, I and my team will definitely be investigating a rust-backed version of sqlite that can use parts of this repo while also taking advantage of direct memory access to the SQLite C ABI. In my haziest aspiration, perhaps we could leverage Rust's borrow primitives to avoid more or most of the lock management weirdness of JS-centric implementations.

If you or anyone is interested in scoping this out and getting involved, please feel free to reach out to me: abhay at withpoly dot com. Happy to make it fully open source, and permissively licensed of course.

[quolpr]

while also taking advantage of direct memory access to the SQLite C ABI. In my haziest aspiration, perhaps we could leverage Rust's borrow primitives to avoid more or most of the lock management weirdness of JS-centric implementations.

I suspect that performance difference will be not noticeable, tbh, and it will still be impossible to avoid lock management. It's not about JS it's about web api(I mean web lock api) with which you will still need to deal even in rust implementation.

[abhay-agarwal]

If your codebase lies more on the wasm side than the js side, then you save a lot on transferring over the wasm-js boundary. That's enough of an incentive to explore how rust-side lock management strategies could save you. Given that Rust gives you finer grained control over who is doing the writing and when, you could be more judicious about how you implement a shared locking strategy.

For example, in my code, I haven't found an easier/more reliable way to do single-writer than to just wrap the whole write call in a weblock. This allows concurrent reads to still occur. But I suspect that you can overcome that requirement if you have better awareness of who is doing the writing, such as compile-time borrow checked handles. That way, you don't need to raise a write lock at all.

But you could be totally right that the difference is negligible. Given our extensive wasm base, we're incentivized to give it a shot.

[daaku]

Did you get this WASM build working with Rust by any chance?

[abhay-agarwal]

I ended up not finding a great pathway to implementation but I'd be happy to collaborate if you think there’s a path forward.

There are two challenges here:

• importing an emscripten built wasm binary into a friendlier wasm-bindgen environment with the C ffi
• re implementing the vfs in rust. This would lukely be a basic version of the idbbatchatomic vfs for maximal compatibility.

[insipx]

Hey @daaku @abhay-agarwal @quolpr @rhashimoto !

I have a crate with a SQLite WASM build that creates a diesel backend working here with the official sqlite build:
https://github.com/xmtp/libxmtp/tree/wasm-backend/diesel-wasm-sqlite

There's a branch that uses Roy's wa-sqlite here: https://github.com/xmtp/libxmtp/tree/wasm-backend-async/diesel-wasm-sqlite

It's a bit tricky to implement the backend with full-async in Rust WASM to be compatible with this build, because of the lack of a good AsyncDrop in rust. Sending a message through a spawned task is a mediocre workaround, because there's no guarantee that the task will run & the statement will be dropped. This can make creating a performant async backend a challenge. For instance, in the sync version above, there are a couple versions of sqlite statements. One of which, when dropped, resets the statement/bindings and frees allocated values but allows the statement to be re-used (to avoid another prepare call). This is trickier in an async world. It's definitely still possible though!

A downside to this is that OPFS is all in JS and just loaded into rust. It would be more complex to create a vfs straight from rust -- i'm not even sure the bindings for that exist in Rust. I would have to look more into how VFS' and their initialization work in SQLite3, but that will definitely be my next step.

However, the WASM module is available so loading values into/out of SQLite WASM memory can occur through pointers, which circumvents the JS part of it.

Future work:

• Get the rust WASM library host to share memory with the SQLite wasm. This enables lots of optimizations for wasm memory allocation that aren't possible simply because of the wasm-bindgen -> JS -> SQLite layer. So, whenever we load a value from SQLite, we have to copy it into our host WASM memory rather than just borrowing it.
  - sharing the memory between modules isn't actually hard, but mixing that in with loading the VFS may get tricky. Haven't really had the time to look into that yet. Any tips here would be nice.
• A custom VFS in Rust WASM would also be ideal
• bunch of other features to make it feature-complete with the official diesel sqlite backend (i.e compatibility with other rust libraries like chrono etc.)

Going to be talking about my work on this lib in a couple days if you're interested:

https://events.xyz/events/93062dcf

[abhay-agarwal]

This is incredible! Thanks for sharing. I’ll definitely be diving into this in the next day or two.

There are a lot of different architectures that have interesting trade-offs, so it'll be great to have this community around this grow and people try different strategies to see what works best.

[insipx]

100% ! Certainly lots of in-browser database solutions emerging these days :)

[daaku]

Very cool! Is the event recording going to be available after it's over?

[insipx]

It will be posted to our youtube channel afterwards

[abhay-agarwal]

HN discussion on SQLite's official wasm build (which doesn't seem to support shared workers?) has caused some more chatter about a Rust-centric build. Curious if anyone else is interested in collaborating?

https://news.ycombinator.com/item?id=41851051#41851864

[insipx]

Once https://github.com/xmtp/diesel-wasm-sqlite is more stable, one of the next steps would be implementing the Rust VFS and passing it to the SQLite blob. Although, the c-wasm-abi work is promising, and we could see a version of SQLite that can be natively compiled into rust without these workarounds in the future

[rkistner]

We went a slightly different route to get something similar: We built a Rust-based SQLite extension, which can then be loaded on any practically platform where you can use SQLite, including WASM. While we haven't tested with Tauri yet, I can't imagine any blockers there.

Some advantages include:

1. Interfacing with SQLite from the Rust code is really simple and fast, likely faster than what it is from JS for example.
2. FFI between the application code (e.g. JS) and Rust code is limited to a single function call to load the extension. Everything else happens via SQLite APIs (custom functions and virtual tables).

The biggest complexity comes in with figuring out how to build and load the extension for each platform, but if you only need to support WASM + Tauri it's not that much.

Our codebase is here: https://github.com/powersync-ja/powersync-sqlite-core/
Some of the ideas are originally from cr-sqlite, which does the same but with a combination of C and Rust: https://github.com/vlcn-io/cr-sqlite/