Add support for IndexedDB

[dsseng]

It'll be useful to have IndexedDB in Deno.
Related to #1657.

[manyuanrong]

Should we use pure TS implementation or on the Rust side?

[manyuanrong]

Or use ready-made C++ implementations
https://cs.chromium.org/chromium/src/content/browser/indexed_db/

[kitsonk]

This is really complicated when you dig into it.

There is no Rust implementation of IndexDB (there is a crate that appears to be squatting, but it really isn't an implementation). Servo does not support IndexedDB, they have a long standing issue to implement. So on the Rust front it is sort of a dead end.

In Node.js, there are no realistic implementations either. Again there is a package squatting on indexeddb, but it isn't really a good package at all. The one thing that seems to have promise is IndexedDBShim which sits on top of WebSQL, and there are native implementations of WebSQL on Node.js using sqlite3.

jsdom doesn't implement IndexedDB or WebSQL, though there is this issue talking about implementing the IndexedDB shim mentioned above, I assume requiring the WebSQL implementation. It looks like it is stalled.

There is also an in memory IndexedDB API written in TypeScript.

There is what looks like a realistic Rust create for sqlite3.

So outside of using a C++ implementation, which would be a break in the pattern, the only realistic path I can easily see is that to use the in memory IndexedDB written in TypeScript, which would not give persistent storage, or we could adapt the in memory version and wire it up to a Rust sqlite3 bindings to persist the data.

I don't think just persisting out to JSON really works, because IndexedDB is a bit more complex than that, it contains concepts like transactions and other database features which is why almost all the browser implementations are based on sqlite3 IIRC.

[kitsonk]

Just updating this, servo still doesn't have it, and Gecko is looking at refactoring, but it is sqlite3 under the hoods. Here is a the most recent post that lays things out most relevantly: servo/servo#6963 (comment).

It would be really good to get localStorage/IndexedDB/kv-store in to Deno, IMO.

[kitsonk]

@anlexN your comment is borderline abusive. The issue is well discussed here of how it is a complex issue. Please keep your conversation civil and on-topic.

[jacobbogers]

HI, just exploring deno a bit, i thought (my mental model) is that deno implemented chromium aswell, (so it 100% compatible (as much as possible) with all webapis)

is the source code of the implementation of indexDB (firefox chrome) open sourced?

[kitsonk]

@jacobbogers Deno does not implement Chromium, it only implements V8. All the Web APIs need to be implemented independently. Chromium tends to implement these in C++ using the v8 binding to inject objects into the isolate.

Yes, both Firefox and Chromium are open source and their code is available, but that doesn't make it magically work for Deno.

[jacobbogers]

@jacobbogers Deno does not implement Chromium, it only implements V8. All the Web APIs need to be implemented independently. Chromium tends to implement these in C++ using the v8 binding to inject objects into the isolate.

Yes, both Firefox and Chromium are open source and their code is available, but that doesn't make it magically work for Deno.

thanks for explaining, well at least if the source is avail (Firefox/Chrome) then we can kindof get some ideas /inspiration how to implement it,

IndexDB is a tuff nut to crack, it has transactions but has no explicit commit (its weird and ingenious way of doing transactions) this would not be easy to build from scratch.

[kitsonk]

@jacobbogers we have ideas/inspiration. One of the biggest factors was that we didn't have easy access to the V8 serialize/deserialize routines, which you need to support "structured cloning" which is needed to be able to store objects. We have that now. The other problem is the underlying DB as well as support under Rust. LevelDB (the one Google created and is in Chromium) is a bit of a mess. Firefox and Safari use SQLite as well as Servo looked at SQLite (can't remember how far they got). The biggest problem is the analogues between how Chromium and Firefox operate, and how Deno operates mean that there is very little overlap. For implementing web standards, we have had a lot better luck looking at a JavaScript shim/polyfill and the specification itself, than we have looking at other browser implementations.

There are the wpt for IndexedDB which helps ensure the standard is implemented correctly, and we now have the wpt integrated into our test process.

[brainless]

Hey everyone just reading the discussions. I think IndexDB for Deno might end up being used in more varied (or larger) ways than in a browser. For example, it is OK to expect data eviction in the browser, but if developers use IndexDB as a persistent backend store (like say SQLite) then the they might want further control of the underlying storage, its space allocation, path, etc.. Ability to backup/restore also comes to mind.

Maybe my thought process is incorrect, just that I felt it is perhaps a good time to share these now, before we have a ready implementation.

[kitsonk]

then the they might want further control of the underlying storage, its space allocation, path, etc..

Then that isn't IndexedDB, that is a native SQLite plugin.

[brainless]

then the they might want further control of the underlying storage, its space allocation, path, etc..

Then that isn't IndexedDB, that is a native SQLite plugin.

You are right @kitsonk this is not IndexDB. I am just saying that people might end up using IndexDB in a different way. Maybe just great documentation will help in case people start using IndexDB as a traditional backend store.

[tracker1]

Small input on this one, investigating leveldb as a potential underlying option to implement IndexedDB on top of, and it looks like most of the rust crates are just bindings to the C library for leveldb and the single rust native implementation hasn't been updated in a long time with a warning.

I do think a rust native leveldb implementation with an IndexedDB compatibility layer over it would be a very good approach. Though, there aren't enough roots planted in that space (yet). Not sure if there is any interest in implementing such a thing in native rust, exposing a C api that matches the existing library for testing, well outside the scope of Deno.

[crowlKats]

I am currently implementing IndexedDB. It will be based on SQLite as that was the decision that was made when picking the backend for WebStorage.

[Platzer]

then the they might want further control of the underlying storage, its space allocation, path, etc..

Then that isn't IndexedDB, that is a native SQLite plugin.

You are right @kitsonk this is not IndexDB. I am just saying that people might end up using IndexDB in a different way. Maybe just great documentation will help in case people start using IndexDB as a traditional backend store.

Think about application Migration in case of a multi application server, it would be nice to just copy a single file instead of extracting a table from a file in some sys directory. 🧐

[crowlKats]

@Platzer well the idb file will be accessible just like the localstorage file.

[Platzer]

@Platzer well the idb file will be accessible just like the localstorage file.

Sorry I hadn't time to try it out, found it with deno info. This should work fine, thx 💪

[petamoriken]

Unfortunately, IndexedDB is not useful at the moment. Specifically, transactions cannot be fully wrapped in Promise because they are auto-committed after the all requests and microtasks are completed. This should be solved by a new API in the future, but we don't know when that will be.

w3c/IndexedDB#34 (comment)

[johnspurlock]

IndexedDB is not useful at the moment

Just wanted to say that while IndexedDB is not ideal, it is most certainly useful, and would be very useful in Deno for writing isomorphic browser/server-side libraries and apps.

[janus-reith]

I am currently implementing IndexedDB. It will be based on SQLite as that was the decision that was made when picking the backend for WebStorage.

Hey @crowlKats, I think support for IndexedDB in Deno would be awesome, have you been able to make progress on this, or did you face any major roadblocks?

Btw hope I won't get too far off topic with this, but with new architecture options like Cloudflare Workers emerging, and hoping Deno Deploy would eventually add such a feature too, there could be a lot of use cases for IndexedDB on the server side. (Although I'm not sure wether storage options with a a standard-compliant implementation would actually be flexible enough for use cases I have in mind.)

[crowlKats]

Hey @crowlKats, I think support for IndexedDB in Deno would be awesome, have you been able to make progress on this, or did you face any major roadblocks?

I havent. A bit after i wrote that comment, i stopped, as:
A) i didnt have the time for such a complex api
B) the api sucks to either make or use
C) I am not all too familiar with database related things
Maybe at some point i will pick it up again

Regarding storage on Deploy:
Having a KV storage is something that we are considering

[janus-reith]

I havent. A bit after i wrote that comment, i stopped, as: A) i didnt have the time for such a complex api B) the api sucks to either make or use C) I am not all too familiar with database related things Maybe at some point i will pick it up again

Regarding storage on Deploy: Having a KV storage is something that we are considering

Thanks for the quick response!

[aaronhuggins]

Anyone looking for a pony or poly fill, I've got IndexedDBShim working here based on a WebSQL ponyfill here.

[jsejcksn]

@crowlKats I think having KV storage is a more accessible first goal than an IDB implementation. Have you considered this idea, and, if so, what attributes would you expect such a feature to have? Here are some examples of what I mean by that:

• each store is namespaced (many kv stores, each accessed by its ID/name)
• each store's API is map-like
• (even if structured clone or circular references would be an issue) at least all JSON types should be supported as values, i.e.

  type JsonValue = boolean | number | null | string | JsonValue[] | { [key: string]: JsonValue | undefined };
  type Value = JsonValue | undefined;

• async?
• backed by SQLite?

[crowlKats]

This issue is mainly about CLI, and there seem to be demand for IDB, so we will implement IDB. Then for feature parity, we probably will implement IDB on deploy as well. For CLI we cant just create some new API, it goes against our principles of being web compatible.

[jsejcksn]

For CLI we cant just create some new API, it goes against our principles of being web compatible

I meant on the Deno namespace, where web-compatibility isn't a criteria (many such APIs already exist there).

e.g. Deno.kvStorage for an object or Deno.KVStorage for a class, etc.

[crowlKats]

But that has to do with web compat, or rather, in this case, incompatibility. What we put in the deno ns and what we dont is carefully considered based on if there is a web api that could take care of it.

[jsejcksn]

I think I now understand the miscommunication (and perhaps I didn't make it clear in my initial message): my suggestion is entirely unrelated to web compatibility. It's simply about offering a built-in API for persistent KV storage of native data types.

(You might consider my suggestion analogous to WICG/kv-storage.)

[crowlKats]

Yes, I understand that, but we wont go that route, since there is a web compatible solution: IDB. having 2 different APIs on a later date would be suboptimal; I wouldnt consider this as an option. KV-Storage is a dead spec. We have had this kind of discussion multiple times internally already, and we havent reached a final conclusion yet

[jsejcksn]

We have had this kind of discussion multiple times internally already, and we havent reached a final conclusion yet

Are there published notes from those? (I was actually searching for kv storage and that's how I found this issue.)

[crowlKats]

No, since this is an ongoing discussion

[aaronhuggins]

@crowlKats I have some interest in implementing this, but I don't want to take something away that you really want to do. If it is okay, should I start from your branch in #14035? Or just cut a fresh branch independent of yours and open a separate PR?

[crowlKats]

@aaronhuggins The PR isnt too far from completion, but still a bit of work left. thing is, it currently isnt high priority, so leaving this on the backburner for a bit. I would advise against starting your own PR

[jsejcksn]

Ref: #18232

Is this groundwork for a persistent storage API in Deno Deploy?

[bartlomieju]

@jsejcksn that's correct.

[benatkin]

a bit tangential, but IIRC localStorage doesn't work in web workers. It is in IndexedDB. I would hope/imagine that Deno maintains this behavior. However it isn't documented here. https://deno.land/[email protected]/runtime/web_storage_api

[idranme]

@crowlKats I have some interest in implementing this, but I don't want to take something away that you really want to do. If it is okay, should I start from your branch in #14035? Or just cut a fresh branch independent of yours and open a separate PR?

I'm kind of looking forward to it.

[alexgleason]

Now that Deno.KV is a thing, would it make sense to make IndexedDB a wrapper around the Deno.KV API instead of yet another SQLite? That way IndexedDB on Deno Deploy would take advantage of FoundationDB, and it would mean improvements to Deno.KV locally (see: #18657) would improve the IndexedDB implementation.