Define flush-to-disk guarantees and control

[inexorabletash]

Define flushing guarantees and expose control
e.g. make transactions not flush-to-disk by default, but introduce a "readwriteflush" mode or {flush: true} option

Currently, FF does not flush (but wants an option), Chrome always flushes.

[sicking]

Yeah. We found this to be a big performance issue. So we'd definitely like to have this. Most likely the right default would be to not flush since many times developers use IndexedDB as an async version of localStorage.

[inexorabletash]

I think we're in favor of this too, just a matter of ensuring out backing store is up for this.

[beidson]

The spec has long explicitly required both the A and D of ACID, and probably implicitly requires the C and I. The spec should not get into requiring specific implementation details of that ACIDity.

If an implementation can be ACID without flushing, good for them.

If an implementation see's a noticeable performance hit while flushing, they should work on that.

I think this would be complexity inappropriate for a web-facing API, and we will be against making it a requirement.

[inexorabletash]

For performance reasons, we (Chrome) are considering making the flush optional. Example:

db.transaction(scope, "readwrite", {preventFlush: true});

[pwnall]

I like relaxedDurability: true a bit more, because it doesn't make any implementation promises. Also, I think the word flush was poisoned, because POSIX has fflush (write to OS buffer cache) / fsync (write to storage medium) while Windows has FlushFileBuffers that writes to the storage medium.

To be clear, we (Chrome) would like to provide a mode that skips writing to the storage medium (fsync). We're not currently interested in skipping the fflush, because the cost of writing to the OS is small compared to typical IPC overhead.

Also, I wouldn't be against Firefox's approach of disabling fsync by default and requiring applications to opt in. Handling power failures is very subtle, and I don't think it's too bad to assume that no apps do it correctly today, so I think it'd make sense to ask developers to explicitly opt into stronger durability guarantees.

[asutherland]

Aside

To clarify, Firefox's IndexedDB is built on top on SQLite and since 2015 has used its Write-Ahead-Log in PRAGMA synchronous=NORMAL mode (SQLite docs). This is supposed to provide durability in the face of browser crashes but may rollback on device crashes.

On the issue

I'd like to instead propose that we address this via storage buckets. When creating a bucket, the site would specify what level of durability it wants the bucket to have (among other things). This provides a ton of implementer latitude just beyond "do we do an fsync at the end", as well as providing a gateway to improved storage eviction and improved user control on what's being stored on their device.

Also, in the event corruption of a database happens (which realistically is a thing that can happen, although thankfully we're past some of the darkest days of this), it makes it easier to limit the spillover damage. I think in theory, per spec, browsers would wipe the entire origin if any database in the origin became corrupt, but in practice I suspect they all just throw errors on the database until you delete it. Buckets would help with this too.

[inexorabletash]

TPAC 2019 Web Apps breakout: #CaliforniaRelaxed

• Attendees agreed with durability control on storage buckets as useful
• Attendees agreed that allowing durability control on transactions was also fine, as long as these are not in conflict
• Proposed switching proposal from a boolean to an enum allows for this: "strict", "relaxed", "default" - the latter would inherit from the bucket's default.

IDL sketch:

enum IDBTransactionDurability { "default", "strict", "relaxed" };

dictionary IDBTransactionOptions {
    IDBTransactionDurability durability = "default";
};

// in IDBDatabase:
  ...
  [NewObject] IDBTransaction transaction(
       (DOMString or sequence<DOMString>) storeNames,
       optional IDBTransactionMode mode = "readonly",
       optional IDBTransactionOptions options = {});
  ...

@asutherland - how does that look? Happy to bikeshed the naming.

Also: Should we add a readonly durability property on IDBTransaction to allow introspection? Since this property is a hint to the UA it's not as critical as in other cases.

[asutherland]

Looks good. We already have consensus on this, but we want to make sure the spec definition of durability should come from the buckets spec once that exists. (Not just inheriting of the behavior.)

At triage just now, we've also agreed that exposing the durability property makes sense.

[inexorabletash]

Awesome. Any volunteers to write a spec PR here?

(IMHO we can start sketching out a definition of durability here and then migrate it to Storage when Buckets land.)

[nolanlawson]

I see that relaxed durability has landed in Chrome 83: https://www.chromestatus.com/feature/5730701489995776

Is there a benchmark that demonstrates the performance difference between relaxed and strict durability? Looking at the Chromium commit, it references a benchmark in a Chromium bug that appears to be Google-internal.

I tried it myself, and so far I am not able to see a performance difference, but I'm not sure exactly what kind of workload would suss out the difference. However anecdotally, I do see the same kind of performance differences between Chromium and other engines described in this post.

[pwnall]

@nolanlawson Sorry, I just noticed your comment here. I opened up the bug referenced by the commit. I opened up the bug mentioned above. The benchmark in this comment may also help.