Add descending order for getAll() and getAllKeys()

[nolanlawson]

In PouchDB we're implementing a "batched cursor" using getAll() and getAllKeys() (pouchdb/pouchdb#6060, pouchdb/pouchdb#6031). E.g. it can be used to paginate through an objectStore n records at a time, given a lower bound and optional upper bound.

Unfortunately we are only able to do this in ascending order, since in descending order the limit will always give us the first n records instead of the last n records. it would be nice if we could do something like:

objectStore.getAllKeys(keyRange, batchSize, 'prev').onsuccess = ...
objectStore.getAll(keyRange, batchSize, 'prev').onsuccess = ...

... where the batchSize would fetch the highest n records instead of the lowest. The downside of this method is that it's tricky to feature-detect unless you insert some dummy data and then try to fetch it, although maybe there's a more clever way.

[nolanlawson]

Just talked with @aliams, and his suggestion was to instead do something like:

objectStore.getAllKeysFromEnd(keyRange, batchSize).onsuccess = ...
objectStore.getAllFromEnd(keyRange, batchSize).onsuccess = ...

This would be feature-detectable, although it's maybe not as elegant as an Iterable or generator or some other fancier alternative.

[aliams]

@inexorabletash, thoughts on this?

[inexorabletash]

Seems plausible.

I'm don't like the ever-growing list of methods as we add query types one by one and thus need to make them feature detectable, but we haven't come up with a good pattern for feature detecting options. (Or maybe we have and I just don't know?) Crazy thoughts like query(range, {options}) and then a helper like queryOptions({options}) that returns the options dict back to you as it was parsed (so unsupported options would vanish). But we should solve that for the whole platform, not just this spec/feature.

Is this a hint that Edge might be implementing here? :)

[aliams]

We thought the same in terms of an ever-growing list of methods. However, we were thinking maybe this was analogous to having indexOf() and lastIndexOf().

@nolanlawson brought up how he would like to be able to efficiently iterate through an object store backwards in PouchDB since currently they're using a cursor for backwards iteration and getAll for forward iteration (if available). He brought up that he raised the issue here and I just wanted to help give it some attention in case we want to define this for v3 of the IDB spec. In terms of implementing such a method, I would support it since it would be helpful for use cases such as the one @nolanlawson mentioned :)

[randomekek]

It's sad that since this is not in the initial spec, future versions need feature detection. Is there any other way forward?

Example 1: getAll({query, count, direction}). Old clients will reject this with an exception as the first parameter is not an index. This allows new clients to feature detect.

Example 2: getAll(query, count, options). Old client reject this with exception - i hope it does?

[inexorabletash]

Example 2 will not throw; extra arguments are ignored. (JS built-in method semantics, which WebIDL-based methods inherit)

[randomekek]

Would having an explicit features object, ie no implicit detection, be a bad idea?
IndexedDB.features = {getAllDirectionParam: true}

[inexorabletash]

There hasn't been a proposal that would apply to all specs that people are happy with.

Feature detection is possible here using a getter on the passed dictionary. Not pretty but it works.

[randomekek]

Thanks, looks like it will be a lot of work, best of luck

[inexorabletash]

TPAC 2019 Web Apps Indexed DB triage notes:

• We could add 'prev' easily to the getAllEntries() proposal (getAll() with both key and value (or index key and primary key) #206)
• Another idea is to allow IDBKeyRange itself to have a direction. This is redundant for cursors, but for get/getAll (etc) it is rather elegant.

We'll try to come up with a holistic proposal.

[AbdulRehman197]

use Curser prev prevUnique I think this work for now

[stefanhaustein]

How about using a negativ limit to indicate "backwards", changing the "count" type from "unsigned long" to "long"?

No limit could be expressed by -Infinity, assuming this would be clamped to the minimum long value?

[SteveBeckerMSFT]

TPAC 2024: The getAllEntries() proposal adds a direction option to support reverse iteration. For consistency, should we also expose this option to getAll() and getAllKeys()?

Providing the direction option on getAllKeys() might be useful for reverse iteration scenarios that don't want to load every value enumerated like getAllEntries() behaves.