-
Notifications
You must be signed in to change notification settings - Fork 105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MessageStore interfaces for sorting and pagination #478
Conversation
7782db6
to
b2d9aac
Compare
454767c
to
2e1ca9c
Compare
I did some more thinking about what this will look like when fully implemented with level, this is what I came away with and would love your thoughts @thehenrytsai & @diehuxx. I generalized the For every type of property we are sorting by, the number of indexes grows. So for now we should really only sort by I think leaving the interface generalized is a good idea so that we can add other higher level abstractions with different types of dateSort fields to different message types and only need to add indexes. Also thinking about pagination, cursor based pagination started making a lot more sense within the DWN context. I think we should start there and move to offset based pagination if there is a strong demand for it, as it will also add overhead to indexing and performance that might not be necessary. Right now I am thinking that the cursor pointer for pagination be the Would be happy to discuss this more in depth over a call. But for now figured looking at the interface and naive implementation will be a good place to start. |
2e1ca9c
to
78714cb
Compare
@thehenrytsai I think this is ready for a review at this point after yesterday's conversation. I was going to add some types/properties to describe the sortable index fields somewhere, but it felt like premature optimization. |
bdc2caf
to
665686a
Compare
665686a
to
8602426
Compare
8e94251
to
97d3930
Compare
1125755
to
6588d01
Compare
74c9f19
to
4cc42be
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🥇 🎸 🚀 Long time coming!
Updating the DWN SDK to the latest version. This Includes: - `encodedData` stored within the `MessageStore` below a certain threshold - `MessageStore` interface now accepts multiple Filters to create OR conditions between filters. - 'MessageStore' interface accepts sorting and pagination inputs - If sorting by `datePublished`, add the property `published` set to true within the filter. Relevent `dwn-sdk-js` PRs: decentralized-identity/dwn-sdk-js#478 decentralized-identity/dwn-sdk-js#456 * upgrade kysley for tuple support * upgrade sdk and queries, first pass -- tests fail * sorting/pagination * store encodedData when available, sort and paginate according to messageCid cusror * added more comments * added comment about removing encodedData from message before encoding for storage * bump version * new release version * npm i after version bump
Starting point for #274
Modified the
DateSort
enum into the following types:This allows explicit types for Records sorting(by created or published) vs generic message sorting (by timestamp).
Currently RecordsQuery specify the default sort to be
RecordsDateSort.CreatedDescending
Everything else defaults to
TimestampDateSort.TimestampDescending
Either as a part of this effort or as a different PR there can be some enhancements to certain parts of the code that either look for only 1 message, or attempt to sort the message they're looking for via Message.getNewestMessage or getOldestMessage.