Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(filecoin-api): paginated queries #1521

Merged
merged 5 commits into from
Jul 23, 2024
Merged

feat(filecoin-api): paginated queries #1521

merged 5 commits into from
Jul 23, 2024

Conversation

alanshaw
Copy link
Member

@alanshaw alanshaw commented Jul 22, 2024

This PR updates filecoin-api to add pagination to the query interfaces. Currently only the first page of results is returned.

This means in the fileocin pipeline cron jobs we never process more than 1 page of items, even though there are sometimes many more items to process and ample execution time remaining in the lambda.

If only the first page of items is processed it gives a false sense of everything operating ok, when actually things may be backing up. It essentially smothers the issue. If the lambda times out because there are too many items to process then this is a good indication that it needs some attention.

Essentially, the interface changes from:

interface Queryable<Q, T> {
  query (q: Q): Promise<Result<T[]>>
}

To this:

interface Pageable {
  cursor?: string
  size?: number
}

interface ListSuccess<T> {
  results: T[]
  cursor?: string
}

interface Queryable<Q, T> {
  query (q: Q, options?: Pageable): Promise<Result<ListSuccess<T>>>
}

Context: we currently have a lot of filecoin/accept receipts we need to issue that are being done in a cron job.

InclusionRecordQueryByGroup
>
export type InclusionStore = Store<InclusionRecordKey, InclusionRecord> &
QueryableStore<InclusionRecordQueryByGroup, InclusionRecord>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I flattened the store types to give us more flexibility and simplicity. They can be composed as shown and we now have the ability to express in types a read only store for querying data (for example).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 here for composable types :)

Copy link
Member

@hannahhoward hannahhoward left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Overall this looks really good. Appreciate the type changes and the paging interface looks really good. Do we have adequate integration testing to insure a safe deploy?

InclusionRecordQueryByGroup
>
export type InclusionStore = Store<InclusionRecordKey, InclusionRecord> &
QueryableStore<InclusionRecordQueryByGroup, InclusionRecord>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 here for composable types :)

@alanshaw
Copy link
Member Author

Do we have adequate integration testing to insure a safe deploy?

Yeah, I'm confident this is safe to deploy. We already have tests that cover the code that uses .query(...). I added coverage reporting to double check:

Screenshot 2024-07-23 at 10 30 52 Screenshot 2024-07-23 at 10 29 54 Screenshot 2024-07-23 at 10 28 35

@alanshaw alanshaw merged commit 25ed7d7 into main Jul 23, 2024
15 checks passed
@alanshaw alanshaw deleted the feat/paginated-queries branch July 23, 2024 09:34
alanshaw pushed a commit that referenced this pull request Jul 23, 2024
🤖 I have created a release *beep* *boop*
---


##
[7.2.0](filecoin-api-v7.1.1...filecoin-api-v7.2.0)
(2024-07-23)


### Features

* **filecoin-api:** paginated queries
([#1521](#1521))
([25ed7d7](25ed7d7))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Peeja added a commit to storacha/upload-service that referenced this pull request Dec 2, 2024
🤖 I have created a release *beep* *boop*
---


## 1.0.0 (2024-12-02)


### ⚠ BREAKING CHANGES

* **upload-api:** integrate agent store for idempotence &
invocation/receipt persistence
([storacha#1444](https://github.com/storacha/upload-service/issues/1444))
* dataStore in storefront renamed to contentStore
* not possible to skip submit queue on storefront service anymore
* return allocated bytes in `store/add` receipt
([storacha#1213](https://github.com/storacha/upload-service/issues/1213))

### Features

* add blob protocol to upload-client
([storacha#1425](https://github.com/storacha/upload-service/issues/1425))
([49aef56](49aef56))
* add support to prepend pieces while buffering to aggregate
([storacha#1301](https://github.com/storacha/upload-service/issues/1301))
([dff1846](dff1846))
* add usage/report capability
([storacha#1079](https://github.com/storacha/upload-service/issues/1079))
([6418b4b](6418b4b))
* aggregator keeping oldest piece ts
([storacha#1188](https://github.com/storacha/upload-service/issues/1188))
([97a7def](97a7def))
* api waits for trigger filecoin pipeline from the client
([storacha#1332](https://github.com/storacha/upload-service/issues/1332))
([421bacb](421bacb))
* filecoin info
([storacha#1091](https://github.com/storacha/upload-service/issues/1091))
([adb2442](adb2442))
* **filecoin-api:** allow custom hashing function to be passed to
aggregate builder
([storacha#1553](https://github.com/storacha/upload-service/issues/1553))
([e2653d4](e2653d4))
* **filecoin-api:** paginated queries
([storacha#1521](https://github.com/storacha/upload-service/issues/1521))
([25ed7d7](25ed7d7))
* move aggregate information out of deals in filecoin/info
([storacha#1192](https://github.com/storacha/upload-service/issues/1192))
([18dc590](18dc590))
* return allocated bytes in `store/add` receipt
([storacha#1213](https://github.com/storacha/upload-service/issues/1213))
([5d52e44](5d52e44))
* router ([#11](#11))
([c810735](c810735))
* upgrade ucanto/transport to 9.1.0 in all packages to get more verbose
errors from HTTP transport on non-ok response
([storacha#1312](https://github.com/storacha/upload-service/issues/1312))
([d6978d7](d6978d7))
* **upload-api:** integrate agent store for idempotence &
invocation/receipt persistence
([storacha#1444](https://github.com/storacha/upload-service/issues/1444))
([c9bf33e](c9bf33e))
* upload-client uploadDirectory, by default, sorts the provided files by
file name to help the user call us in a way that is deterministic and
minimizes cost
([storacha#1173](https://github.com/storacha/upload-service/issues/1173))
([8cd2555](8cd2555))
* use digest in `blob/accept` location commitment
([storacha#1480](https://github.com/storacha/upload-service/issues/1480))
([ade45eb](ade45eb))


### Fixes

* aggregate offer invocation cid wrong
([storacha#1063](https://github.com/storacha/upload-service/issues/1063))
([90a5a4d](90a5a4d))
* avoid duplicates on aggregator buffer concat
([storacha#1259](https://github.com/storacha/upload-service/issues/1259))
([9e64bab](9e64bab))
* check service did in w3filecoin
([storacha#1476](https://github.com/storacha/upload-service/issues/1476))
([11b00bf](11b00bf))
* configure max pieces
([storacha#1566](https://github.com/storacha/upload-service/issues/1566))
([71674ed](71674ed))
* dealer offer store keys without space
([storacha#1066](https://github.com/storacha/upload-service/issues/1066))
([301f411](301f411))
* drop filecoin storefront skip submit queue option
([storacha#1371](https://github.com/storacha/upload-service/issues/1371))
([1114383](1114383))
* enable storefront signer to be different from main service signer
([storacha#1072](https://github.com/storacha/upload-service/issues/1072))
([21ded3c](21ded3c))
* filecoin test use blob
([storacha#1422](https://github.com/storacha/upload-service/issues/1422))
([359c0b7](359c0b7))
* **filecoin-api:** parallel put to piece accept queue
([storacha#1560](https://github.com/storacha/upload-service/issues/1560))
([e7cbb6d](e7cbb6d))
* issue where typedoc docs would only show full docs for w3up-client
([storacha#1141](https://github.com/storacha/upload-service/issues/1141))
([0b8d3f3](0b8d3f3))
* lint ([storacha#1095](https://github.com/storacha/upload-service/issues/1095))
([f9cc770](f9cc770))
* migrate repo
([storacha#1389](https://github.com/storacha/upload-service/issues/1389))
([475a287](475a287))
* package metadata
([storacha#1161](https://github.com/storacha/upload-service/issues/1161))
([b8a1cc2](b8a1cc2))
* receipt chain has wrong CID because no expiration is set
([storacha#1060](https://github.com/storacha/upload-service/issues/1060))
([dfb46d8](dfb46d8))
* rename blob and index client capabilities
([storacha#1478](https://github.com/storacha/upload-service/issues/1478))
([17e3a31](17e3a31))
* repo URLs
([storacha#1550](https://github.com/storacha/upload-service/issues/1550))
([e02ddf3](e02ddf3))
* return piece accept receipt error
([storacha#1512](https://github.com/storacha/upload-service/issues/1512))
([05283cf](05283cf))
* revert enable storefront signer to be different from main service
signer ([storacha#1075](https://github.com/storacha/upload-service/issues/1075))
([80cdde0](80cdde0))
* storefront content store rename and separation for test
([storacha#1409](https://github.com/storacha/upload-service/issues/1409))
([05e5db3](05e5db3))
* storefront events cron with max concurrency
([storacha#1191](https://github.com/storacha/upload-service/issues/1191))
([11010c9](11010c9))
* upgrade ucanto core
([storacha#1127](https://github.com/storacha/upload-service/issues/1127))
([5ce4d22](5ce4d22))
* upgrade ucanto libs and format filecoin api
([storacha#1359](https://github.com/storacha/upload-service/issues/1359))
([87ca098](87ca098))
* upload API test fixes
([6b0d72d](6b0d72d))
* use one-webcrypto from npm
([storacha#1525](https://github.com/storacha/upload-service/issues/1525))
([9345c54](9345c54))


### Other Changes

* Add `pnpm dev` to watch-build all packages
([storacha#1533](https://github.com/storacha/upload-service/issues/1533))
([07970ef](07970ef))
* appease linter
([782c6d0](782c6d0))
* **main:** release filecoin-api 4.0.2
([storacha#1061](https://github.com/storacha/upload-service/issues/1061))
([5918ff2](5918ff2))
* **main:** release filecoin-api 4.0.3
([storacha#1064](https://github.com/storacha/upload-service/issues/1064))
([ef4ce7a](ef4ce7a))
* **main:** release filecoin-api 4.0.4
([storacha#1067](https://github.com/storacha/upload-service/issues/1067))
([c3ed3b9](c3ed3b9))
* **main:** release filecoin-api 4.0.5
([storacha#1073](https://github.com/storacha/upload-service/issues/1073))
([9cf2fc0](9cf2fc0))
* **main:** release filecoin-api 4.0.6
([storacha#1077](https://github.com/storacha/upload-service/issues/1077))
([40d0e7a](40d0e7a))
* **main:** release filecoin-api 4.1.0
([storacha#1085](https://github.com/storacha/upload-service/issues/1085))
([9b752f9](9b752f9))
* **main:** release filecoin-api 4.1.1
([storacha#1133](https://github.com/storacha/upload-service/issues/1133))
([3621b42](3621b42))
* **main:** release filecoin-api 4.1.2
([storacha#1148](https://github.com/storacha/upload-service/issues/1148))
([fa6405d](fa6405d))
* **main:** release filecoin-api 4.2.0
([storacha#1164](https://github.com/storacha/upload-service/issues/1164))
([dfdf762](dfdf762))
* **main:** release filecoin-api 4.3.0
([storacha#1194](https://github.com/storacha/upload-service/issues/1194))
([a1aa231](a1aa231))
* **main:** release filecoin-api 4.3.1
([storacha#1205](https://github.com/storacha/upload-service/issues/1205))
([7f42f39](7f42f39))
* **main:** release filecoin-api 4.4.0
([storacha#1303](https://github.com/storacha/upload-service/issues/1303))
([bb74ffc](bb74ffc))
* **main:** release filecoin-api 4.5.0
([storacha#1314](https://github.com/storacha/upload-service/issues/1314))
([d31ff06](d31ff06))
* **main:** release filecoin-api 4.6.0
([storacha#1346](https://github.com/storacha/upload-service/issues/1346))
([4b166fd](4b166fd))
* **main:** release filecoin-api 4.6.1
([storacha#1363](https://github.com/storacha/upload-service/issues/1363))
([efcb912](efcb912))
* **main:** release filecoin-api 5.0.0
([storacha#1384](https://github.com/storacha/upload-service/issues/1384))
([cb5b628](cb5b628))
* **main:** release filecoin-api 5.0.1
([storacha#1398](https://github.com/storacha/upload-service/issues/1398))
([2e3ef61](2e3ef61))
* **main:** release filecoin-api 6.0.0
([storacha#1414](https://github.com/storacha/upload-service/issues/1414))
([939e839](939e839))
* **main:** release filecoin-api 6.0.1
([storacha#1424](https://github.com/storacha/upload-service/issues/1424))
([f8c6c1d](f8c6c1d))
* **main:** release filecoin-api 7.0.0
([storacha#1430](https://github.com/storacha/upload-service/issues/1430))
([604d300](604d300))
* **main:** release filecoin-api 7.1.0
([storacha#1481](https://github.com/storacha/upload-service/issues/1481))
([9d38080](9d38080))
* **main:** release filecoin-api 7.1.1
([storacha#1513](https://github.com/storacha/upload-service/issues/1513))
([d4f5f45](d4f5f45))
* **main:** release filecoin-api 7.2.0
([storacha#1522](https://github.com/storacha/upload-service/issues/1522))
([0aebf9f](0aebf9f))
* **main:** release filecoin-api 7.2.1
([storacha#1528](https://github.com/storacha/upload-service/issues/1528))
([5b8e148](5b8e148))
* **main:** release filecoin-api 7.3.0
([storacha#1540](https://github.com/storacha/upload-service/issues/1540))
([77b31e9](77b31e9))
* **main:** release filecoin-api 7.3.1
([storacha#1561](https://github.com/storacha/upload-service/issues/1561))
([61d408a](61d408a))
* **main:** release filecoin-api 7.3.2
([storacha#1567](https://github.com/storacha/upload-service/issues/1567))
([534d3be](534d3be))
* package renames
([0f797ed](0f797ed))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants