-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(filecoin-api): paginated queries #1521
Conversation
InclusionRecordQueryByGroup | ||
> | ||
export type InclusionStore = Store<InclusionRecordKey, InclusionRecord> & | ||
QueryableStore<InclusionRecordQueryByGroup, InclusionRecord> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I flattened the store types to give us more flexibility and simplicity. They can be composed as shown and we now have the ability to express in types a read only store for querying data (for example).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 here for composable types :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Overall this looks really good. Appreciate the type changes and the paging interface looks really good. Do we have adequate integration testing to insure a safe deploy?
InclusionRecordQueryByGroup | ||
> | ||
export type InclusionStore = Store<InclusionRecordKey, InclusionRecord> & | ||
QueryableStore<InclusionRecordQueryByGroup, InclusionRecord> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 here for composable types :)
🤖 I have created a release *beep* *boop* --- ## [7.2.0](filecoin-api-v7.1.1...filecoin-api-v7.2.0) (2024-07-23) ### Features * **filecoin-api:** paginated queries ([#1521](#1521)) ([25ed7d7](25ed7d7)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
🤖 I have created a release *beep* *boop* --- ## 1.0.0 (2024-12-02) ### ⚠ BREAKING CHANGES * **upload-api:** integrate agent store for idempotence & invocation/receipt persistence ([storacha#1444](https://github.com/storacha/upload-service/issues/1444)) * dataStore in storefront renamed to contentStore * not possible to skip submit queue on storefront service anymore * return allocated bytes in `store/add` receipt ([storacha#1213](https://github.com/storacha/upload-service/issues/1213)) ### Features * add blob protocol to upload-client ([storacha#1425](https://github.com/storacha/upload-service/issues/1425)) ([49aef56](49aef56)) * add support to prepend pieces while buffering to aggregate ([storacha#1301](https://github.com/storacha/upload-service/issues/1301)) ([dff1846](dff1846)) * add usage/report capability ([storacha#1079](https://github.com/storacha/upload-service/issues/1079)) ([6418b4b](6418b4b)) * aggregator keeping oldest piece ts ([storacha#1188](https://github.com/storacha/upload-service/issues/1188)) ([97a7def](97a7def)) * api waits for trigger filecoin pipeline from the client ([storacha#1332](https://github.com/storacha/upload-service/issues/1332)) ([421bacb](421bacb)) * filecoin info ([storacha#1091](https://github.com/storacha/upload-service/issues/1091)) ([adb2442](adb2442)) * **filecoin-api:** allow custom hashing function to be passed to aggregate builder ([storacha#1553](https://github.com/storacha/upload-service/issues/1553)) ([e2653d4](e2653d4)) * **filecoin-api:** paginated queries ([storacha#1521](https://github.com/storacha/upload-service/issues/1521)) ([25ed7d7](25ed7d7)) * move aggregate information out of deals in filecoin/info ([storacha#1192](https://github.com/storacha/upload-service/issues/1192)) ([18dc590](18dc590)) * return allocated bytes in `store/add` receipt ([storacha#1213](https://github.com/storacha/upload-service/issues/1213)) ([5d52e44](5d52e44)) * router ([#11](#11)) ([c810735](c810735)) * upgrade ucanto/transport to 9.1.0 in all packages to get more verbose errors from HTTP transport on non-ok response ([storacha#1312](https://github.com/storacha/upload-service/issues/1312)) ([d6978d7](d6978d7)) * **upload-api:** integrate agent store for idempotence & invocation/receipt persistence ([storacha#1444](https://github.com/storacha/upload-service/issues/1444)) ([c9bf33e](c9bf33e)) * upload-client uploadDirectory, by default, sorts the provided files by file name to help the user call us in a way that is deterministic and minimizes cost ([storacha#1173](https://github.com/storacha/upload-service/issues/1173)) ([8cd2555](8cd2555)) * use digest in `blob/accept` location commitment ([storacha#1480](https://github.com/storacha/upload-service/issues/1480)) ([ade45eb](ade45eb)) ### Fixes * aggregate offer invocation cid wrong ([storacha#1063](https://github.com/storacha/upload-service/issues/1063)) ([90a5a4d](90a5a4d)) * avoid duplicates on aggregator buffer concat ([storacha#1259](https://github.com/storacha/upload-service/issues/1259)) ([9e64bab](9e64bab)) * check service did in w3filecoin ([storacha#1476](https://github.com/storacha/upload-service/issues/1476)) ([11b00bf](11b00bf)) * configure max pieces ([storacha#1566](https://github.com/storacha/upload-service/issues/1566)) ([71674ed](71674ed)) * dealer offer store keys without space ([storacha#1066](https://github.com/storacha/upload-service/issues/1066)) ([301f411](301f411)) * drop filecoin storefront skip submit queue option ([storacha#1371](https://github.com/storacha/upload-service/issues/1371)) ([1114383](1114383)) * enable storefront signer to be different from main service signer ([storacha#1072](https://github.com/storacha/upload-service/issues/1072)) ([21ded3c](21ded3c)) * filecoin test use blob ([storacha#1422](https://github.com/storacha/upload-service/issues/1422)) ([359c0b7](359c0b7)) * **filecoin-api:** parallel put to piece accept queue ([storacha#1560](https://github.com/storacha/upload-service/issues/1560)) ([e7cbb6d](e7cbb6d)) * issue where typedoc docs would only show full docs for w3up-client ([storacha#1141](https://github.com/storacha/upload-service/issues/1141)) ([0b8d3f3](0b8d3f3)) * lint ([storacha#1095](https://github.com/storacha/upload-service/issues/1095)) ([f9cc770](f9cc770)) * migrate repo ([storacha#1389](https://github.com/storacha/upload-service/issues/1389)) ([475a287](475a287)) * package metadata ([storacha#1161](https://github.com/storacha/upload-service/issues/1161)) ([b8a1cc2](b8a1cc2)) * receipt chain has wrong CID because no expiration is set ([storacha#1060](https://github.com/storacha/upload-service/issues/1060)) ([dfb46d8](dfb46d8)) * rename blob and index client capabilities ([storacha#1478](https://github.com/storacha/upload-service/issues/1478)) ([17e3a31](17e3a31)) * repo URLs ([storacha#1550](https://github.com/storacha/upload-service/issues/1550)) ([e02ddf3](e02ddf3)) * return piece accept receipt error ([storacha#1512](https://github.com/storacha/upload-service/issues/1512)) ([05283cf](05283cf)) * revert enable storefront signer to be different from main service signer ([storacha#1075](https://github.com/storacha/upload-service/issues/1075)) ([80cdde0](80cdde0)) * storefront content store rename and separation for test ([storacha#1409](https://github.com/storacha/upload-service/issues/1409)) ([05e5db3](05e5db3)) * storefront events cron with max concurrency ([storacha#1191](https://github.com/storacha/upload-service/issues/1191)) ([11010c9](11010c9)) * upgrade ucanto core ([storacha#1127](https://github.com/storacha/upload-service/issues/1127)) ([5ce4d22](5ce4d22)) * upgrade ucanto libs and format filecoin api ([storacha#1359](https://github.com/storacha/upload-service/issues/1359)) ([87ca098](87ca098)) * upload API test fixes ([6b0d72d](6b0d72d)) * use one-webcrypto from npm ([storacha#1525](https://github.com/storacha/upload-service/issues/1525)) ([9345c54](9345c54)) ### Other Changes * Add `pnpm dev` to watch-build all packages ([storacha#1533](https://github.com/storacha/upload-service/issues/1533)) ([07970ef](07970ef)) * appease linter ([782c6d0](782c6d0)) * **main:** release filecoin-api 4.0.2 ([storacha#1061](https://github.com/storacha/upload-service/issues/1061)) ([5918ff2](5918ff2)) * **main:** release filecoin-api 4.0.3 ([storacha#1064](https://github.com/storacha/upload-service/issues/1064)) ([ef4ce7a](ef4ce7a)) * **main:** release filecoin-api 4.0.4 ([storacha#1067](https://github.com/storacha/upload-service/issues/1067)) ([c3ed3b9](c3ed3b9)) * **main:** release filecoin-api 4.0.5 ([storacha#1073](https://github.com/storacha/upload-service/issues/1073)) ([9cf2fc0](9cf2fc0)) * **main:** release filecoin-api 4.0.6 ([storacha#1077](https://github.com/storacha/upload-service/issues/1077)) ([40d0e7a](40d0e7a)) * **main:** release filecoin-api 4.1.0 ([storacha#1085](https://github.com/storacha/upload-service/issues/1085)) ([9b752f9](9b752f9)) * **main:** release filecoin-api 4.1.1 ([storacha#1133](https://github.com/storacha/upload-service/issues/1133)) ([3621b42](3621b42)) * **main:** release filecoin-api 4.1.2 ([storacha#1148](https://github.com/storacha/upload-service/issues/1148)) ([fa6405d](fa6405d)) * **main:** release filecoin-api 4.2.0 ([storacha#1164](https://github.com/storacha/upload-service/issues/1164)) ([dfdf762](dfdf762)) * **main:** release filecoin-api 4.3.0 ([storacha#1194](https://github.com/storacha/upload-service/issues/1194)) ([a1aa231](a1aa231)) * **main:** release filecoin-api 4.3.1 ([storacha#1205](https://github.com/storacha/upload-service/issues/1205)) ([7f42f39](7f42f39)) * **main:** release filecoin-api 4.4.0 ([storacha#1303](https://github.com/storacha/upload-service/issues/1303)) ([bb74ffc](bb74ffc)) * **main:** release filecoin-api 4.5.0 ([storacha#1314](https://github.com/storacha/upload-service/issues/1314)) ([d31ff06](d31ff06)) * **main:** release filecoin-api 4.6.0 ([storacha#1346](https://github.com/storacha/upload-service/issues/1346)) ([4b166fd](4b166fd)) * **main:** release filecoin-api 4.6.1 ([storacha#1363](https://github.com/storacha/upload-service/issues/1363)) ([efcb912](efcb912)) * **main:** release filecoin-api 5.0.0 ([storacha#1384](https://github.com/storacha/upload-service/issues/1384)) ([cb5b628](cb5b628)) * **main:** release filecoin-api 5.0.1 ([storacha#1398](https://github.com/storacha/upload-service/issues/1398)) ([2e3ef61](2e3ef61)) * **main:** release filecoin-api 6.0.0 ([storacha#1414](https://github.com/storacha/upload-service/issues/1414)) ([939e839](939e839)) * **main:** release filecoin-api 6.0.1 ([storacha#1424](https://github.com/storacha/upload-service/issues/1424)) ([f8c6c1d](f8c6c1d)) * **main:** release filecoin-api 7.0.0 ([storacha#1430](https://github.com/storacha/upload-service/issues/1430)) ([604d300](604d300)) * **main:** release filecoin-api 7.1.0 ([storacha#1481](https://github.com/storacha/upload-service/issues/1481)) ([9d38080](9d38080)) * **main:** release filecoin-api 7.1.1 ([storacha#1513](https://github.com/storacha/upload-service/issues/1513)) ([d4f5f45](d4f5f45)) * **main:** release filecoin-api 7.2.0 ([storacha#1522](https://github.com/storacha/upload-service/issues/1522)) ([0aebf9f](0aebf9f)) * **main:** release filecoin-api 7.2.1 ([storacha#1528](https://github.com/storacha/upload-service/issues/1528)) ([5b8e148](5b8e148)) * **main:** release filecoin-api 7.3.0 ([storacha#1540](https://github.com/storacha/upload-service/issues/1540)) ([77b31e9](77b31e9)) * **main:** release filecoin-api 7.3.1 ([storacha#1561](https://github.com/storacha/upload-service/issues/1561)) ([61d408a](61d408a)) * **main:** release filecoin-api 7.3.2 ([storacha#1567](https://github.com/storacha/upload-service/issues/1567)) ([534d3be](534d3be)) * package renames ([0f797ed](0f797ed)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This PR updates filecoin-api to add pagination to the query interfaces. Currently only the first page of results is returned.
This means in the fileocin pipeline cron jobs we never process more than 1 page of items, even though there are sometimes many more items to process and ample execution time remaining in the lambda.
If only the first page of items is processed it gives a false sense of everything operating ok, when actually things may be backing up. It essentially smothers the issue. If the lambda times out because there are too many items to process then this is a good indication that it needs some attention.
Essentially, the interface changes from:
To this:
Context: we currently have a lot of
filecoin/accept
receipts we need to issue that are being done in a cron job.