Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix!: replace dag walkers with generic CID extraction from blocks #447

Merged
merged 11 commits into from
Sep 16, 2024

Conversation

achingbrain
Copy link
Member

Replace the codec-specific .dagWalkers property with a generic dag walker internally that uses the Block interface from the multicodecs module.

  • Removes the .dagWalkers property from the Helia interface
  • Adds getCodec and getHasher to retrieve codecs and hashers by code
  • Adds loadCodec and loadHasher options to allow sync or async loading of extra codecs/hashes in addition to staticlly configured ones in the codecs/hashers keys

BREAKING CHANGE: the .dagWalkers property has been removed

Change checklist

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation if necessary (this includes comments as well)
  • I have added tests that prove my fix is effective or that my feature works

Replace the codec-specific `.dagWalkers` property with a generic
dag walker internally that uses the `Block` interface from the
`multicodecs` module.

- Removes the `.dagWalkers` property from the Helia interface
- Adds `getCodec` and `getHasher` to retrieve codecs and hashers by code
- Adds `loadCodec` and `loadHasher` options to allow sync or async loading of extra codecs/hashes in addition to staticlly configured ones in the `codecs`/`hashers` keys

BREAKING CHANGE: the `.dagWalkers` property has been removed
@achingbrain achingbrain requested a review from a team as a code owner February 22, 2024 18:21
@achingbrain
Copy link
Member Author

I think the pinning benchmarks should be re-run with this PR before merging. The multiformats/block .links() function works by deserializing a block into an object, then recursively walking every property and yielding any value that can be turned into a CID.

This could potentially be more expensive than deserializing a block, collecting CIDs during deserialization and then yielding them all as the DAGWalkers do.

Copy link
Member

@SgtPooki SgtPooki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few things that could be changed, not sure if the networked-storage accepting an additional arg was done explicitly for some reason though

packages/car/src/index.ts Show resolved Hide resolved
packages/utils/src/index.ts Outdated Show resolved Hide resolved
packages/utils/src/utils/networked-storage.ts Outdated Show resolved Hide resolved
packages/utils/src/utils/networked-storage.ts Outdated Show resolved Hide resolved
packages/utils/src/utils/get-codec.ts Show resolved Hide resolved
packages/utils/src/utils/get-hasher.ts Show resolved Hide resolved
packages/utils/src/utils/get-hasher.ts Show resolved Hide resolved
packages/utils/src/utils/is-promise.ts Show resolved Hide resolved
packages/utils/test/fixtures/create-dag.ts Show resolved Hide resolved
packages/utils/test/utils/networked-storage.spec.ts Outdated Show resolved Hide resolved
@achingbrain achingbrain force-pushed the fix/remove-dag-walkers branch from 9bb948d to c31f5e1 Compare April 3, 2024 10:47
@SgtPooki
Copy link
Member

SgtPooki commented Aug 1, 2024

we need to run benchmarks to ensure no perf regression

@achingbrain achingbrain force-pushed the fix/remove-dag-walkers branch from da1386a to d071a30 Compare September 13, 2024 17:40
@achingbrain achingbrain changed the base branch from main to deps/update-libp2p-2.x.x September 13, 2024 17:40
@achingbrain achingbrain mentioned this pull request Sep 13, 2024
5 tasks
Base automatically changed from deps/update-libp2p-2.x.x to main September 15, 2024 11:08
An error occurred while trying to automatically change base from deps/update-libp2p-2.x.x to main September 15, 2024 11:08
@achingbrain
Copy link
Member Author

Perf seems fine - the benchmark added by #633 shows comparable performance between this branch and main.

@achingbrain achingbrain merged commit 5ff6998 into main Sep 16, 2024
18 checks passed
@achingbrain achingbrain deleted the fix/remove-dag-walkers branch September 16, 2024 08:45
@achingbrain achingbrain mentioned this pull request Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants