Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add worker_threads support to backfilling #1

Open
2 tasks
ashayp22 opened this issue Jul 12, 2024 · 1 comment
Open
2 tasks

Add worker_threads support to backfilling #1

ashayp22 opened this issue Jul 12, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@ashayp22
Copy link
Contributor

Bun v1.1.13 added more support for worker_threads, which should now let us use worker_threads for parallelizing the unstake event backfilling.

Update

  • Retried with Bun v1.1.19, and now facing this error. The error is still being resolved in this PR.

TODO

  • Replace Promise.all with thread.js Thread Pool in createBackfiller.ts
  • Benchmark performance in threading vs promises
@ashayp22 ashayp22 added the enhancement New feature or request label Jul 12, 2024
@ashayp22
Copy link
Contributor Author

ashayp22 commented Aug 3, 2024 •
edited
Loading

#11 sped up backfilling with async/await. Threads will be helpful for computationally intense indexing, but no such indexers exist as of right now (both unstake-indexer and quark-indexer are I/O intensive). We should look into this enhancement based on if computationally intense indexers are created.

@ashayp22 ashayp22 mentioned this issue Aug 3, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ashayp22