Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase performance for precache operation #45

Merged
merged 5 commits into from
Oct 3, 2023

Conversation

junderw
Copy link
Member

@junderw junderw commented Sep 17, 2023

Flushing after every call is slow.

How slow?

Extremely slow. The performance (highly dependent on the physical media the disk uses) on my 5000MB/s NVMe SSD is about 2x.

Cuts the time to precache in half.

I took this opportunity to re-activate the UtxoCache caching.

Also, we should generate the popular scripts txt on each startup in prod. And I moved the precache into the background so the server can still serve requests.

TODO:

  1. Merge Fix: Fix Stats and Utxo cache #44 (This will make sure the precache is clean. No negative balances etc.)

@junderw junderw marked this pull request as draft September 17, 2023 15:35
@junderw junderw force-pushed the junderw/perf-precache-speed branch from c2b6f81 to 8332760 Compare September 18, 2023 23:56
@junderw junderw changed the title WIP: Increase performance for precache operation Increase performance for precache operation Sep 18, 2023
@junderw junderw marked this pull request as ready for review September 18, 2023 23:57
@junderw junderw force-pushed the junderw/perf-precache-speed branch 2 times, most recently from 53b46cc to d5d2573 Compare September 19, 2023 02:12
@junderw
Copy link
Member Author

junderw commented Sep 19, 2023

I rebased on top of #44 since we really need to merge that before this.

@junderw junderw force-pushed the junderw/perf-precache-speed branch from d5d2573 to 811c6c4 Compare September 19, 2023 03:47
@junderw
Copy link
Member Author

junderw commented Sep 19, 2023

  1. Moved precache to background, it starts after the REST server starts.
  2. Added runtime parameter for deciding how many threads to use.
  3. The prod defaults for threads is 100, which will give it plenty of space to answer requests (though they may be slower)

@wiz wiz force-pushed the junderw/perf-precache-speed branch from 9c261eb to 69c97c1 Compare October 3, 2023 04:21
Copy link
Member

@wiz wiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested ACK @ d49f752

@wiz wiz merged commit 18e041c into mempool Oct 3, 2023
7 checks passed
@wiz wiz deleted the junderw/perf-precache-speed branch October 3, 2023 05:53
SatoKentaNayoro pushed a commit to boolnetwork/mempool-electrs that referenced this pull request Nov 29, 2024
Increase performance for precache operation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants