Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(filecoin-proofs): increase parallelism in window-post #1580

Merged
merged 1 commit into from
Mar 30, 2022
Merged

perf(filecoin-proofs): increase parallelism in window-post #1580

merged 1 commit into from
Mar 30, 2022

Conversation

someonegg
Copy link
Contributor

This improves the parallelism when generating merkle tree.

@cryptonemo
Copy link
Collaborator

Thanks for submitting, this, but this has been tried in the past and didn't appear to make a measurable difference: https://github.com/filecoin-project/rust-fil-proofs/pull/1286/files#diff-5bf9d3ebeba9a4d7a43ef35fae48b5c36e9e7fe50b4eec99515d8a1fc1f15010R360

The problem is that we're instantiating trees at this point, not building them in parallel. If you can show positive measurements around this (and describe the I/O system you're on), it would help.

@someonegg
Copy link
Contributor Author

NFS protocol

When the page cache of nfs-server becomes the bottleneck, time between "generate_window_post: start" and "vanilla_proofs:start" will increase from 1 minute to more than 6 minutes.

According to the io log, this stage needs to read the last 32 bytes of each last-tree file.

@cryptonemo
Copy link
Collaborator

Can you comment on the improved time with this change?

@someonegg
Copy link
Contributor Author

With this change, the time is reduced to about 1 minute.

@cryptonemo cryptonemo merged commit 8e77cf7 into filecoin-project:master Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants