Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize bloom filter #2588

Merged
merged 20 commits into from
Jan 8, 2024
Merged

Optimize bloom filter #2588

merged 20 commits into from
Jan 8, 2024

Conversation

StephenButtolph
Copy link
Contributor

@StephenButtolph StephenButtolph commented Jan 4, 2024

Why this should be merged

This PR improves the performance of handling bloom filters over the p2p network.


Old Marshal

BenchmarkMarshal-12    	   15477	     77311 ns/op	   39992 B/op	      19 allocs/op

New Marshal

BenchmarkMarshal-12    	  426568	      2802 ns/op	   12288 B/op	       1 allocs/op

Old Parse

BenchmarkParse-12    	   24679	     46319 ns/op	   13120 B/op	      17 allocs/op

New Parse

BenchmarkParse-12    	13572466	     88.68 ns/op	     112 B/op	       2 allocs/op

Old Contains

BenchmarkContains-12    	47176765	        25.04 ns/op	       0 B/op	       0 allocs/op

New Contains

BenchmarkContains-12    	48683229	        24.56 ns/op	       0 B/op	       0 allocs/op

How this works

Note: This PR is not backwards compatible. Updated nodes will drop requested from un-updated nodes (and visa versa).

The major improvements from this PR come from:

  • pre-allocating the full byte slice for Marshal
  • converting entries from []uint64 to []byte to avoid needing to allocate a potentially large entries array and then individually pack entries into the array in Parse.
  • removal of an unused bitmask in Contains

How this was tested

  • CI
  • Fuji

@StephenButtolph StephenButtolph added the enhancement New feature or request label Jan 4, 2024
@StephenButtolph StephenButtolph added this to the v1.10.18 milestone Jan 4, 2024
@StephenButtolph StephenButtolph self-assigned this Jan 4, 2024
// positive probability of a bloom filter with [numEntries] after [count]
// additions.
//
// It is guaranteed to return a value in the range [minHashes, maxHashes].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

Copy link
Contributor

@patrick-ogrady patrick-ogrady left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressing my nits in #2591

pubsub/bloom/filter.go Outdated Show resolved Hide resolved
Copy link
Contributor

@dhrubabasu dhrubabasu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🏆 - left small comments

@StephenButtolph StephenButtolph added this pull request to the merge queue Jan 8, 2024
Merged via the queue into dev with commit bba8e75 Jan 8, 2024
17 checks passed
@StephenButtolph StephenButtolph deleted the optimize-bloom-filter branch January 8, 2024 03:38
chand1012 pushed a commit to multisig-labs/avalanchego that referenced this pull request Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants