Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark CBOR encoding/decoding using mainnet data #783

Merged

Conversation

fxamacker
Copy link
Member

@fxamacker fxamacker commented Apr 8, 2021

Closes #765
Updates #738

Description

Create benchmarks that use testdata/*.cbor.gz files.

Add initial 6 *.cbor.gz files using data from a mainnet snapshot.

  • Filenames currently include v3 or v2 to indicate version (v4 is default but can be included in the name).
  • The 16 character hex portion is a truncated SHA256 of the .cbor.gz file.

For safety:

  • limit number of files processed (currently 100).
  • use io.LimitReader to limit the max bytes uncompressed from each file (currently 16 MiB).

Caveats

The SHA256 isn't verified by the benchmark. We can do that or we can do it manually using sha256sum command and an extra file if desired.

I chose SHA256 because it's included in Go.


For contributor use:

  • Targeted PR against feature/storage-optimizations branch
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work
  • Code follows the standards mentioned here
  • Updated relevant documentation
  • Re-reviewed Files changed in the Github PR explorer
  • Added appropriate labels

Create benchmarks that use testdata/*.cbor.gz files.

Add initial 6 *.cbor.gz files using data from a mainnet snapshot.

For safety, limit number of files processed and use io.LimitReader to
limit the max bytes uncompressed from each file.

Closes onflow#765
Use the current year for the copyright because it's a new file.
Copy link
Member

@turbolent turbolent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

@turbolent turbolent merged commit f04b15d into onflow:feature/storage-optimizations Apr 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants