Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize btcwire Serialization #27

Closed
davecgh opened this issue Oct 29, 2013 · 1 comment
Closed

Optimize btcwire Serialization #27

davecgh opened this issue Oct 29, 2013 · 1 comment
Labels
optimization

Comments

@davecgh
Copy link
Member

davecgh commented Oct 29, 2013

Latest profiling since the last round of optimizations has shown that 3rd most time consuming process, at 7% of the overall processing time, is being spent doing serialization (mostly for the purposes of hashing).

Performing some benchmarks has shown the binary.Write and binary.Read are significantly slower (due to using reflection and interfaces) than serializing directly into a buffer and writing it. They also create additional internal buffers to do their work, so it results in a lot of extra allocations which means more garbage and GC cycles.

For example, the following benchmark results of serializing a transaction using the aforementioned techniques show it's roughly 10x faster. Considering the sheer number of transactions in the block chain that are serialized and the fact they are serialized more than once per transaction, this adds up.

BenchmarkWriteTx          500000              5437 ns/op
BenchmarkWriteTxNew      5000000               505 ns/op
davecgh added a commit to btcsuite/btcwire that referenced this issue Oct 31, 2013
@davecgh
Slightly optimize writeVarInt for the normal case.
cbf648a 
Most variable length integers are smaller numbers, so this commit reverses
the order of the if checks in the writeVarInt to assume smaller numbers
are more common.

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27
davecgh added a commit to btcsuite/btcwire that referenced this issue Oct 31, 2013
@davecgh
Expose new SerializeSize API for transactions.
26cb71d 
This commit adds a new function named SerializeSize to the public API for
MsgTx, TxOut, and TxIn which can be used to determine how many bytes the
serialized data would take without having to actually serialize it.

The following benchmark shows the difference between using the new
function to get the serialize size for a typical transaction and
serializing into a temporary buffer and taking the length of it:

Bufffer: BenchmarkTxSerializeSizeBuffer     200000           7050 ns/op
New:     BenchmarkTxSerializeSizeNew     100000000             18 ns/op

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 5, 2013
@davecgh
Optimize writeVarInt.
50c10fa 
Before:

BenchmarkWriteVarInt1   10000000               173 ns/op
BenchmarkWriteVarInt3    2000000               965 ns/op
BenchmarkWriteVarInt5    2000000               966 ns/op
BenchmarkWriteVarInt9    2000000               968 ns/op

After:

BenchmarkWriteVarInt1   20000000               101 ns/op
BenchmarkWriteVarInt3   20000000               136 ns/op
BenchmarkWriteVarInt5   10000000               142 ns/op
BenchmarkWriteVarInt9   10000000               156 ns/op

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 5, 2013
@davecgh
Slightly optimize readVarInt.
6263efc 
This commit slightly optimizes the readVarInt function in the case of
multiple-byte variable length integers.  It also reduces the amount of
memory garbage it generates.

Before:

BenchmarkReadVarInt1     5000000               386 ns/op
BenchmarkReadVarInt3     5000000               693 ns/op
BenchmarkReadVarInt5     2000000               793 ns/op
BenchmarkReadVarInt9     5000000               709 ns/op

After:

BenchmarkReadVarInt1     5000000               387 ns/op
BenchmarkReadVarInt3     5000000               471 ns/op
BenchmarkReadVarInt5     5000000               575 ns/op
BenchmarkReadVarInt9     5000000               473 ns/op

This is part ef the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 5, 2013
@davecgh
Optimize readVarString.
4002051 
Before:

BenchmarkReadVarStr4     1000000              1698 ns/op
BenchmarkReadVarStr10    1000000              1812 ns/op

After:

BenchmarkReadVarStr4     2000000               853 ns/op
BenchmarkReadVarStr10    5000000               712 ns/op

This is part ef the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize writeVarString.
8a1828a 
Before:

BenchmarkWriteVarStr4    1000000              1114 ns/op
BenchmarkWriteVarStr10   1000000              1352 ns/op

After:

BenchmarkWriteVarStr4    5000000               291 ns/op
BenchmarkWriteVarStr10  10000000               248 ns/op

This is part ef the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize writeOutPoint.
f54b010 
Before: BenchmarkWriteOutPoint    500000              2664 ns/op
After:  BenchmarkWriteOutPoint  10000000               151 ns/op

This is part ef the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize readOutPoint.
73f4ee6 
Before: BenchmarkReadOutPoint     500000              2946 ns/op
After:  BenchmarkReadOutPoint    5000000               582 ns/op

This is part ef the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize writeTxOut.
a7e3ee6 
Before: BenchmarkWriteTxOut       500000              4050 ns/op
After:  BenchmarkWriteTxOut     10000000               248 ns/op

This is part ef the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize readTxOut.
cece305 
Before: BenchmarkReadTxOut        500000              4576 ns/op
After:  BenchmarkReadTxOut       2000000               871 ns/op

This is part ef the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize readTxIn.
59da692 
Before: BenchmarkReadTxIn        1000000              2435 ns/op
After:  BenchmarkReadTxIn        1000000              1427 ns/op

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize writeTxIn.
1a1f938 
Before: BenchmarkWriteTxIn       5000000               422 ns/op
After:  BenchmarkWriteTxIn       5000000               389 ns/op

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize transaction Serialize.
a81e081 
The benchmark results for the current commit:

Before: BenchmarkSerializeTx     1000000              1233 ns/op
After:  BenchmarkSerializeTx     1000000              1083 ns/op

The cumulative benchmark results since commit b7b700f:

Before: BenchmarkSerializeTx     1000000              5437 ns/op
After:  BenchmarkSerializeTx     1000000              1083 ns/op

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize transaction Deserialize.
9b7e352 
The benchmark results for the current commit:

Before: BenchmarkDeserializeTx    500000              4018 ns/op
After:  BenchmarkDeserializeTx    500000              3780 ns/op

The cumulative benchmark results since commit b7b700f:

Before: BenchmarkDeserializeTx    200000             10665 ns/op
After:  BenchmarkDeserializeTx    500000              3780 ns/op

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 6, 2013
@davecgh
Optimize readElement.
bc85a31 
The following benchmark results show the results for deserializing a block header:

Before: BenchmarkReadBlockHeader          500000              5916 ns/op
After:  BenchmarkReadBlockHeader         1000000              2078 ns/op

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
davecgh added a commit to btcsuite/btcwire that referenced this issue Nov 7, 2013
@davecgh
Optimize writeElement.
9ee6a8a 
This commit modifies the writeElement function to have a "fast path" which uses type
assertions for all of the types which btcwire write so the more expensive
reflection-based binary.Write can be avoided.

Also, this changes all cases that were writing raw ShaHash (32-byte) arrays (which
requires a stack copy) instead simply passing the pointer.

The following benchmark results show the results for serializing a block header
after these changes:

Before: BenchmarkWriteBlockHeader         500000              5566 ns/op
After:  BenchmarkWriteBlockHeader        1000000               991 ns/op

This is part of the ongoing effort to optimize serialization as noted in
btcsuite/btcd#27.
@davecgh
Copy link
Member Author

davecgh commented Nov 7, 2013

This has been completed as of commit btcsuite/btcwire@6f61e0a

@davecgh davecgh closed this as completed Nov 7, 2013
@davecgh davecgh mentioned this issue Nov 7, 2013
Closed
gsalgado pushed a commit to gsalgado/btcd that referenced this issue Jul 29, 2014
@jrick
Add listaddresstransactions extension.
5bf8f89 
Fixes btcsuite#27.
jrick pushed a commit to jrick/btcd that referenced this issue Oct 12, 2017
@jcvernaleo
Remove -v from go test.
ac946ab 
This caused unhelpful clutter in travis output.

Closes btcsuite#27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
optimization
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@davecgh