Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Prevector Optimizations #2083

Merged
merged 4 commits into from
Dec 20, 2020
Merged

[Core] Prevector Optimizations #2083

merged 4 commits into from
Dec 20, 2020

Conversation

random-zebra
Copy link

Backports from Bitcoin a series of optimizations for prevector operations, especially around resizing, which takes a significative portion of the time spent by of ReadBlockFromDisk().

Running the benchmark tool before the patch:

#Benchmark,                    count,  min,    max,    avg,    minc,   maxc,   avgc
-----------                    ------  ----    ----    ----    -----   -----   ----
PrevectorClearTrivial,         3328,   312631, 329431, 320311, 812854, 856536, 832823
PrevectorDestructorTrivial,    3328,   317011, 353344, 325141, 824243, 918711, 845382
PrevectorResizeTrivial,        3328,   321333, 327672, 324713, 835482, 851963, 844269

and after:

#Benchmark,                    count, min,    max,    avg,    minc,   maxc,   avgc
-----------                    ------ ----    ----    ----    -----   -----   ----
PrevectorClearTrivial,         65536, 15726,  16368,  16067,  40889,  42559,  41775
PrevectorDestructorTrivial,    65536, 15865,  16416,  16141,  41249,  42684,  41968
PrevectorResizeTrivial,        65536, 16036,  16320,  16213,  41696,  42433,  42156

shows that prevector::clear(), prevector::~prevector(), and prevector::resize() become (on avg) 20X faster.

Based on:

random-zebra and others added 4 commits December 19, 2020 18:25
@random-zebra
Add new prevector benchmarks.
e6741d0 
>>> backports bitcoin/bitcoin@f0e7aa7

This prepares for a series of two additional commits which optimize
prevector performance.
@random-zebra
Reduce redundant code of prevector and speed it up
c412d5f 
>>> based on bitcoin/bitcoin@e46be25
(updated with more recent changes)

In prevector.h, the code which like item_ptr(size()) apears in the loop.
Both item_ptr() and size() judge whether values are held directly or
indirectly, but in most cases it is sufficient to make that judgement
once outside the loop.

This PR adds 2 private function fill() which has the loop to initialize
by specified value (or iterator of the other prevector's element),
but don't call item_ptr() in their loop.
Other functions(assign(), constructor, operator=(), insert())
that has similar loop, call fill() instead of original loop.

Also, resize() was changed like fill(), but it calls the default
constructor for that element each time.
@Empact @random-zebra
Drop defunct IS_TRIVIALLY_CONSTRUCTIBLE handling from prevector.h
4cb188e 
It's now only referenced from the bench, so leave it there. This allows
us to
drop the associated includes as well.
@ldm5180 @random-zebra
warnings: Compiler warning on memset usage for non-trivial type
d856989 
Problem:
- IS_TRIVIALLY_CONSTRUCTIBLE macro does not work correctly resulting
  in `memset()` usage to set a non-trivial type to 0 when
  `nontrivial_t` is passed in from the tests.
- Warning reported by GCC when compiling with `--enable-werror`.

Solution:
- Use the standard algorithm `std::fill_n()` and let the compiler
  determine the optimal way of looping or using `memset()`.
@random-zebra random-zebra self-assigned this Dec 19, 2020
Fuzzbawls
Fuzzbawls approved these changes Dec 20, 2020
View reviewed changes
Copy link
Collaborator

@Fuzzbawls Fuzzbawls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK d856989

furszy
furszy approved these changes Dec 20, 2020
View reviewed changes
Copy link

@furszy furszy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite cool. utACK d856989

As far as i'm seeing, c412d5f contains changes coming from 5aad635 as well. Guess that we are not back porting it entirely here because IS_TRIVIALLY_CONSTRUCTIBLE is dropped later on.

@random-zebra
Copy link
Author

Guess that we are not back porting it entirely here because IS_TRIVIALLY_CONSTRUCTIBLE is dropped later on

Yep. exactly. Had to squash a couple changes because this backport was partially out of order (e.g. you can see the last cherry-pick resulted in the addition of only header includes).

@random-zebra random-zebra added this to the 5.0.0 milestone Dec 20, 2020
@furszy furszy merged commit 95ed10a into PIVX-Project:master Dec 20, 2020
Fuzzbawls pushed a commit to Fuzzbawls/PIVX that referenced this pull request Dec 21, 2020
@random-zebra @Fuzzbawls
Add new prevector benchmarks.
e6ffe2a 
>>> backports bitcoin/bitcoin@f0e7aa7

This prepares for a series of two additional commits which optimize
prevector performance.

Github-Pull: PIVX-Project#2083
Rebased-From: e6741d0
Fuzzbawls pushed a commit to Fuzzbawls/PIVX that referenced this pull request Dec 21, 2020
@random-zebra @Fuzzbawls
Reduce redundant code of prevector and speed it up
9675199 
>>> based on bitcoin/bitcoin@e46be25
(updated with more recent changes)

In prevector.h, the code which like item_ptr(size()) apears in the loop.
Both item_ptr() and size() judge whether values are held directly or
indirectly, but in most cases it is sufficient to make that judgement
once outside the loop.

This PR adds 2 private function fill() which has the loop to initialize
by specified value (or iterator of the other prevector's element),
but don't call item_ptr() in their loop.
Other functions(assign(), constructor, operator=(), insert())
that has similar loop, call fill() instead of original loop.

Also, resize() was changed like fill(), but it calls the default
constructor for that element each time.

Github-Pull: PIVX-Project#2083
Rebased-From: c412d5f
Fuzzbawls pushed a commit to Fuzzbawls/PIVX that referenced this pull request Dec 21, 2020
@Empact @Fuzzbawls
Drop defunct IS_TRIVIALLY_CONSTRUCTIBLE handling from prevector.h
359f0c6 
It's now only referenced from the bench, so leave it there. This allows
us to
drop the associated includes as well.

Github-Pull: PIVX-Project#2083
Rebased-From: 4cb188e
Fuzzbawls pushed a commit to Fuzzbawls/PIVX that referenced this pull request Dec 21, 2020
@ldm5180 @Fuzzbawls
warnings: Compiler warning on memset usage for non-trivial type
1555406 
Problem:
- IS_TRIVIALLY_CONSTRUCTIBLE macro does not work correctly resulting
  in `memset()` usage to set a non-trivial type to 0 when
  `nontrivial_t` is passed in from the tests.
- Warning reported by GCC when compiling with `--enable-werror`.

Solution:
- Use the standard algorithm `std::fill_n()` and let the compiler
  determine the optimal way of looping or using `memset()`.

Github-Pull: PIVX-Project#2083
Rebased-From: d856989
@Fuzzbawls Fuzzbawls mentioned this pull request Dec 21, 2020
Merged
furszy added a commit that referenced this pull request Dec 21, 2020
@furszy
Merge #2078: [Backport] 5.0 backports
6c8abbd 
9390f8e [Refactor] Pass caught logic_error by reference in CreateSig (random-zebra)
7f37d7e Only use one set of parentheses for optional arguments (Fuzzbawls)
19f5eae [RPC] Use "shield" nomenclature instead of "shielded" (Fuzzbawls)
0cb01c9 GUI: settings information, fix missing initial masternodes count value. (furszy)
20bd97a GUI: ccontrol: remove column checkbox extra whitespace in tree mode (random-zebra)
1555406 warnings: Compiler warning on memset usage for non-trivial type (Lenny Maiorani)
359f0c6 Drop defunct IS_TRIVIALLY_CONSTRUCTIBLE handling from prevector.h (Ben Woosley)
9675199 Reduce redundant code of prevector and speed it up (random-zebra)
e6ffe2a Add new prevector benchmarks. (random-zebra)
04756ee Fixup various formatting inconsistencies with RPC help output (Fuzzbawls)
41fc192 Don't bother registering the `multisend` command (Fuzzbawls)
bbbc7c9 Fix typo in `getsaplingnotescount` help text (Fuzzbawls)
87c4d21 [Build] Unify header file extensions (Fuzzbawls)
411edee [BUG] Miner not paying valid finalized budget if <20 active masternodes (random-zebra)
dabe995 Tests: minor fixes for mining_pos_reorg_py (random-zebra)
03ec6bb Wallet: return unconfirmed balance for all spendable coins by default (random-zebra)
e9a9d54 Consensus: Allow setting old spork key before its expiration (random-zebra)
c5abbda Consensus: update spork keys for v5 (random-zebra)
60f1535 GUI: Dashboard, fix type filtering disappearance. (furszy)
1f28551 Params: clear testnet checkpoints (random-zebra)
5b809b2 Params: Fix PoS block value for testnet to 5 (random-zebra)
4c847eb Params: Update testnet params - remove unused ones (random-zebra)
f2ea00d Params: change testnet datadir to testnet5 (random-zebra)
2444e54 BUG: Miner don't check for PoS activation at negative height (random-zebra)
f3b01ba Consensus: Budget: Fix instant killing of proposals (random-zebra)
0d0fe24 [BUG][GUI] Receive widget: check typeRole before refreshing view (random-zebra)
40e3203 Refactor: initialize dummy Spend/Output descriptions only once (random-zebra)
6ea78ad Sapling: Skip proofs and signatures during fee calculation loop (random-zebra)
0aa9439 Sapling: add option to skip ProveAndSing in TransactionBuilder::Build (random-zebra)
ff2dec2 Sapling: Decouple ProveAndSign from Build (random-zebra)
5b7e86d [Wallet] Do not perform ECDSA signing in the fee calculation inner loop (random-zebra)
7c87241 [Consensus] Bump TxVersion::SAPLING to 3 (random-zebra)
05fe877 Cleanup: remove unneeded check (time-based) for version 4 header (random-zebra)
a29f1e3 Miner: set proper block version based on upgrade enforcements (random-zebra)
8e0a9d9 Consensus: reject blocks with outdated version (<8) after v5 enforcement (random-zebra)
25e2816 BUG: Fix TestBlockValidity for PoW blocks (random-zebra)
1466f0b [GUI] Connect alert signal (furszy)
6850224 Move GetWarnings() into its own file. (furszy)
edf51b3 util GetWarnings, adding "gui" parameter type. (furszy)
60cc6f2 Eliminate data races for strMiscWarning and fLargeWork*Found. (furszy)
3fc1597 Make QT runawayException call GetWarnings instead of directly access strMiscWarning. (furszy)
0ece82d Separate CTranslationInterface from CClientUIInterface (furszy)
a46f00d [Cleanup] Remove unused global fields from util.h/cpp (furszy)
ee5306f Tests: Update sapling_malleable-sigs test, now failing (random-zebra)
8780161 Tests: update sighash json test data for saplingVersion txes (random-zebra)
0a82c10 Script: Properly sign/verify shielded tx with sigversion sapling (random-zebra)
6b5d96d Tests: Add trolling exploit demo (random-zebra)
460a875 Tests: fix order of sapData serialization (random-zebra)
43f7ca7 BUG: Guard access to tx.sapData in connectBlock (random-zebra)
c12ff2b BUG: SSPKM, guard GetDebit/GetCredit for non-shield v2 txes (random-zebra)
5797e2b [GUI][Trivial] Adjust CoinControl column widths for readability (Ambassador)
8c500df Tests: Add sapling_fillblock functional test (random-zebra)
ce7f265 Consensus: check total size of all shielded txes inside a block (random-zebra)
7b8425a Miner: do not include more than 750 kB of shielded txes inside a block (random-zebra)
c314d7b Trivial: Add Zcash copyright (random-zebra)
99fa644 Refactor: Decouple ShieldedSendManyTo from sendtoaddress/sendmany (random-zebra)
2b7b466 [Test] Add case for (shielded) sendtoaddress (random-zebra)
25b2513 [RPC] Redirect sendtoaddress to shieldedsendmany when shielded recipient (random-zebra)
8c2c328 Refactor: always check mnp blockhash in CMasternodePing::CheckAndUpdate (random-zebra)
aca3764 MN: pass sigTime to CMasternodePing constructor (random-zebra)
1ca9476 MN: pass lastPing to CMasternodeBroadcast constructor (random-zebra)
974b965 [Validation] Remove unused DisconnectBlock function definition. (furszy)
5a33bb1 [Validation] Remove unused DisconnectBlocks function. (furszy)
76ae626 [Validation] Remove unused ReprocessBlocks function. (furszy)
626170e [Masternde] Refactor activeMasternode.GetKeys callers to not expect a bool. (furszy)
545ca77 [Masternode] Clean now unused global strMasterNodePrivKey. (furszy)
8a88cd9 [Masternode] Stop parsing the MN private key string on every finalized budget vote. (furszy)
83080ea [Masternode] Stop parsing the masternode private key string on every ping. (furszy)
77a3959 [Masternode] Cache decoded Masternode private key (furszy)
d4b8a32 [net] Avoid initialization to a value that is never read (furszy)
bbc3a7e Revert "Use async name resolving to improve net thread responsiveness" (furszy)
e9469b1 Use Socks5ErrorString() to decode error responses from socks proxy. (furszy)
2e23740 Make Socks5() InterruptibleRecv() timeout/failures informative. (furszy)
7eefaba SOCKS5 connecting and connected messages with -debug=net. (furszy)
6baf377 Make failures to connect via Socks5() more informative and less unnecessarily scary. (furszy)
aff68db [GUI] Transaction detail, scroll memo fix, starting from the top. (furszy)
38895a9 Make tests pass after 2020 (Bernhard M. Wiedemann)
ac8d818 [Wallet] BugFix: automatic upgrade for unlocked wallet isn't upgrading the wallet feature version. (furszy)
b1133c4 [Tests] Check spent nullifier in sapling_wallet_listreceived (random-zebra)
9576897 [RPC] Add nullifiers to listshieldedunspent output (random-zebra)
f99a176 [BUG][RPC] Fix check in wallet upgrade RPC (random-zebra)
fb58f5f [Miner] Update block header's current version to v8 (Fuzzbawls)
99c2d5e [TierTwo] Remove unneeded extra logging. (furszy)
3066165 [Sapling] Transparent inputs selection, fixing inverted order. Sort decrementally instead of incrementally. (furszy)
0eaf735 [Test] Adding sapling transaction validation unit test coverage. (furszy)
f695d12 CMakeList, pivxd and test_pivx fix for missing MacOS framework. (furszy)

Pull request description:

  backports the following PRs from master to the 5.0 branch:

  #2041
  #2042
  #2046
  #2045
  #2052
  #2047
  #2049
  #2048
  #2059
  #2055
  #2054
  #2056
  #2058
  #2060
  #2050
  #2061
  #2062
  #2033
  #2064
  #2063
  #2067
  #2051
  #2065
  #2066
  #2068
  #2069
  #2073
  #2071
  #2080
  #2081
  #2084
  #2087
  #2088
  #2083
  #2070
  #2075
  #2086
  #2091

ACKs for top commit:
  random-zebra:
    utACK 9390f8e
  furszy:
    utACK 9390f8e

Tree-SHA512: aca2dc8fd3b8d234a27ba7b128168cc3e48b0b775b5699f8fe53e84ac548eb99121b3f0304939896c6e876ffd252f2023958ccb38a344425093948f206238892
@random-zebra random-zebra mentioned this pull request Jan 19, 2021
Merged
furszy added a commit that referenced this pull request Jan 25, 2021
@furszy
Merge #2148: [Core] Prevector Optimizations 2
07b88da 
b886a4c Fix header guards using reserved identifiers (Dan Raviv)
b389b3f speed up Unserialize_impl for prevector (Akio Nakamura)
13d2102 Minimal fix to slow prevector tests as stopgap measure (Jeremy Rubin)

Pull request description:

  Backports bitcoin#12324.
  The `DeserializeAndCheckBlock` benchmark (introduced in #2146) shows a speedup of about 4% (not as much as the upstream PR, due to the optimizations already included in #2083).

  Cherry-picks also
  - bitcoin#8671 (with minimal changes to the random context, due to bitcoin#8914 and bitcoin#9792 being already ported out of order).
  - bitcoin#11151

ACKs for top commit:
  Fuzzbawls:
    utACK b886a4c
  furszy:
    utACK b886a4c and merging..

Tree-SHA512: f5de40e5acfb0b875d413d8995d71dd90489730fe4853f0be03d76a1c44ec95eaeb28c0c40d8e91906f23529ad26501bda4f9779ce466cd8603ed97f1662ca98
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@furszy furszy furszy approved these changes

@Fuzzbawls Fuzzbawls Fuzzbawls approved these changes

Assignees

@random-zebra random-zebra

Labels
Refactoring Upstream
Projects
None yet
Milestone
5.0.0
Development

Successfully merging this pull request may close these issues.
5 participants
@random-zebra @furszy @Fuzzbawls @Empact @ldm5180