Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limiting orphan block storage #1675

Merged
merged 1 commit into from
Apr 7, 2020
Merged

Limiting orphan block storage #1675

merged 1 commit into from
Apr 7, 2020

Conversation

neonknight64
Copy link
Contributor

Description

  • Orphan block cleanup is performed when a single orphan block is added and on chain reorgs. When a single orphan block is added, then the cheaper cleanup_orphans_single function is called to remove a single orphan block from the orphan pool when the storage limits are reached. The more expensive cleanup_orphans_comprehensive function is called when a chain reorg is performed and many orphan blocks have been added to the block orphan pool that could have resulted in the storage limits being exceeded.
  • Added get_orphan_count to the blockchain backends and provided implementations for lmdb_db and memory_db that can be used to query the number of blocks in the orphan block pool.
  • Added BlockchainDatabaseConfig to the blockchain db to enable the orphan storage capacity to be set.

Motivation and Context

These changes enforces a storage limit on the orphan block pool, limiting the number of orphan blocks that are kept.

How Has This Been Tested?

  • Provided two tests for testing the two different orphan block cleanup mechanisms. The orphan_cleanup_on_add test checks the removal of a single orphan block when the storage capacity is reached on an add block operation. The orphan_cleanup_on_reorg test checks the removal of a set of orphan blocks when the storage capacity is reached on a chain reorg operation, when many blocks were added to the orphan pool.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Feature refactor (No new feature or functional changes, but performance or technical debt improvements)
  • New Tests
  • Documentation

Checklist:

  • I'm merging against the development branch.
  • I ran cargo-fmt --all before pushing.
  • I have squashed my commits into a single commit.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.

CjS77
CjS77 previously approved these changes Apr 6, 2020
View reviewed changes
Copy link
Collaborator

@CjS77 CjS77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

base_layer/core/src/chain_storage/consts.rs Outdated Show resolved Hide resolved
- These changes enforces a storage limit on the orphan block pool, li…
ac9e660 
…miting the number of orphan blocks that are kept.

- Orphan block cleanup is performed when a single orphan block is added and on chain reorgs. When a single orphan block is added, then the cheaper cleanup_orphans_single function is called to remove a single orphan block from the orphan pool when the storage limits are reached. The more expensive cleanup_orphans_comprehensive function is called when a chain reorg is performed and many orphan blocks have been added to the block orphan pool that could have resulted in the storage limits being exceeded.
- Added get_orphan_count to the blockchain backends and provided implementations for lmdb_db and memory_db that can be used to query the number of blocks in the orphan block pool.
- Added BlockchainDatabaseConfig to the blockchain db to enable the orphan storage capacity to be set.
- Provided two tests for testing the two different orphan block cleanup mechanisms. The orphan_cleanup_on_add test checks the removal of a single orphan block when the storage capacity is reached on an add block operation. The orphan_cleanup_on_reorg test checks the removal of a set of orphan blocks when the storage capacity is reached on a chain reorg operation, when many blocks were added to the orphan pool.
@neonknight64 neonknight64 force-pushed the yuko-orphan_block_removal branch from 9ece6dd to ac9e660 Compare April 7, 2020 05:44
SWvheerden
SWvheerden approved these changes Apr 7, 2020
View reviewed changes
Copy link
Collaborator

@SWvheerden SWvheerden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one question

@CjS77 CjS77 merged commit 9936bc9 into development Apr 7, 2020
@CjS77 CjS77 deleted the yuko-orphan_block_removal branch April 7, 2020 10:00
SWvheerden added a commit that referenced this pull request Mar 20, 2024
@hansieodendaal @SWvheerden
feat: cache new block template (#6222)
e0ad342 
Description
---
In the merge mining proxy for the block template protocol:
- Cached a new block template and used it in case it is asked
repetitively for the same best block.
- Added an exit clause to the potential endless loop.

Motivation and Context
---
Under certain conditions, a merge mining proxy request for a new block
template from the base node would be repeated many times for the same
best block height, without the final request for a block template that
includes the coinbase transaction. This wastes many resources,
especially within the mempool where the template must be constructed.
```
2024-03-15 16:49:43.873749100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1675
2024-03-15 16:56:35.702267000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1676
2024-03-15 16:56:57.709323100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1677
2024-03-15 16:59:04.747989700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.840957100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.849282000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.852855000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.953873500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.954461600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:04.958619200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.054517300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.056071600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.057842800 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.058706200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.184297100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.184568300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.186171600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.308025900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.308188600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.411993800 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.413148500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.436028800 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.457553200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.569510200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.573500600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.658578700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.659258900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.693116700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.695350400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.713246600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.724627700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.734541000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.904546600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.909569300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.913773800 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.914761900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.918061100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:05.928847000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.125242100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.125427400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.130678700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.367898700 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.385011900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.386722300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.388034000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.406540100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.645604600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.654252400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.659640400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.661591500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.829952300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.838337400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.843845500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:06.847976400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.036353600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.038593100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.042252500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.070333200 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.204174900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.207776400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.208302900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.211634900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.218055300 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.367296100 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.367378600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.369022900 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.372211400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.504687000 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.505760400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:07.505823500 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:09.578821600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 16:59:09.829455600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1678
2024-03-15 17:00:11.282291600 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1679
2024-03-15 17:01:13.476999400 [minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new block template at height: #1680
```

How Has This Been Tested?
---
System-level testing

What process can a PR reviewer use to test or verify this change?
---
- Review code changes
- System-level testing:
- These log entries should not be repeated for the same best block
height:
  ```
2024-03-18 15:12:50.362269600
[minotari_mm_proxy::proxy::block_template_protocol] DEBUG Requested new
block template at height: #1832 (try 4)
  ```
  - Look for these log entries instead:
  ```
2024-03-18 15:13:46.016447400
[minotari_mm_proxy::proxy::block_template_protocol] DEBUG Used existing
new block template at height: #1836 (try 1)
  ```

<!-- Checklist -->
<!-- 1. Is the title of your PR in the form that would make nice release
notes? The title, excluding the conventional commit
tag, will be included exactly as is in the CHANGELOG, so please think
about it carefully. -->


Breaking Changes
---

- [x] None
- [ ] Requires data directory on base node to be deleted
- [ ] Requires hard fork
- [ ] Other - Please specify

<!-- Does this include a breaking change? If so, include this line as a
footer -->
<!-- BREAKING CHANGE: Description what the user should do, e.g. delete a
database, resync the chain -->

---------

Co-authored-by: SW van Heerden <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SWvheerden SWvheerden SWvheerden approved these changes

@CjS77 CjS77 CjS77 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@neonknight64 @SWvheerden @CjS77