Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Allow updating configuration of changes tries #3201

Merged
merged 80 commits into from
Jan 16, 2020
Merged
Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
611f481
DigestItem::ChangesTrieSignal
svyatonik Jun 25, 2019
8b65b04
introduce changes_trie::State
svyatonik Jun 26, 2019
0cd05df
introduce config activation block
svyatonik Jun 26, 2019
40ed2c2
ChangesTrieSignal::as_new_configuration
svyatonik Jun 26, 2019
c01ecd6
moved well_known_cache_keys to client
svyatonik Jun 27, 2019
73f1933
extracted DbChangesTrieStorage to separate file
svyatonik Jun 27, 2019
315303a
change meaning of none in blockchain cache
svyatonik Jun 28, 2019
67642d1
changes trie config (FULL) cache draft
svyatonik Jul 1, 2019
a7e240d
eliminating const ChangesTrieConfiguration
svyatonik Jul 1, 2019
5256d03
delay pruning
svyatonik Jul 2, 2019
698e2ff
continue elimination
svyatonik Jul 3, 2019
04a8fa0
do not prune CT config from cache
svyatonik Jul 3, 2019
afda822
removed redundant code
svyatonik Jul 3, 2019
e08183d
fix some TODOs
svyatonik Jul 4, 2019
368563f
Merge branch 'master' into changes_tries_update_configuration
svyatonik Jul 24, 2019
01e3792
introduce ConfigurationRange
svyatonik Jul 29, 2019
c1913fc
use Configuration range in build
svyatonik Jul 29, 2019
fb6ae9c
build skewed digest
svyatonik Jul 29, 2019
e429061
remove debug print
svyatonik Jul 29, 2019
e6acae8
extracted surface iterator
svyatonik Jul 30, 2019
68527fb
key_changes works with skewed digests
svyatonik Jul 30, 2019
5a839e5
fix client build
svyatonik Jul 30, 2019
9729912
add test for NeverPrune
svyatonik Jul 30, 2019
bca439a
fix TODO
svyatonik Jul 30, 2019
383795a
fixed some TODOs
svyatonik Jul 30, 2019
f6993f6
more tests
svyatonik Jul 31, 2019
8b70450
fixing TODOs
svyatonik Aug 1, 2019
1b3b565
fixed compilation
svyatonik Aug 1, 2019
fa7d631
Merge branch 'master' into changes_tries_update_configuration
svyatonik Aug 1, 2019
1de7f77
update runtime version
svyatonik Aug 1, 2019
ce78f66
git rid of large tuple
svyatonik Aug 1, 2019
82e74d5
too long lines
svyatonik Aug 1, 2019
cfb1cf1
config_activation_block -> zero
svyatonik Aug 1, 2019
94979f8
obsolete TODO
svyatonik Aug 1, 2019
dd5b602
removed unjustified expect
svyatonik Aug 1, 2019
7260719
update TODOs with issue number
svyatonik Aug 1, 2019
c563d30
Merge branch 'master' into changes_tries_update_configuration
svyatonik Aug 6, 2019
4552586
new CT pruning algorithm
svyatonik Aug 7, 2019
62a5aa8
Merge branch 'master' into changes_tries_update_configuration
svyatonik Aug 13, 2019
27eecae
BlockIdOrHeader isn't really required
svyatonik Aug 14, 2019
a158250
Merge branch 'master' into changes_tries_update_configuration
svyatonik Nov 8, 2019
ce92672
Merge branch 'master' into changes_tries_update_configuration
svyatonik Nov 11, 2019
9da96a5
removed debug leftovers + some docs
svyatonik Nov 12, 2019
f8a14f5
more docs
svyatonik Nov 12, 2019
68c4db0
more post-merge fixes
svyatonik Nov 12, 2019
d63d65e
more post-merge fixes
svyatonik Nov 12, 2019
9baac30
Merge branch 'master' into changes_tries_update_configuration
svyatonik Nov 12, 2019
6da8b23
revertes some unnecessary changes
svyatonik Nov 12, 2019
d302d1e
reverted unnecessary changes
svyatonik Nov 12, 2019
f4a82de
fix compilation + unnecessary changes
svyatonik Nov 12, 2019
801169c
(restart CI)
svyatonik Nov 12, 2019
eae183b
fix cache update when finalizing multiple blocks
svyatonik Nov 12, 2019
6d2b4db
fixed tests
svyatonik Nov 12, 2019
d7abf44
Merge branch 'master' into changes_tries_update_configuration
svyatonik Nov 25, 2019
6b3d12d
Merge branch 'master' into changes_tries_update_configuration
svyatonik Nov 29, 2019
d89c979
Merge remote-tracking branch 'origin/master' into changes_tries_updat…
svyatonik Dec 11, 2019
ed31869
Merge remote-tracking branch 'origin/master' into changes_tries_updat…
svyatonik Dec 17, 2019
af3c925
collect_extrinsics -> set_collect_extrinsics
svyatonik Jan 5, 2020
935fb7b
restore lost test
svyatonik Jan 5, 2020
9da28c8
do not calculate block number twice
svyatonik Jan 5, 2020
bf7fea2
Update primitives/blockchain/src/error.rs
svyatonik Jan 5, 2020
3d570f1
map_err -> unwrap_or
svyatonik Jan 5, 2020
b898191
document get_at Result
svyatonik Jan 5, 2020
7db5596
delete abandoned file
svyatonik Jan 5, 2020
9481f59
added weight for set_changes_trie_config
svyatonik Jan 5, 2020
05630d8
Merge branch 'changes_tries_update_configuration' of https://github.c…
svyatonik Jan 5, 2020
06755b0
prefer_configs -> fail_if_disabled
svyatonik Jan 5, 2020
d0b2de2
Update client/api/src/backend.rs
svyatonik Jan 5, 2020
2f795e6
Merge branch 'changes_tries_update_configuration' of https://github.c…
svyatonik Jan 5, 2020
c4c8871
Update client/db/src/changes_tries_storage.rs
svyatonik Jan 6, 2020
51fa091
CommitOperation+merge -> CommitOperations
svyatonik Jan 6, 2020
98d1958
Merge branch 'changes_tries_update_configuration' of https://github.c…
svyatonik Jan 7, 2020
46d4721
fixed test compilation
svyatonik Jan 7, 2020
39c5f64
merged two different CTRange structs
svyatonik Jan 9, 2020
ee16f37
Merge remote-tracking branch 'origin/master' into changes_tries_updat…
svyatonik Jan 9, 2020
25452a5
lost file
svyatonik Jan 9, 2020
54f3d47
Merge remote-tracking branch 'origin/master' into changes_tries_updat…
svyatonik Jan 13, 2020
233afad
uggrade db from v0 to v1 (init CT cache + add column)
svyatonik Jan 14, 2020
0997bd6
Merge branch 'master' into changes_tries_update_configuration
gavofyork Jan 16, 2020
7658040
fix after merge
svyatonik Jan 16, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 41 additions & 10 deletions client/api/src/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ use sp_core::offchain::OffchainStorage;
use sp_runtime::{generic::BlockId, Justification, Storage};
use sp_runtime::traits::{Block as BlockT, NumberFor};
use sp_state_machine::backend::Backend as StateBackend;
use sp_state_machine::{ChangesTrieStorage as StateChangesTrieStorage, ChangesTrieTransaction};
use sp_state_machine::{ChangesTrieState, ChangesTrieStorage as StateChangesTrieStorage, ChangesTrieTransaction};
use crate::{
blockchain::{
Backend as BlockchainBackend, well_known_cache_keys
Expand Down Expand Up @@ -237,8 +237,6 @@ pub trait Backend<Block, H>: AuxStore + Send + Sync where
type Blockchain: BlockchainBackend<Block>;
/// Associated state backend type.
type State: StateBackend<H>;
/// Changes trie storage.
type ChangesTrieStorage: PrunableStateChangesTrieStorage<Block, H>;
/// Offchain workers local storage.
type OffchainStorage: OffchainStorage;

Expand All @@ -265,7 +263,7 @@ pub trait Backend<Block, H>: AuxStore + Send + Sync where
fn used_state_cache_size(&self) -> Option<usize>;

/// Returns reference to changes trie storage.
fn changes_trie_storage(&self) -> Option<&Self::ChangesTrieStorage>;
fn changes_trie_storage(&self) -> Option<&dyn PrunableStateChangesTrieStorage<Block, H>>;

/// Returns a handle to offchain storage.
fn offchain_storage(&self) -> Option<Self::OffchainStorage>;
Expand Down Expand Up @@ -313,16 +311,29 @@ pub trait Backend<Block, H>: AuxStore + Send + Sync where
fn get_import_lock(&self) -> &RwLock<()>;
}

/// Changes trie configuration range.
#[derive(Debug)]
pub struct ChangesTrieConfigurationRange<Block: BlockT> {
/// Zero block of this configuration. First trie that uses this configuration is build at the next block.
pub zero: (NumberFor<Block>, Block::Hash),
/// End block where last trie that uses this configuration has been build. None if configuration is active.
pub end: Option<(NumberFor<Block>, Block::Hash)>,
/// Configuration itself. None if changes tries are disabled within this range.
pub config: Option<ChangesTrieConfiguration>,
}

/// Changes trie storage that supports pruning.
pub trait PrunableStateChangesTrieStorage<Block: BlockT, H: Hasher>:
StateChangesTrieStorage<H, NumberFor<Block>>
{
/// Get number block of oldest, non-pruned changes trie.
fn oldest_changes_trie_block(
&self,
config: &ChangesTrieConfiguration,
best_finalized: NumberFor<Block>,
) -> NumberFor<Block>;
/// Get reference to StateChangesTrieStorage.
fn storage(&self) -> &dyn StateChangesTrieStorage<H, NumberFor<Block>>;
/// Get coniguration at given block.
svyatonik marked this conversation as resolved.
Show resolved Hide resolved
fn configuration_at(&self, at: &BlockId<Block>) -> sp_blockchain::Result<ChangesTrieConfigurationRange<Block>>;
/// Get end block (inclusive) of oldest pruned max-level (or skewed) digest trie blocks range.
/// It is guaranteed that we have no any changes tries before (and including) this block.
/// It is guaranteed that all existing changes tries after this block are not yet pruned (if created).
fn oldest_pruned_digest_range_end(&self) -> NumberFor<Block>;
}

/// Mark for all Backend implementations, that are making use of state data, stored locally.
Expand All @@ -347,3 +358,23 @@ where
/// locally, or prepares request to fetch that data from remote node.
fn remote_blockchain(&self) -> Arc<dyn RemoteBlockchain<Block>>;
}

/// Return changes tries state at given block.
pub fn changes_tries_state_at_block<'a, Block: BlockT, H: Hasher>(
block: &BlockId<Block>,
maybe_storage: Option<&'a dyn PrunableStateChangesTrieStorage<Block, H>>,
) -> sp_blockchain::Result<Option<ChangesTrieState<'a, H, NumberFor<Block>>>>
where
H: Hasher<Out=Block::Hash>,
{
let storage = match maybe_storage {
Some(storage) => storage,
None => return Ok(None),
};

let config_range = storage.configuration_at(block)?;
match config_range.config {
Some(config) => Ok(Some(ChangesTrieState::new(config, config_range.zero.0, storage.storage()))),
None => Ok(None),
}
}
1 change: 1 addition & 0 deletions client/api/src/call_executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ where
NC: FnOnce() -> result::Result<R, String> + UnwindSafe,
>(&self,
state: &S,
state_block: &BlockId<B>,
overlay: &mut OverlayedChanges,
method: &str,
call_data: &[u8],
Expand Down
4 changes: 2 additions & 2 deletions client/api/src/light.rs
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,8 @@ pub struct RemoteReadChildRequest<Header: HeaderT> {
/// Remote key changes read request.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RemoteChangesRequest<Header: HeaderT> {
/// Changes trie configuration.
pub changes_trie_config: ChangesTrieConfiguration,
/// All changes trie configurations that are valid within [first_block; last_block].
pub changes_trie_configs: Vec<(Header::Number, Option<Header::Number>, ChangesTrieConfiguration)>,
svyatonik marked this conversation as resolved.
Show resolved Hide resolved
/// Query changes from range of blocks, starting (and including) with this hash...
pub first_block: (Header::Number, Header::Hash),
/// ...ending (and including) with this hash. Should come after first_block and
Expand Down
14 changes: 8 additions & 6 deletions client/consensus/aura/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -653,19 +653,20 @@ fn initialize_authorities_cache<A, B, C>(client: &C) -> Result<(), ConsensusErro
};

// check if we already have initialized the cache
let map_err = |error| sp_consensus::Error::from(sp_consensus::Error::ClientImport(
format!(
"Error initializing authorities cache: {}",
error,
)));

let genesis_id = BlockId::Number(Zero::zero());
let genesis_authorities: Option<Vec<A>> = cache
.get_at(&well_known_cache_keys::AUTHORITIES, &genesis_id)
.get_at(&well_known_cache_keys::AUTHORITIES, &genesis_id).map_err(map_err)?
svyatonik marked this conversation as resolved.
Show resolved Hide resolved
.and_then(|(_, _, v)| Decode::decode(&mut &v[..]).ok());
if genesis_authorities.is_some() {
return Ok(());
}

let map_err = |error| sp_consensus::Error::from(sp_consensus::Error::ClientImport(
format!(
"Error initializing authorities cache: {}",
error,
)));
let genesis_authorities = authorities(client, &genesis_id)?;
cache.initialize(&well_known_cache_keys::AUTHORITIES, genesis_authorities.encode())
.map_err(map_err)?;
Expand All @@ -684,6 +685,7 @@ fn authorities<A, B, C>(client: &C, at: &BlockId<B>) -> Result<Vec<A>, Consensus
.cache()
.and_then(|cache| cache
.get_at(&well_known_cache_keys::AUTHORITIES, at)
.unwrap_or(None)
.and_then(|(_, _, v)| Decode::decode(&mut &v[..]).ok())
)
.or_else(|| AuraApi::authorities(&*client.runtime_api(), at).ok())
Expand Down
Loading