Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastic scaling: runtime dependency tracking and enactment #3479

Merged
merged 53 commits into from
Mar 21, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
442185e
Initial draft changes
alindima Feb 16, 2024
799fabe
bugfixes
alindima Feb 26, 2024
9f3ba62
filter descendants of disputed candidates
alindima Feb 26, 2024
5b13eb7
some simplifications
alindima Feb 27, 2024
ecc5088
assert that candidates of a para are sorted in chain dependency order…
alindima Feb 28, 2024
0fb7b8c
deduplicate some of the logic for freeing cores
alindima Feb 28, 2024
0ed6fc3
update some comments
alindima Feb 28, 2024
0a08fa0
unify dropped candidates errors
alindima Feb 28, 2024
9e51e20
add more logs
alindima Feb 28, 2024
4e154e6
remove some todos
alindima Feb 28, 2024
ccef35b
review comments
alindima Feb 29, 2024
3f67898
some more nits
alindima Feb 29, 2024
c4fedd7
Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…
alindima Feb 29, 2024
58b4129
add runtime migration to inclusion storage
alindima Feb 29, 2024
23a8d34
add migration tests
alindima Mar 1, 2024
e0d9dff
don't allow candidate cycles
alindima Mar 1, 2024
0198fb3
fix bug
alindima Mar 1, 2024
267fa05
make tests compile and make paras_inherent tests pass
alindima Mar 1, 2024
7733b25
fix inclusion tests
alindima Mar 1, 2024
c32f925
fix some more tests
alindima Mar 1, 2024
40e2933
clippy
alindima Mar 1, 2024
e8f5d2c
Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…
alindima Mar 1, 2024
630261e
some review comments
alindima Mar 11, 2024
7dc5358
simplify update_pending_availability_and_get_freed_cores
alindima Mar 11, 2024
3793f20
fix clippy
alindima Mar 11, 2024
6c0d062
add prdoc
alindima Mar 11, 2024
74acc46
fix availability_cores runtime API
alindima Mar 11, 2024
4dd28a2
Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…
alindima Mar 11, 2024
5afb901
remove some unused errors and reintroduce ValidationDataHashMismatch
alindima Mar 11, 2024
a8c10be
add more unit tests to inclusion module
alindima Mar 12, 2024
d2309a7
more inclusion unit tests
alindima Mar 13, 2024
c59bce0
more tests to paras_inherent
alindima Mar 13, 2024
5e1a351
more paras_inherent tests
alindima Mar 14, 2024
37ff4c5
optimise update_pending_availability_and_get_freed_cores
alindima Mar 15, 2024
a92523b
make force-enact work on all cores
alindima Mar 15, 2024
b77806e
some review comments
alindima Mar 15, 2024
318b8da
call free_cores_and_fill_claimqueue only once
alindima Mar 15, 2024
376d19c
add removal of pendingbitfields storage to the migration and add more…
alindima Mar 15, 2024
8cb5ff6
Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…
alindima Mar 15, 2024
98f4a0e
one more test for paras_inherent
alindima Mar 18, 2024
e537513
".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…
Mar 18, 2024
f440217
".git/.scripts/commands/bench/bench.sh" --subcommand=pallet --runtime…
Mar 18, 2024
2999589
Merge branch 'master' into alindima/elastic-scaling-runtime
alindima Mar 18, 2024
a745a63
fix runtime API panic
alindima Mar 19, 2024
1b70b11
Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…
alindima Mar 19, 2024
c3060bd
fix clippy
alindima Mar 19, 2024
9f52b8d
map_candidates_to_cores: check core index for single core if ElasticS…
alindima Mar 20, 2024
c27de6a
add a couple more test cases
alindima Mar 20, 2024
e05693c
review comment
alindima Mar 20, 2024
bb76778
Merge branch 'master' into alindima/elastic-scaling-runtime
alindima Mar 20, 2024
91f705c
improve test
alindima Mar 21, 2024
0c8dc20
Merge remote-tracking branch 'origin/master' into alindima/elastic-sc…
alindima Mar 21, 2024
b8d2212
Merge branch 'master' into alindima/elastic-scaling-runtime
eskimor Mar 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion polkadot/runtime/parachains/src/inclusion/migration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

pub use v1::MigrateToV1;

mod v0 {
pub mod v0 {
use crate::inclusion::{Config, Pallet};
use bitvec::{order::Lsb0 as BitOrderLsb0, vec::BitVec};
use frame_support::{storage_alias, Twox64Concat};
Expand Down
15 changes: 14 additions & 1 deletion polkadot/runtime/parachains/src/inclusion/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,10 @@ impl<H, N> CandidatePendingAvailability<H, N> {
}

/// Get the relay-chain block number this was backed in.
pub(crate) fn backed_in_number(&self) -> &N {
pub(crate) fn backed_in_number(&self) -> &N
where
N: Clone,
{
&self.backed_in_number
}

Expand Down Expand Up @@ -149,6 +152,16 @@ impl<H, N> CandidatePendingAvailability<H, N> {
self.relay_parent_number.clone()
}

/// Get the candidate backing group.
pub(crate) fn backing_group(&self) -> GroupIndex {
self.backing_group
}

/// Get the candidate's backers.
pub(crate) fn backers(&self) -> &BitVec<u8, BitOrderLsb0> {
&self.backers
}

#[cfg(any(feature = "runtime-benchmarks", test))]
pub(crate) fn new(
core: CoreIndex,
Expand Down
48 changes: 36 additions & 12 deletions polkadot/runtime/parachains/src/runtime_api_impl/v7.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ use crate::{
scheduler::{self, CoreOccupied},
session_info, shared,
};
use frame_support::traits::{GetStorageVersion, StorageVersion};
use frame_system::pallet_prelude::*;
use primitives::{
async_backing::{
Expand Down Expand Up @@ -92,18 +93,41 @@ pub fn availability_cores<T: initializer::Config>() -> Vec<CoreState<T::Hash, Bl
.enumerate()
.map(|(i, core)| match core {
CoreOccupied::Paras(entry) => {
let pending_availability = <inclusion::Pallet<T>>::pending_availability_with_core(
entry.para_id(),
CoreIndex(i as u32),
)
.expect("Occupied core always has pending availability; qed");

let backed_in_number = *pending_availability.backed_in_number();
// Due to https://github.com/paritytech/polkadot-sdk/issues/64, using the new storage types would cause
eskimor marked this conversation as resolved.
Show resolved Hide resolved
// this runtime API to panic. We explicitly handle the storage for version 0 to
// prevent that. When removing the inclusion v0 -> v1 migration, this bit of code
// can also be removed.
let pending_availability = if <inclusion::Pallet<T>>::on_chain_storage_version() ==
StorageVersion::new(0)
{
inclusion::migration::v0::PendingAvailability::<T>::get(entry.para_id())
.expect("Occupied core always has pending availability; qed")
} else {
let candidate = <inclusion::Pallet<T>>::pending_availability_with_core(
entry.para_id(),
CoreIndex(i as u32),
)
.expect("Occupied core always has pending availability; qed");

// Translate to the old candidate format, as we don't need the commitments now.
inclusion::migration::v0::CandidatePendingAvailability {
eskimor marked this conversation as resolved.
Show resolved Hide resolved
core: candidate.core_occupied(),
hash: candidate.candidate_hash(),
descriptor: candidate.candidate_descriptor().clone(),
availability_votes: candidate.availability_votes().clone(),
backers: candidate.backers().clone(),
relay_parent_number: candidate.relay_parent_number(),
backed_in_number: candidate.backed_in_number().clone(),
backing_group: candidate.backing_group(),
}
};

let backed_in_number = pending_availability.backed_in_number;

// Use the same block number for determining the responsible group as what the
// backing subsystem would use when it calls validator_groups api.
let backing_group_allocation_time =
pending_availability.relay_parent_number() + One::one();
pending_availability.relay_parent_number + One::one();
CoreState::Occupied(OccupiedCore {
next_up_on_available: <scheduler::Pallet<T>>::next_up_on_available(CoreIndex(
i as u32,
Expand All @@ -113,13 +137,13 @@ pub fn availability_cores<T: initializer::Config>() -> Vec<CoreState<T::Hash, Bl
next_up_on_time_out: <scheduler::Pallet<T>>::next_up_on_time_out(CoreIndex(
i as u32,
)),
availability: pending_availability.availability_votes().clone(),
availability: pending_availability.availability_votes.clone(),
group_responsible: group_responsible_for(
backing_group_allocation_time,
pending_availability.core_occupied(),
pending_availability.core,
),
candidate_hash: pending_availability.candidate_hash(),
candidate_descriptor: pending_availability.candidate_descriptor().clone(),
candidate_hash: pending_availability.hash,
candidate_descriptor: pending_availability.descriptor,
})
},
CoreOccupied::Free => {
Expand Down
Loading