Skip to content

Commit

Permalink
Preserve reducer order in schema conversion (#1987)
Browse files Browse the repository at this point in the history 
Signed-off-by: Ingvar Stepanyan <[email protected]>
Co-authored-by: james gilles <[email protected]>
  • Loading branch information
@RReverser @kazimuth
RReverser and kazimuth authored Nov 16, 2024
1 parent a7a1d36 commit ac1d222
Show file tree
Hide file tree
Showing 5 changed files with 10 additions and 7 deletions.
1 change: 1 addition & 0 deletions Cargo.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 2 additions & 4 deletions crates/core/src/host/module_host.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -238,14 +238,12 @@ impl ModuleInfo {
}

/// A bidirectional map between `Identifiers` (reducer names) and `ReducerId`s.
/// Invariant: the reducer names are in alphabetical order.
/// Invariant: the reducer names are in the same order as they were declared in the `ModuleDef`.
pub struct ReducersMap(IndexSet<Box<str>>);

impl<'a> FromIterator<&'a str> for ReducersMap {
fn from_iter<T: IntoIterator<Item = &'a str>>(iter: T) -> Self {
let mut sorted = Vec::from_iter(iter);
sorted.sort();
ReducersMap(sorted.into_iter().map_into().collect())
Self(iter.into_iter().map_into().collect())
}
}

Expand Down
1 change: 1 addition & 0 deletions crates/schema/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ spacetimedb-data-structures.workspace = true
spacetimedb-sql-parser.workspace = true

anyhow.workspace = true
indexmap.workspace = true
itertools.workspace = true
lazy_static.workspace = true
thiserror.workspace = true
Expand Down
7 changes: 5 additions & 2 deletions crates/schema/src/def.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ use crate::schema::{Schema, TableSchema};
use crate::type_for_generate::{AlgebraicTypeUse, ProductTypeDef, TypespaceForGenerate};
use deserialize::ReducerArgsDeserializeSeed;
use hashbrown::Equivalent;
use indexmap::IndexMap;
use itertools::Itertools;
use spacetimedb_data_structures::error_stream::{CollectAllErrors, CombineErrors, ErrorStream};
use spacetimedb_data_structures::map::HashMap;
Expand Down Expand Up @@ -84,7 +85,9 @@ pub struct ModuleDef {
tables: IdentifierMap<TableDef>,

/// The reducers of the module definition.
reducers: IdentifierMap<ReducerDef>,
/// Note: this is using IndexMap because reducer order is important
/// and must be preserved for future calls to `__call_reducer__`.
reducers: IndexMap<Identifier, ReducerDef>,

/// The type definitions of the module definition.
types: HashMap<ScopedTypeName, TypeDef>,
Expand Down Expand Up @@ -355,7 +358,7 @@ impl From<ModuleDef> for RawModuleDefV9 {

RawModuleDefV9 {
tables: to_raw(tables, |table: &RawTableDefV9| &table.name),
reducers: to_raw(reducers, |reducer: &RawReducerDefV9| &reducer.name),
reducers: reducers.into_iter().map(|(_, def)| def.into()).collect(),
types: to_raw(types, |type_: &RawTypeDefV9| &type_.name),
misc_exports: vec![],
typespace,
Expand Down
2 changes: 1 addition & 1 deletion crates/schema/src/def/validate/v9.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -729,7 +729,7 @@ fn identifier(name: Box<str>) -> Result<Identifier> {

fn check_scheduled_reducers_exist(
tables: &IdentifierMap<TableDef>,
reducers: &IdentifierMap<ReducerDef>,
reducers: &IndexMap<Identifier, ReducerDef>,
) -> Result<()> {
tables
.values()
Expand Down

2 comments on commit ac1d222

@github-actions
Copy link

@github-actions github-actions bot commented on ac1d222 Nov 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmarking failed. Please check the workflow run for details.

@github-actions
Copy link

@github-actions github-actions bot commented on ac1d222 Nov 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6426 6397 0.45% 6526 6481 0.69%
sqlite 5579 5579 0.00% 5941 6009 -1.13%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 76586 76592 -0.01% 77042 77076 -0.04%
stdb_raw u32_u64_str no_index 64 128 2 string 118828 119923 -0.91% 119522 120687 -0.97%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25113 25082 0.12% 25727 25678 0.19%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24078 24049 0.12% 24602 24445 0.64%
sqlite u32_u64_str no_index 64 128 2 string 144695 144695 0.00% 146119 146125 -0.00%
sqlite u32_u64_str no_index 64 128 1 u64 124044 124044 0.00% 125294 125186 0.09%
sqlite u32_u64_str btree_each_column 64 128 1 u64 131361 131361 0.00% 132739 132819 -0.06%
sqlite u32_u64_str btree_each_column 64 128 2 string 134494 134494 0.00% 136078 136096 -0.01%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 878628 870940 0.88% 934724 890532 4.96%
stdb_raw u32_u64_str btree_each_column 64 128 1030117 1021063 0.89% 1089957 1080211 0.90%
sqlite u32_u64_str unique_0 64 128 398330 398336 -0.00% 413148 416760 -0.87%
sqlite u32_u64_str btree_each_column 64 128 983637 983637 0.00% 1012351 1026403 -1.37%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 153810 153724 0.06% 153964 153838 0.08%
stdb_raw u32_u64_str unique_0 64 16835 16749 0.51% 16973 16847 0.75%
sqlite u32_u64_str unique_0 1024 1068275 1068275 0.00% 1071695 1071621 0.01%
sqlite u32_u64_str unique_0 64 76261 76279 -0.02% 77427 77387 0.05%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50248 50214 0.07%
64 bsatn 25509 25509 0.00% 27651 27821 -0.61%
16 bsatn 8200 8200 0.00% 9458 9628 -1.77%
16 json 12188 12188 0.00% 14160 14126 0.24%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 20130326 20388882 -1.27% 20782904 21068940 -1.36%
stdb_raw u32_u64_str unique_0 64 128 1287506 1277818 0.76% 1364462 1350966 1.00%
sqlite u32_u64_str unique_0 1024 1024 1802188 1802188 0.00% 1811300 1811458 -0.01%
sqlite u32_u64_str unique_0 64 128 128534 128534 0.00% 131458 131522 -0.05%
On-disk benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6431 6402 0.45% 6527 6498 0.45%
sqlite 5621 5621 0.00% 6043 6111 -1.11%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 76591 76597 -0.01% 77027 77081 -0.07%
stdb_raw u32_u64_str no_index 64 128 2 string 118833 118839 -0.01% 119403 119567 -0.14%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25115 25087 0.11% 25753 25639 0.44%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24083 24054 0.12% 24587 24446 0.58%
sqlite u32_u64_str no_index 64 128 1 u64 125965 125965 0.00% 127479 127491 -0.01%
sqlite u32_u64_str no_index 64 128 2 string 146616 146616 0.00% 148340 148414 -0.05%
sqlite u32_u64_str btree_each_column 64 128 2 string 136616 136616 0.00% 138606 138696 -0.06%
sqlite u32_u64_str btree_each_column 64 128 1 u64 133457 133457 0.00% 135233 135369 -0.10%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 827301 821854 0.66% 851957 871536 -2.25%
stdb_raw u32_u64_str btree_each_column 64 128 978368 970546 0.81% 1036498 1028860 0.74%
sqlite u32_u64_str unique_0 64 128 415857 415857 0.00% 429935 433417 -0.80%
sqlite u32_u64_str btree_each_column 64 128 1021908 1021908 0.00% 1049906 1063426 -1.27%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 153815 153729 0.06% 153953 153827 0.08%
stdb_raw u32_u64_str unique_0 64 16840 16754 0.51% 16978 16852 0.75%
sqlite u32_u64_str unique_0 1024 1071343 1071343 0.00% 1075025 1075087 -0.01%
sqlite u32_u64_str unique_0 64 78033 78033 0.00% 79291 79393 -0.13%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50248 50214 0.07%
64 bsatn 25509 25509 0.00% 27651 27821 -0.61%
16 bsatn 8200 8200 0.00% 9458 9628 -1.77%
16 json 12188 12188 0.00% 14160 14126 0.24%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 19046452 18899226 0.78% 19765766 19603906 0.83%
stdb_raw u32_u64_str unique_0 64 128 1240805 1231074 0.79% 1316659 1306392 0.79%
sqlite u32_u64_str unique_0 1024 1024 1809749 1809743 0.00% 1818359 1818455 -0.01%
sqlite u32_u64_str unique_0 64 128 132654 132654 0.00% 135488 135628 -0.10%

Please sign in to comment.