Current SMT replacement strategy not sustainable

[hansieodendaal]

Time to update the SMT at the tip grows linearly with time, see stats below. For these tests the SMT database made up 5.87% of the total database size.

(Tests were conducted as part of #6242)

Name                             | Entries | Depth | Branch Pages | Leaf Pages | Overflow Pages | Est. Size (MiB) | % of total
-------------------------------- | ------- | ----- | ------------ | ---------- | -------------- | --------------- | ----------
metadata                         | 8       | 1     | 0            | 1          | 0              | 0.00            | 0.00%
headers                          | 9106    | 3     | 10           | 1908       | 0              | 7.49            | 0.24%
header_accumulated_data          | 9106    | 3     | 3            | 396        | 0              | 1.56            | 0.05%
mmr_peak_data                    | 9106    | 3     | 8            | 1368       | 0              | 5.38            | 0.17%
block_hashes                     | 9106    | 3     | 3            | 160        | 0              | 0.64            | 0.02%
utxos                            | 1153237 | 5     | 15499        | 574762     | 0              | 2305.71         | 73.44%
inputs                           | 245695  | 4     | 983          | 35730      | 0              | 143.41          | 4.57%
txos_hash_to_index               | 1153237 | 4     | 695          | 48182      | 0              | 190.93          | 6.08%
kernels                          | 190500  | 4     | 821          | 21991      | 0              | 89.11           | 2.84%
kernel_excess_index              | 190500  | 4     | 133          | 7925       | 0              | 31.48           | 1.00%
kernel_excess_sig_index          | 190500  | 4     | 268          | 10284      | 0              | 41.22           | 1.31%
kernel_mmr_size_index            | 9106    | 2     | 1            | 59         | 0              | 0.23            | 0.01%
utxo_commitment_index            | 907542  | 4     | 383          | 24654      | 0              | 97.80           | 3.11%
contract_index                   | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
unique_id_index                  | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
deleted_txo_hash_to_header_index | 245695  | 4     | 144          | 10220      | 0              | 40.48           | 1.29%
orphans                          | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
orphan_accumulated_data          | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
monero_seed_height               | 1       | 1     | 0            | 1          | 0              | 0.00            | 0.00%
orphan_chain_tips                | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
orphan_parent_map_index          | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
bad_blocks                       | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
reorgs                           | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
validator_nodes                  | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
tip_utxo_smt                     | 1       | 1     | 0            | 1          | 47176          | 184.29          | 5.87%
validator_nodes_mapping          | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%
template_registrations           | 0       | 0     | 0            | 0          | 0              | 0.00            | 0.00%

[hansieodendaal]

The biggest contributor has been suggested by @stringhandler (thanks!) to be the serialization. I did not realize it so each of my trace logs serialized the SMT again to print the size, aggravating the issue. At 184MB sized SMT:

• serialize check in 144.74ms (21%)
• serialize in 476.29ms (69%)
• lmdb write in 68.97ms (10%)

2024-04-03 05:56:45.448523900 [tari_core::chain_storage::lmdb_db::helpers] TRACE lmdb_replace - 184 MB, serialize check in 143.36ms, serialize in 466.61ms
2024-04-03 05:56:48.979679000 [tari_core::chain_storage::lmdb_db::helpers] TRACE lmdb_replace - 184 MB, serialize check in 144.74ms, serialize in 476.29ms
2024-04-03 05:56:49.048703000 [tari_core::chain_storage::lmdb_db::lmdb] TRACE lmdb_replace - 184 MB, lmdb write in 68.97ms
2024-04-03 05:56:49.667435300 [tari_core::chain_storage::lmdb_db::lmdb_db] TRACE Inserted 184 MB with key '08000000' into 'tip_utxo_smt' (size 908540) in 1.31s

[hansieodendaal]

Improved in the latest commit in #6242 (much better but not optimal yet)

2024-04-03 08:54:38.424826600 [tari_core::chain_storage::lmdb_db::helpers] TRACE lmdb_replace - 184 MB, serialize check in 11.00µs, serialize in 458.09ms
2024-04-03 08:54:38.491925800 [tari_core::chain_storage::lmdb_db::lmdb] TRACE lmdb_replace - 184 MB, lmdb write in 67.04ms
2024-04-03 08:54:38.501529800 [tari_core::chain_storage::lmdb_db::lmdb_db] TRACE Inserted ~194 MB with key '08000000' into 'tip_utxo_smt' (size 908601) in 534.84ms

[SWvheerden]

fixed with #6265