Prepare Release 0.5.0 / Update to Substrate 3.0 (#248)

* begin preparing release

* changelog

* update to substrate 3.0

* update polkadot-archive and node-template cargo.lock

* fix stack overflow when getting runtime_version

* make rustfmt ci happy

* fix formatting for ci

* fix some clippy lints

* fix substrate ref

* Update CHANGELOG.md

Co-authored-by: David <[email protected]>

* Update CHANGELOG.md

Co-authored-by: David <[email protected]>

* Update CHANGELOG.md

Co-authored-by: David <[email protected]>

* Update CHANGELOG.md

Co-authored-by: David <[email protected]>

* Update CHANGELOG.md

Co-authored-by: David <[email protected]>

* clippy lint

* fix failed runtime decode

* allow redundant slicing

* add note on clippy allows

* Update CHANGELOG.md

Co-authored-by: David <[email protected]>

* add wiki link about state-tracing

* fix acronyms

* fix clippy lint

* add 0.8.28 wasm blobls

* add 0.29 wasm blobs

* revert awaiting future

* fix some nits

* get_import_lock cannot  return lock

* fix clippy lint

Co-authored-by: David <[email protected]>

CHANGELOG.md

@@ -1,5 +1,8 @@
 # **[Unreleased]**
 
+
+# [v0.5.0]
+
 ## Substrate Archive
 - [Changed] Use SQLX for migrations
   - this will create a new table in the database, `sqlx_migrations` or similiar.
@@ -12,29 +15,40 @@
   - [Changed] no longer need to instantiate a client and manually pass it to the Archive
   - [Changed] rename `run_with` to `drive`
   - [Changed] Archive now accepts just a postgres URL instead of a postgres URL split into its parts. This should
-  make configuring the archive more straightforward. Takes from environment variable `DATABASE_URL` if not passed to the 
+  make configuring the archive more straightforward. Takes from environment variable `DATABASE_URL` if not passed to the
   archive directly
 - [Added] Archive now reads the `CHAIN_DATA_DB` environment variable if the path to the backend chain database is not passed directly.
-- [Changed] `drive` changed from sync to async function
 - [Removed] Archive no longer needs an RPC url to function
+- [Changed] Print out the Postgres URL at startup
+- [Added] `max_block_load` to configure the maximum number of blocks loaded at once
+- [Added] Two new options to the configuration relative to state tracing:
+  - `targets` for specifying runtime targets to trace in WASM
+  - `folder` where WASM blobs with tracing enabled are kept.
+  - More on state-tracing here: https://github.com/paritytech/substrate-archive/wiki/6.)-State-Tracing-&-Balance-Reconciliation  
+- [Changed] Archive config is now separated into sections for readability. Migration is manual but looking at the new `archive.conf` in `polkadot-archive` or `node-template-archive` folders should help.
 
 ### Internal Changes
-- [QoL] upgrade to SQLx 0.4.0
-- [perf] Overhaul of block indexing. Now uses a Iterator to only collect batches of blocks from the database, 
-taking advantage of sequential read-speeds. Gathering blocks by RPC is no longer done.
+- [Changed] Postgres SQL queries are now type checked
+- [QoL] Refactor file layout to `substrate-archive` and `substrate-archive-backend`.
+- [perf] Decouple Database actors
+- [QoL] upgrade to SQLx 0.5.0
+- [perf] Overhaul of block indexing. Now uses a Iterator to only collect batches of blocks from the database,
+taking advantage of the better reading performance of sequential data access. Gathering blocks by RPC is no longer done.
   - [perf] a new module `runtime_version_cache` is introduced in order to cache and run a binary search on runtime version & blocks.
 - [perf] better queries for the set difference between the storage and blocks table
    - makes querying for missing storage more efficient
 - [err] Better handling of SQL errors
-- [perf] switch to a leaner, 'lower-level' actor framework (xtra) 
+- [perf] switch to a leaner, 'lower-level' actor framework (xtra)
 - [perf] switch to a background-task-queue for executing blocks. This uses significantly less memory and
   persists blocks that need to be executed on-disk.
 - [QoL] remove the last frame dependency, `frame-system`. Archive now relies only on generic traits defined in substrate-core.
 - new method of batch inserts avoids starving the Postgres Pool of connections
+- [perf] Notification stream from Postgres indexes storage changes in the background.
+- [Changed] Switch substrate-archive `LocalCallExecutor` with Substrate's `LocalCallExecutor`
 
 ## Polkadot Archive
 - [Changed] Config file is now optional. Can configure polkadot archive entirely through environment variables.
-  - the environment variables that need to be set are `CHAIN_DATA_DB` and `DATABASE_URL`. 
+  - the environment variables that need to be set are `CHAIN_DATA_DB` and `DATABASE_URL`.
 - [Changed] Polkadot archive will archive `polkadot` by default if the `--chain` CLI option is not passed.
 - [Changed] remove `rpc_url` from the polkadot-archive TOML configuration file
 - [Changed] All options in config file apart from `db_url`.
@@ -54,5 +68,5 @@ taking advantage of sequential read-speeds. Gathering blocks by RPC is no longer
 # **[v0.3.0]**
 - Use a rocksdb-backed substrate client instead of RPC for indexing
 - Create a CLI for indexing kusama
-- New PostgreSQL Schema 
+- New PostgreSQL Schema
 - Actors to model dataflow