Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Service(Client(Execution(Other("Exported method GrandpaApi_grandpa_authorities is not found")))) #4838

Open
2 tasks done
letsbangout opened this issue Jun 19, 2024 · 17 comments
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known.

Comments

@letsbangout
Copy link

Is there an existing issue?

  • I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

  • This is not a support question.

Description of bug

Polkadot substrate-node fails to launch with asset-hub-[polkadot(-genesis)] or [westend]. Throws the following output:

root@assethub-phx4:~# ./polkadot-sdk/target/release/substrate-node --chain=./polkadot-sdk/cumulus/parachains/chain-specs/asset-hub-polkadot.json
2024-06-19 19:04:32 Substrate Node
2024-06-19 19:04:32 ✌️  version 3.0.0-dev-55a13abcd2f
2024-06-19 19:04:32 ❤️  by Parity Technologies <[email protected]>, 2017-2024
2024-06-19 19:04:32 📋 Chain specification: Polkadot Asset Hub
2024-06-19 19:04:32 🏷  Node name: necessary-ray-1865
2024-06-19 19:04:32 👤 Role: FULL
2024-06-19 19:04:32 💾 Database: RocksDb at /root/.local/share/substrate-node/chains/asset-hub-polkadot/db/full
2024-06-19 19:04:35 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
Error: Service(Client(Execution(Other("Exported method GrandpaApi_grandpa_authorities is not found"))))
root@assethub-phx4:~# ./polkadot-sdk/target/release/substrate-node --chain=./polkadot-sdk/cumulus/parachains/chain-specs/asset-hub-polkadot.json --no-grandpa
2024-06-19 19:04:46 Substrate Node
2024-06-19 19:04:46 ✌️  version 3.0.0-dev-55a13abcd2f
2024-06-19 19:04:46 ❤️  by Parity Technologies <[email protected]>, 2017-2024
2024-06-19 19:04:46 📋 Chain specification: Polkadot Asset Hub
2024-06-19 19:04:46 🏷  Node name: scandalous-chalk-9680
2024-06-19 19:04:46 👤 Role: FULL
2024-06-19 19:04:46 💾 Database: RocksDb at /root/.local/share/substrate-node/chains/asset-hub-polkadot/db/full
2024-06-19 19:04:48 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
Error: Service(Client(Execution(Other("Exported method GrandpaApi_grandpa_authorities is not found"))))
root@assethub-phx4:~#

Steps to reproduce

  1. Follow the guide at https://github.com/paritytech/polkadot-sdk/tree/master/cumulus#asset-hub- (this made very little sense to me, it's still not apparent if I cloned the right repo)
  2. Start with a freshly installed Ubuntu 20.04 LTS:

apt -y update
apt -y install build-essential
apt install --assume-yes git clang curl libssl-dev llvm libudev-dev make protobuf-compiler
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
git clone https://github.com/paritytech/polkadot-sdk
cd polkadot-sdk
cargo build --release --locked --bin substrate-node

  1. Launch as per above
@letsbangout letsbangout added I10-unconfirmed Issue might be valid, but it's not yet known. I2-bug The node fails to follow expected behavior. labels Jun 19, 2024
@letsbangout
Copy link
Author

It's worthy of note, I have an existing node running on the 'polkadot' binary provided by the project's apt repo, also on Ubuntu 20.04 LTS. When I copy the .json file above to that node and launch with the polkadot binary, the error output identical to above.

@bkchr
Copy link
Member

bkchr commented Jun 20, 2024

If you want to start a parachain, use polkadot-parachain. If you want to launch a relay chain, use polkadot as binary. That will make it work.

@bkchr bkchr closed this as completed Jun 20, 2024
@letsbangout
Copy link
Author

I wouldn't say I'm trying to "start" a parachain. I'm trying to run a node on asset-hub-polkadot. It doesn't work with the 'polkadot' binary as aforementioned. And I believe i've already tried with 'polkadot-parachain' with exactly the same result. Perhaps you could point to a guide showing how to get this going, because I don't think one exists and there is a gaping hole in the documentation in this regard.

@bkchr
Copy link
Member

bkchr commented Jun 20, 2024

@letsbangout
Copy link
Author

https://github.com/paritytech/polkadot-sdk/tree/master/cumulus#asset-hub-

@kianenigma we need to make the more prominent.

Thanks for the reply. Now it seems we're getting somewhere :)

Note that I quoted this exact URL in my bug report. I hope you'll forgive my ignorance, mostly not knowing how things are done in the "polkadot world". But I am directed by your URL that I should build a binary called polkadot-parachain. It mentions "This repository also contains the Asset Hub runtimes." I presume that repo is in fact the same one where this documentation lies, either https://github.com/paritytech/polkadot-sdk or https://github.com/paritytech/polkadot-sdk/cumulus

I've tried building polkadot-parachain using the command supplied by the document, but it seems that is not a valid target:

root@assethub-phx4:~# git clone https://github.com/paritytech/polkadot-sdk
Cloning into 'polkadot-sdk'...
remote: Enumerating objects: 2605358, done.
remote: Counting objects: 100% (25315/25315), done.
remote: Compressing objects: 100% (6028/6028), done.
remote: Total 2605358 (delta 17853), reused 23721 (delta 16527), pack-reused 2580043
Receiving objects: 100% (2605358/2605358), 4.76 GiB | 28.61 MiB/s, done.
Resolving deltas: 100% (2215453/2215453), done.
Updating files: 100% (5900/5900), done.
root@assethub-phx4:~#
root@assethub-phx4:~# cd polkadot-sdk/
root@assethub-phx4:~/polkadot-sdk# cargo build --release --locked --bin polkadot-parachain
warning: /root/polkadot-sdk/cumulus/client/relay-chain-rpc-interface/Cargo.toml: `default_features` is deprecated in favor of `default-features` and will not work in the 2024 edition
(in the `smoldot` dependency)
warning: /root/polkadot-sdk/cumulus/client/relay-chain-rpc-interface/Cargo.toml: `default_features` is deprecated in favor of `default-features` and will not work in the 2024 edition
(in the `smoldot-light` dependency)
    Updating crates.io index
  Downloaded curve25519-dalek v4.1.3
  Downloaded 1 crate (306.0 KB) in 0.52s
error: no bin target named `polkadot-parachain`.
Available bin targets:
    polkadot
    polkadot-execute-worker
    polkadot-prepare-worker
    substrate-node

root@assethub-phx4:~/polkadot-sdk# cd cumulus/
root@assethub-phx4:~/polkadot-sdk/cumulus# cargo build --release --locked --bin polkadot-parachain
warning: /root/polkadot-sdk/cumulus/client/relay-chain-rpc-interface/Cargo.toml: `default_features` is deprecated in favor of `default-features` and will not work in the 2024 edition
(in the `smoldot` dependency)
warning: /root/polkadot-sdk/cumulus/client/relay-chain-rpc-interface/Cargo.toml: `default_features` is deprecated in favor of `default-features` and will not work in the 2024 edition
(in the `smoldot-light` dependency)
error: no bin target named `polkadot-parachain`.
Available bin targets:
    polkadot
    polkadot-execute-worker
    polkadot-prepare-worker
    substrate-node

root@assethub-phx4:~/polkadot-sdk/cumulus#

As per above I have already tried the substrate-node target which doesn't work. Hence why I've treated this, heretofore, as a bug.

Finally, for the sake of completeness, I did build the polkadot target in the manner directed by the URL. That throws substantially the same error as in the initial bug report:

root@assethub-phx4:~/polkadot-sdk# ./target/release/polkadot --chain ./cumulus/parachains/chain-specs/asset-hub-polkadot.json
2024-06-21 03:13:19 Parity Polkadot
2024-06-21 03:13:19 ✌️  version 1.13.0-74decbbdf22
2024-06-21 03:13:19 ❤️  by Parity Technologies <[email protected]>, 2017-2024
2024-06-21 03:13:19 📋 Chain specification: Polkadot Asset Hub
2024-06-21 03:13:19 🏷  Node name: unequal-love-9657
2024-06-21 03:13:19 👤 Role: FULL
2024-06-21 03:13:19 💾 Database: RocksDb at /root/.local/share/polkadot/chains/asset-hub-polkadot/db/full
2024-06-21 03:13:23 🔨 Initializing Genesis block/state (state: 0xc1ef…2c80, header-hash: 0x68d5…de2f)
2024-06-21 03:13:23 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2024-06-21 03:13:23 Essential task `txpool-background` failed. Shutting down service.
Error:
   0: Execution failed: Other: Exported method GrandpaApi_grandpa_authorities is not found

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
root@assethub-phx4:~/polkadot-sdk#

Is there anything else I should try, or shall we reopen and call this one a bug? Thanks again. :)

@bkchr bkchr reopened this Jun 21, 2024
@bkchr
Copy link
Member

bkchr commented Jun 21, 2024

cargo run -p polkadot-parachain-bin --release -- --chain asset-hub-polkadot this will start the node and let it sync the chain.

@letsbangout
Copy link
Author

Fascinating. After running the above command, cargo decided to build me a binary called polkadot-parachain. After the first invocation via cargo, I can now invoke it directly per below. Hopefully this report has been beneficial for you guys, it certainly has for me. Please let me know if there is any further testing you'd like me to do in my environment. Thanks again.

root@assethub-phx4:~/polkadot-sdk# ./target/release/polkadot-parachain --chain asset-hub-polkadot
2024-06-22 06:39:59 Polkadot parachain
2024-06-22 06:40:00 ✌️  version 4.0.0-74decbbdf22
2024-06-22 06:40:00 ❤️  by Parity Technologies <[email protected]>, 2017-2024
2024-06-22 06:40:00 📋 Chain specification: Polkadot Asset Hub
2024-06-22 06:40:00 🏷  Node name: nutritious-burn-7149
2024-06-22 06:40:00 👤 Role: FULL
2024-06-22 06:40:00 💾 Database: RocksDb at /root/.local/share/polkadot-parachain/chains/asset-hub-polkadot/db/full
2024-06-22 06:40:02 Parachain id: Id(1000)
2024-06-22 06:40:02 Parachain Account: 5Ec4AhPZk8STuex8Wsi9TwDtJQxKqzPJRCH7348Xtcs9vZLJ
2024-06-22 06:40:02 Is collating: no
2024-06-22 06:44:15 [Relaychain] 🏷  Local node identity is: 12D3KooWAYXELyWFkCj1VS51v384ZcJS74q5vVRKf4RCxFd3cKWa
2024-06-22 06:44:15 [Relaychain] Running libp2p network backend
2024-06-22 06:44:15 [Relaychain] 💻 Operating system: linux
2024-06-22 06:44:15 [Relaychain] 💻 CPU architecture: x86_64
2024-06-22 06:44:15 [Relaychain] 💻 Target environment: gnu
2024-06-22 06:44:15 [Relaychain] 💻 CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
2024-06-22 06:44:15 [Relaychain] 💻 CPU cores: 1
2024-06-22 06:44:15 [Relaychain] 💻 Memory: 14637MB
2024-06-22 06:44:15 [Relaychain] 💻 Kernel: 5.4.0-182-generic
2024-06-22 06:44:15 [Relaychain] 💻 Linux distribution: Ubuntu 20.04.6 LTS
2024-06-22 06:44:15 [Relaychain] 💻 Virtual machine: yes
2024-06-22 06:44:15 [Relaychain] 📦 Highest known block at #7673996
2024-06-22 06:44:15 [Relaychain] 〽️ Prometheus exporter started at 127.0.0.1:9616
2024-06-22 06:44:15 [Relaychain] Running JSON-RPC server: addr=127.0.0.1:9945, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2024-06-22 06:44:15 [Relaychain] 🏁 CPU score: 634.64 MiBs
2024-06-22 06:44:15 [Relaychain] 🏁 Memory score: 8.64 GiBs
2024-06-22 06:44:15 [Relaychain] 🏁 Disk score (seq. writes): 975.63 MiBs
2024-06-22 06:44:15 [Relaychain] 🏁 Disk score (rand. writes): 225.69 MiBs
2024-06-22 06:44:15 [Parachain] Using default protocol ID "sup" because none is configured in the chain specs
2024-06-22 06:44:15 [Parachain] 🏷  Local node identity is: 12D3KooWKqs9B9LAPiVBxHYzCDdXrhTi17ZFP6795oVC3Q6Ymqs6
2024-06-22 06:44:15 [Parachain] Running libp2p network backend
2024-06-22 06:44:15 [Parachain] 💻 Operating system: linux
2024-06-22 06:44:15 [Parachain] 💻 CPU architecture: x86_64
2024-06-22 06:44:15 [Parachain] 💻 Target environment: gnu
2024-06-22 06:44:15 [Parachain] 💻 CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
2024-06-22 06:44:15 [Parachain] 💻 CPU cores: 1
2024-06-22 06:44:15 [Parachain] 💻 Memory: 14637MB
2024-06-22 06:44:15 [Parachain] 💻 Kernel: 5.4.0-182-generic
2024-06-22 06:44:15 [Parachain] 💻 Linux distribution: Ubuntu 20.04.6 LTS
2024-06-22 06:44:15 [Parachain] 💻 Virtual machine: yes
2024-06-22 06:44:15 [Parachain] 📦 Highest known block at #6527496
2024-06-22 06:44:15 [Parachain] 〽️ Prometheus exporter started at 127.0.0.1:9615
2024-06-22 06:44:16 [Parachain] Running JSON-RPC server: addr=127.0.0.1:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2024-06-22 06:44:16 [Parachain] 🏁 CPU score: 634.64 MiBs
2024-06-22 06:44:16 [Parachain] 🏁 Memory score: 8.64 GiBs
2024-06-22 06:44:16 [Parachain] 🏁 Disk score (seq. writes): 975.63 MiBs
2024-06-22 06:44:16 [Parachain] 🏁 Disk score (rand. writes): 225.69 MiBs
2024-06-22 06:44:16 [Relaychain] 🔍 Discovered new external address for our node: /ip4/184.164.75.37/tcp/30334/ws/p2p/12D3KooWAYXELyWFkCj1VS51v384ZcJS74q5vVRKf4RCxFd3cKWa
2024-06-22 06:44:16 [Parachain] 🔍 Discovered new external address for our node: /ip4/184.164.75.37/tcp/30333/ws/p2p/12D3KooWKqs9B9LAPiVBxHYzCDdXrhTi17ZFP6795oVC3Q6Ymqs6
2024-06-22 06:44:17 [Relaychain] 💔 The bootnode you want to connect to at `/dns/polkadot-bootnode.turboflakes.io/tcp/30300/p2p/12D3KooWHJBMZgt7ymAdTRtadPcGXpJw79vBGe8z53r9JMkZW7Ha` provided a different peer ID `12D3KooWHbq8r4UWhXjuuW3cfwmc6ebFaTH6sQ6X43CUJKMMrMQg` than the one you expect `12D3KooWHJBMZgt7ymAdTRtadPcGXpJw79vBGe8z53r9JMkZW7Ha`.
2024-06-22 06:44:20 [Relaychain] ⚙️  Syncing, target=#21324619 (5 peers), best: #7674037 (0xed3d…5e70), finalized #7673856 (0x0140…98f1), ⬇ 1.3MiB/s ⬆ 103.9kiB/s
2024-06-22 06:44:21 [Parachain] ⚙️  Syncing, target=#6527791 (8 peers), best: #6527496 (0x43f2…13da), finalized #1719 (0x35c6…23e1), ⬇ 873.6kiB/s ⬆ 5.9kiB/s
2024-06-22 06:44:23 [Parachain] ♻️  Reorg on #6527496,0x43f2…13da to #6527497,0x8d71…babc, common ancestor #6527495,0x13e1…72bd
2024-06-22 06:44:23 [Parachain] 🏆 Imported #6527497 (0x16db…8e04 → 0x8d71…babc)
2024-06-22 06:44:24 [Relaychain] 🏆 Imported #7674125 (0x5203…d37e → 0x16e0…fec8)

@bkchr
Copy link
Member

bkchr commented Jun 23, 2024

Okay perfect!

@letsbangout
Copy link
Author

Whereas this issue is still marked "open", and
Whereas the basis of this issue seems to be that polkadot-sdk is inadequately documented as to how to obtain a polkadot-parachain binary as we have seen:

I'd like to point out that there is another glaring error in the docs: I see no indication as to what kind of system resources polkadot-sdk needs to run a binary. In the URL we have both referenced above, https://github.com/paritytech/polkadot-sdk/tree/master/cumulus#asset-hub- there is no mention of i.e. how much RAM is appropriate to dedicate to this task. Neither does the Polkadot node guide (i.e. to run a Polkadot full node or archive node) at https://wiki.polkadot.network/docs/maintain-sync - no mention of system resources, in particular, RAM.

I have Kusama and Polkadot nodes running (on the Ubuntu 20.04 repo binary) for a few days now, both seem to only actually need about 1.6GB of RAM:

root@kusama-phx4:~# free
              total        used        free      shared  buff/cache   available
Mem:       14989076     1558112      146404        1432    13284560    13256632
Swap:             0           0           0
root@kusama-phx4:~# systemctl status kusama
● kusama.service - Kusama daemon
     Loaded: loaded (/etc/systemd/system/kusama.service; enabled; vendor preset>
     Active: active (running) since Wed 2024-06-19 17:37:17 UTC; 4 days ago
   Main PID: 83984 (kusama.sh)
      Tasks: 138 (limit: 17457)
     Memory: 13.6G

and

root@polkadot-phx4:~# free
              total        used        free      shared  buff/cache   available
Mem:       14989076     1683416      122564        1432    13183096    13131796
Swap:             0           0           0
root@polkadot-phx4:~# systemctl status polka
● polka.service - Polkadot daemon
     Loaded: loaded (/etc/systemd/system/polka.service; enabled; vendor preset:>
     Active: active (running) since Wed 2024-06-19 22:44:06 UTC; 4 days ago
   Main PID: 1414 (polka.sh)
      Tasks: 134 (limit: 17457)
     Memory: 13.7G

I am currently seeing an out-of-memory condition on polkadot-parachain. Before I treat this as a potentially separate issue, could you guys advise on how much RAM the parachain daemon should need to run? And perhaps also mark this issue for a documentation correction on multiple fronts, if you agree with my assessment?

Thanks.

@letsbangout
Copy link
Author

letsbangout commented Jun 24, 2024

Edit: I have the polkadot-parachain VM provisioned exactly as the others (polkadot/kusama): Ubuntu 20.04 LTS, 15GB RAM, 8 vCPU's on a dual Xeon 2695v4 machine, and NVMe disk space sufficient to run each chain as necessary.

The invocation is as follows:

root@assethub-phx4:~# cat launch.sh
#!/bin/bash
./polkadot-sdk/target/release/polkadot-parachain --chain asset-hub-polkadot

@bkchr
Copy link
Member

bkchr commented Jun 24, 2024

I am currently seeing an out-of-memory condition on polkadot-parachain. Before I treat this as a potentially separate issue, could you guys advise on how much RAM the parachain daemon should need to run?

Which version are you using?

@letsbangout
Copy link
Author

I am currently seeing an out-of-memory condition on polkadot-parachain. Before I treat this as a potentially separate issue, could you guys advise on how much RAM the parachain daemon should need to run?

Which version are you using?

This is the entire output, including version number. Note that it is quite slow to initialize the database, perhaps 5 minutes, which is much slower than my full archives of Polkadot and Kusama btw. Specifically I mean that the stage that says Database: RocksDb at /root/.local/share/polkadot-parachain/chains/asset-hub-polkadot/db/full hangs there for 5 or so minutes.

root@assethub-phx4:~# ./launch.sh
2024-06-24 06:12:33 Polkadot parachain
2024-06-24 06:12:33 ✌️  version 4.0.0-74decbbdf22
2024-06-24 06:12:33 ❤️  by Parity Technologies <[email protected]>, 2017-2024
2024-06-24 06:12:33 📋 Chain specification: Polkadot Asset Hub
2024-06-24 06:12:33 🏷  Node name: ordinary-ball-2313
2024-06-24 06:12:33 👤 Role: FULL
2024-06-24 06:12:33 💾 Database: RocksDb at /root/.local/share/polkadot-parachain/chains/asset-hub-polkadot/db/full
2024-06-24 06:12:36 Parachain id: Id(1000)
2024-06-24 06:12:36 Parachain Account: 5Ec4AhPZk8STuex8Wsi9TwDtJQxKqzPJRCH7348Xtcs9vZLJ
2024-06-24 06:12:36 Is collating: no
2024-06-24 06:19:10 [Relaychain] 🏷  Local node identity is: 12D3KooWAYXELyWFkCj1VS51v384ZcJS74q5vVRKf4RCxFd3cKWa
2024-06-24 06:19:10 [Relaychain] Running libp2p network backend
2024-06-24 06:19:10 [Relaychain] 💻 Operating system: linux
2024-06-24 06:19:10 [Relaychain] 💻 CPU architecture: x86_64
2024-06-24 06:19:10 [Relaychain] 💻 Target environment: gnu
2024-06-24 06:19:10 [Relaychain] 💻 CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
2024-06-24 06:19:10 [Relaychain] 💻 CPU cores: 1
2024-06-24 06:19:10 [Relaychain] 💻 Memory: 14637MB
2024-06-24 06:19:10 [Relaychain] 💻 Kernel: 5.4.0-182-generic
2024-06-24 06:19:10 [Relaychain] 💻 Linux distribution: Ubuntu 20.04.6 LTS
2024-06-24 06:19:10 [Relaychain] 💻 Virtual machine: yes
2024-06-24 06:19:10 [Relaychain] 📦 Highest known block at #7956480
2024-06-24 06:19:10 [Relaychain] 〽️ Prometheus exporter started at 127.0.0.1:9616
2024-06-24 06:19:10 [Relaychain] Running JSON-RPC server: addr=127.0.0.1:9945, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2024-06-24 06:19:10 [Relaychain] 🏁 CPU score: 642.85 MiBs
2024-06-24 06:19:10 [Relaychain] 🏁 Memory score: 7.93 GiBs
2024-06-24 06:19:10 [Relaychain] 🏁 Disk score (seq. writes): 964.45 MiBs
2024-06-24 06:19:10 [Relaychain] 🏁 Disk score (rand. writes): 249.91 MiBs
2024-06-24 06:19:10 [Parachain] Using default protocol ID "sup" because none is configured in the chain specs
2024-06-24 06:19:10 [Parachain] 🏷  Local node identity is: 12D3KooWKqs9B9LAPiVBxHYzCDdXrhTi17ZFP6795oVC3Q6Ymqs6
2024-06-24 06:19:10 [Parachain] Running libp2p network backend
2024-06-24 06:19:10 [Parachain] 💻 Operating system: linux
2024-06-24 06:19:10 [Parachain] 💻 CPU architecture: x86_64
2024-06-24 06:19:10 [Parachain] 💻 Target environment: gnu
2024-06-24 06:19:10 [Parachain] 💻 CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
2024-06-24 06:19:10 [Parachain] 💻 CPU cores: 1
2024-06-24 06:19:10 [Parachain] 💻 Memory: 14637MB
2024-06-24 06:19:10 [Parachain] 💻 Kernel: 5.4.0-182-generic
2024-06-24 06:19:10 [Parachain] 💻 Linux distribution: Ubuntu 20.04.6 LTS
2024-06-24 06:19:10 [Parachain] 💻 Virtual machine: yes
2024-06-24 06:19:10 [Parachain] 📦 Highest known block at #6535804
2024-06-24 06:19:10 [Parachain] 〽️ Prometheus exporter started at 127.0.0.1:9615
2024-06-24 06:19:10 [Parachain] Running JSON-RPC server: addr=127.0.0.1:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2024-06-24 06:19:10 [Parachain] 🏁 CPU score: 642.85 MiBs
2024-06-24 06:19:10 [Parachain] 🏁 Memory score: 7.93 GiBs
2024-06-24 06:19:10 [Parachain] 🏁 Disk score (seq. writes): 964.45 MiBs
2024-06-24 06:19:10 [Parachain] 🏁 Disk score (rand. writes): 249.91 MiBs
2024-06-24 06:19:10 [Relaychain] 🔍 Discovered new external address for our node: /ip4/184.164.75.37/tcp/30334/ws/p2p/12D3KooWAYXELyWFkCj1VS51v384ZcJS74q5vVRKf4RCxFd3cKWa
2024-06-24 06:19:10 [Parachain] 🔍 Discovered new external address for our node: /ip4/184.164.75.37/tcp/30333/ws/p2p/12D3KooWKqs9B9LAPiVBxHYzCDdXrhTi17ZFP6795oVC3Q6Ymqs6
2024-06-24 06:19:11 [Relaychain] 💔 The bootnode you want to connect to at `/dns/polkadot-bootnode.turboflakes.io/tcp/30300/p2p/12D3KooWHJBMZgt7ymAdTRtadPcGXpJw79vBGe8z53r9JMkZW7Ha` provided a different peer ID `12D3KooWHbq8r4UWhXjuuW3cfwmc6ebFaTH6sQ6X43CUJKMMrMQg` than the one you expect `12D3KooWHJBMZgt7ymAdTRtadPcGXpJw79vBGe8z53r9JMkZW7Ha`.
2024-06-24 06:19:15 [Relaychain] ⚙️  Syncing, target=#21352984 (5 peers), best: #7956621 (0xafa4…ca59), finalized #7955968 (0xae47…95da), ⬇ 6.8MiB/s ⬆ 59.5kiB/s
2024-06-24 06:19:15 [Parachain] ⚙️  Syncing, target=#6541792 (8 peers), best: #6535804 (0xb443…392e), finalized #60230 (0x826c…01fe), ⬇ 4.6MiB/s ⬆ 11.5kiB/s
2024-06-24 06:19:20 [Relaychain] ⚙️  Syncing 31.8 bps, target=#21352985 (5 peers), best: #7956780 (0x620b…f0fe), finalized #7955968 (0xae47…95da), ⬇ 622.4kiB/s ⬆ 94.7kiB/s
2024-06-24 06:19:20 [Parachain] ⚙️  Syncing 38.4 bps, target=#6541792 (8 peers), best: #6535996 (0x478d…7c02), finalized #60230 (0x826c…01fe), ⬇ 4.9MiB/s ⬆ 0.6kiB/s
2024-06-24 06:19:25 [Relaychain] ⚙️  Syncing 37.0 bps, target=#21352985 (7 peers), best: #7956965 (0x91e2…e1b1), finalized #7955968 (0xae47…95da), ⬇ 763.3kiB/s ⬆ 54.0kiB/s
2024-06-24 06:19:25 [Parachain] ⚙️  Syncing  0.0 bps, target=#6541793 (8 peers), best: #6535996 (0x478d…7c02), finalized #60230 (0x826c…01fe), ⬇ 444.2kiB/s ⬆ 0.5kiB/s
2024-06-24 06:19:30 [Relaychain] ⚙️  Syncing  5.4 bps, target=#21352986 (9 peers), best: #7956992 (0x6870…22aa), finalized #7955968 (0xae47…95da), ⬇ 278.3kiB/s ⬆ 144.9kiB/s
2024-06-24 06:19:30 [Parachain] ⚙️  Syncing  0.0 bps, target=#6541793 (8 peers), best: #6535996 (0x478d…7c02), finalized #60230 (0x826c…01fe), ⬇ 40.4kiB/s ⬆ 6.6kiB/s
2024-06-24 06:19:35 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21352987 (9 peers), best: #7956992 (0x6870…22aa), finalized #7955968 (0xae47…95da), ⬇ 305.5kiB/s ⬆ 43.9kiB/s
2024-06-24 06:19:35 [Parachain] ⚙️  Syncing 39.0 bps, target=#6541793 (7 peers), best: #6536191 (0xc34e…14d0), finalized #60230 (0x826c…01fe), ⬇ 299.6kiB/s ⬆ 4.0kiB/s
2024-06-24 06:19:40 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21352988 (11 peers), best: #7956992 (0x6870…22aa), finalized #7955968 (0xae47…95da), ⬇ 183.0kiB/s ⬆ 91.9kiB/s
2024-06-24 06:19:40 [Parachain] ⚙️  Syncing 171.8 bps, target=#6541794 (7 peers), best: #6537050 (0xbd39…3cfa), finalized #60230 (0x826c…01fe), ⬇ 2.7kiB/s ⬆ 0.9kiB/s
2024-06-24 06:19:45 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21352989 (12 peers), best: #7956992 (0x6870…22aa), finalized #7955968 (0xae47…95da), ⬇ 78.7kiB/s ⬆ 29.5kiB/s
2024-06-24 06:19:45 [Parachain] ⚙️  Syncing 184.0 bps, target=#6541794 (7 peers), best: #6537970 (0x27c3…8dd5), finalized #60230 (0x826c…01fe), ⬇ 1.2kiB/s ⬆ 1.3kiB/s
2024-06-24 06:19:50 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21352990 (11 peers), best: #7956992 (0x6870…22aa), finalized #7955968 (0xae47…95da), ⬇ 58.1kiB/s ⬆ 21.5kiB/s
2024-06-24 06:19:50 [Parachain] ⚙️  Syncing 78.8 bps, target=#6541795 (7 peers), best: #6538364 (0x1eb1…6d83), finalized #60230 (0x826c…01fe), ⬇ 6.7MiB/s ⬆ 9.5kiB/s
2024-06-24 06:19:55 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21352991 (11 peers), best: #7956992 (0x6870…22aa), finalized #7955968 (0xae47…95da), ⬇ 147.6kiB/s ⬆ 62.5kiB/s
2024-06-24 06:19:55 [Parachain] ⚙️  Syncing 125.8 bps, target=#6541795 (7 peers), best: #6538993 (0x8ee7…2575), finalized #60230 (0x826c…01fe), ⬇ 3.6MiB/s ⬆ 2.3kiB/s
[527309.194579] Out of memory: Killed process 397075 (polkadot-parach) total-vm:1243853456kB, anon-rss:14503208kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:37464kB oom_score_adj:0
./launch.sh: line 2: 397075 Killed                  ./polkadot-sdk/target/release/polkadot-parachain --chain asset-hub-polkadot
root@assethub-phx4:~#

@bkchr
Copy link
Member

bkchr commented Jun 24, 2024

Please run with -ldb=trace,state-db=trace. I have some assumption on what is going on.

@letsbangout
Copy link
Author

See below for the last 100 lines of the console log. If you need the whole thing I suppose I can pastebin it.

root@assethub-phx4:~# tail -n 100 ah-trace.log
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x95b4b43d00427f745548e97cad07462024064af475834c5012edb1e8ffa9adc8
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x95b4b43d00427f745548e97cad07462024064af475834c5012edb1e8ffa9adc8
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x95b4b43d00427f745548e97cad07462024064af475834c5012edb1e8ffa9adc8
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Releasing reference for 0x95b4b43d00427f745548e97cad07462024064af475834c5012edb1e8ffa9adc8
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x95b4b43d00427f745548e97cad07462024064af475834c5012edb1e8ffa9adc8
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x95b4b43d00427f745548e97cad07462024064af475834c5012edb1e8ffa9adc8
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x95b4b43d00427f745548e97cad07462024064af475834c5012edb1e8ffa9adc8
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db: [Parachain] Inserted uncanonicalized changeset 6544057.0 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf (51 inserted, 50 deleted)
2024-06-30 06:51:19.560 DEBUG tokio-runtime-worker db: [Parachain] DB Commit 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf (6544057), best=true, state=true, existing=false, finalized=false
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x95b4b43d00427f745548e97cad07462024064af475834c5012edb1e8ffa9adc8
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker db: [Parachain] Canonicalize block #6539960 (0xb4d6da9a3a9b20b7e6095c814efa8656079416c01e02a95baaf0bc865ad13ed9)
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db: [Parachain] Canonicalizing 0xb4d6da9a3a9b20b7e6095c814efa8656079416c01e02a95baaf0bc865ad13ed9
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xb4d6da9a3a9b20b7e6095c814efa8656079416c01e02a95baaf0bc865ad13ed9
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db: [Parachain] Discarding 1 records
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db: [Parachain] Adding to pruning window: 0xb4d6da9a3a9b20b7e6095c814efa8656079416c01e02a95baaf0bc865ad13ed9 (34 inserted, 34 deleted)
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db: [Parachain] Importing 6539960, base=6539704
2024-06-30 06:51:19.560 TRACE tokio-runtime-worker state-db: [Parachain] Pruning 0x1a8913f6711cefd4821221e3184f7ced2b0323f3400044c4c1f6336eae8cd774 (34 deleted)
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker db: [Parachain] DB Commit done 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Discarding unpinned non-canon block: 0xb4d6da9a3a9b20b7e6095c814efa8656079416c01e02a95baaf0bc865ad13ed9
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.561 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.562 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.563 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.563 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.563 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.571 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db::pin: [Parachain] Releasing reference for 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db: [Parachain] Inserted uncanonicalized changeset 6544058.0 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a (49 inserted, 48 deleted)
2024-06-30 06:51:19.572 DEBUG tokio-runtime-worker db: [Parachain] DB Commit 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a (6544058), best=true, state=true, existing=false, finalized=false
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa4c3efa6cd90604c56330bd9d5805b7dd9cb10293a0bd0e4e533e56e05496eaf
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker db: [Parachain] Canonicalize block #6539961 (0x80656d6f8b74d573794c252901615165ea7bc1e132abf26b3358130681fb7e7e)
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db: [Parachain] Canonicalizing 0x80656d6f8b74d573794c252901615165ea7bc1e132abf26b3358130681fb7e7e
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x80656d6f8b74d573794c252901615165ea7bc1e132abf26b3358130681fb7e7e
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db: [Parachain] Discarding 1 records
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db: [Parachain] Adding to pruning window: 0x80656d6f8b74d573794c252901615165ea7bc1e132abf26b3358130681fb7e7e (34 inserted, 34 deleted)
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db: [Parachain] Importing 6539961, base=6539705
2024-06-30 06:51:19.572 TRACE tokio-runtime-worker state-db: [Parachain] Pruning 0x0df3be128eb3bd5ffe9500650ce0b66457b886013576765915e5998510099e63 (42 deleted)
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker db: [Parachain] DB Commit done 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Discarding unpinned non-canon block: 0x80656d6f8b74d573794c252901615165ea7bc1e132abf26b3358130681fb7e7e
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.573 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.574 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.574 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.574 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.574 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a
2024-06-30 06:51:19.574 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x437211a0607308c191e427d6c213969facd00ba50d3e5a847966bb4d20e0569a    ./launch.sh: line 2: 417512 Killed                  ./polkadot-sdk/target/release/polkadot-parachain --chain asset-hub-polkadot -ldb=trace,state-db=trace
root@assethub-phx4:~#

@letsbangout
Copy link
Author

Out of curiosity I re-launched this VM with 25GB of RAM, all other specs remaining equal to before. I no longer get an out-of-memory condition when launching. However, I see that despite having plenty of peers, the Parachain is no longer being updated. I'm mentioning this in case it is somehow connected to the out-of-memory problem. Unless we need more Relaychain blocks in order to advance the Parachain?

See below.

root@assethub-phx4:~# tail ah-notrace.log  -f
2024-07-02 04:24:34 Polkadot parachain
2024-07-02 04:24:34 ✌️  version 4.0.0-74decbbdf22
2024-07-02 04:24:34 ❤️  by Parity Technologies <[email protected]>, 2017-2024
2024-07-02 04:24:34 📋 Chain specification: Polkadot Asset Hub
2024-07-02 04:24:34 🏷  Node name: flimsy-rest-5678
2024-07-02 04:24:34 👤 Role: FULL
2024-07-02 04:24:34 💾 Database: RocksDb at /root/.local/share/polkadot-parachain/chains/asset-hub-polkadot/db/full
2024-07-02 04:24:37 Parachain id: Id(1000)
2024-07-02 04:24:37 Parachain Account: 5Ec4AhPZk8STuex8Wsi9TwDtJQxKqzPJRCH7348Xtcs9vZLJ
2024-07-02 04:24:37 Is collating: no
2024-07-02 04:28:13 [Relaychain] 🏷  Local node identity is: 12D3KooWAYXELyWFkCj1VS51v384ZcJS74q5vVRKf4RCxFd3cKWa
2024-07-02 04:28:13 [Relaychain] Running libp2p network backend
2024-07-02 04:28:13 [Relaychain] 💻 Operating system: linux
2024-07-02 04:28:13 [Relaychain] 💻 CPU architecture: x86_64
2024-07-02 04:28:13 [Relaychain] 💻 Target environment: gnu
2024-07-02 04:28:13 [Relaychain] 💻 CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
2024-07-02 04:28:13 [Relaychain] 💻 CPU cores: 1
2024-07-02 04:28:13 [Relaychain] 💻 Memory: 24418MB
2024-07-02 04:28:13 [Relaychain] 💻 Kernel: 5.4.0-187-generic
2024-07-02 04:28:13 [Relaychain] 💻 Linux distribution: Ubuntu 20.04.6 LTS
2024-07-02 04:28:13 [Relaychain] 💻 Virtual machine: yes
2024-07-02 04:28:13 [Relaychain] 📦 Highest known block at #7986949
2024-07-02 04:28:13 [Relaychain] 〽️ Prometheus exporter started at 127.0.0.1:9616
2024-07-02 04:28:13 [Relaychain] Running JSON-RPC server: addr=127.0.0.1:9945, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2024-07-02 04:28:13 [Relaychain] 🏁 CPU score: 661.03 MiBs
2024-07-02 04:28:13 [Relaychain] 🏁 Memory score: 8.06 GiBs
2024-07-02 04:28:13 [Relaychain] 🏁 Disk score (seq. writes): 994.61 MiBs
2024-07-02 04:28:13 [Relaychain] 🏁 Disk score (rand. writes): 241.32 MiBs
2024-07-02 04:28:13 [Parachain] Using default protocol ID "sup" because none is configured in the chain specs
2024-07-02 04:28:13 [Parachain] 🏷  Local node identity is: 12D3KooWKqs9B9LAPiVBxHYzCDdXrhTi17ZFP6795oVC3Q6Ymqs6
2024-07-02 04:28:13 [Parachain] Running libp2p network backend
2024-07-02 04:28:13 [Parachain] 💻 Operating system: linux
2024-07-02 04:28:13 [Parachain] 💻 CPU architecture: x86_64
2024-07-02 04:28:13 [Parachain] 💻 Target environment: gnu
2024-07-02 04:28:13 [Parachain] 💻 CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
2024-07-02 04:28:13 [Parachain] 💻 CPU cores: 1
2024-07-02 04:28:13 [Parachain] 💻 Memory: 24418MB
2024-07-02 04:28:13 [Parachain] 💻 Kernel: 5.4.0-187-generic
2024-07-02 04:28:13 [Parachain] 💻 Linux distribution: Ubuntu 20.04.6 LTS
2024-07-02 04:28:13 [Parachain] 💻 Virtual machine: yes
2024-07-02 04:28:13 [Parachain] 📦 Highest known block at #6559571
2024-07-02 04:28:13 [Parachain] 〽️ Prometheus exporter started at 127.0.0.1:9615
2024-07-02 04:28:13 [Parachain] Running JSON-RPC server: addr=127.0.0.1:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2024-07-02 04:28:13 [Parachain] 🏁 CPU score: 661.03 MiBs
2024-07-02 04:28:13 [Parachain] 🏁 Memory score: 8.06 GiBs
2024-07-02 04:28:13 [Parachain] 🏁 Disk score (seq. writes): 994.61 MiBs
2024-07-02 04:28:13 [Parachain] 🏁 Disk score (rand. writes): 241.32 MiBs
2024-07-02 04:28:14 [Relaychain] 🔍 Discovered new external address for our node: /ip4/184.164.75.37/tcp/30334/ws/p2p/12D3KooWAYXELyWFkCj1VS51v384ZcJS74q5vVRKf4RCxFd3cKWa
2024-07-02 04:28:14 [Parachain] 🔍 Discovered new external address for our node: /ip4/184.164.75.37/tcp/30333/ws/p2p/12D3KooWKqs9B9LAPiVBxHYzCDdXrhTi17ZFP6795oVC3Q6Ymqs6
2024-07-02 04:28:18 [Relaychain] ⚙️  Syncing, target=#21466551 (1 peers), best: #7986965 (0x3291…0632), finalized #7986688 (0xf21b…4f10), ⬇ 409.1kiB/s ⬆ 67.7kiB/s
2024-07-02 04:28:18 [Parachain] ⚙️  Syncing, target=#6597689 (5 peers), best: #6559571 (0xb396…d61a), finalized #60230 (0x826c…01fe), ⬇ 1.9MiB/s ⬆ 8.8kiB/s
2024-07-02 04:28:23 [Relaychain] ⚙️  Syncing 38.2 bps, target=#21466552 (2 peers), best: #7987156 (0xa9b1…74f1), finalized #7986688 (0xf21b…4f10), ⬇ 893.3kiB/s ⬆ 63.6kiB/s
2024-07-02 04:28:23 [Parachain] ⚙️  Syncing 38.4 bps, target=#6597690 (5 peers), best: #6559763 (0xed3a…3bfb), finalized #60230 (0x826c…01fe), ⬇ 3.6MiB/s ⬆ 1.4kiB/s
2024-07-02 04:28:28 [Relaychain] ⚙️  Syncing  8.8 bps, target=#21466553 (3 peers), best: #7987200 (0x59f3…ef04), finalized #7986688 (0xf21b…4f10), ⬇ 2.1MiB/s ⬆ 56.9kiB/s
2024-07-02 04:28:28 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597690 (5 peers), best: #6559763 (0xed3a…3bfb), finalized #60230 (0x826c…01fe), ⬇ 3.6MiB/s ⬆ 5.0kiB/s
2024-07-02 04:28:29 [Parachain] Report 12D3KooWJEGZgNHRGWw4JAgPyxU4tKS6WDNNR5A5gCrKRC7db5Je: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.
2024-07-02 04:28:33 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21466554 (4 peers), best: #7987200 (0x59f3…ef04), finalized #7986688 (0xf21b…4f10), ⬇ 3.3MiB/s ⬆ 99.3kiB/s
2024-07-02 04:28:33 [Parachain] ⚙️  Syncing 72.6 bps, target=#6597691 (7 peers), best: #6560126 (0x12f6…e4fd), finalized #60230 (0x826c…01fe), ⬇ 1.2MiB/s ⬆ 3.5kiB/s
2024-07-02 04:28:38 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21466555 (6 peers), best: #7987200 (0x59f3…ef04), finalized #7986688 (0xf21b…4f10), ⬇ 1.1MiB/s ⬆ 36.3kiB/s
2024-07-02 04:28:38 [Parachain] ⚙️  Syncing 157.2 bps, target=#6597691 (7 peers), best: #6560912 (0xdf02…23bb), finalized #60230 (0x826c…01fe), ⬇ 532.0kiB/s ⬆ 7.5kiB/s
2024-07-02 04:28:43 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21466556 (7 peers), best: #7987200 (0x59f3…ef04), finalized #7986688 (0xf21b…4f10), ⬇ 231.5kiB/s ⬆ 113.9kiB/s
2024-07-02 04:28:43 [Parachain] ⚙️  Syncing 156.4 bps, target=#6597692 (7 peers), best: #6561694 (0xeeb7…3ae0), finalized #60230 (0x826c…01fe), ⬇ 84.9kiB/s ⬆ 9.5kiB/s
2024-07-02 04:28:48 [Relaychain] ⚙️  Syncing 11.2 bps, target=#21466556 (9 peers), best: #7987256 (0xb296…b0c2), finalized #7987200 (0x59f3…ef04), ⬇ 337.0kiB/s ⬆ 66.9kiB/s
2024-07-02 04:28:48 [Parachain] ⚙️  Syncing 61.8 bps, target=#6597692 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 3.1MiB/s ⬆ 4.6kiB/s
2024-07-02 04:28:53 [Relaychain] ⚙️  Syncing 36.8 bps, target=#21466557 (9 peers), best: #7987440 (0x9ac2…8ec1), finalized #7987200 (0x59f3…ef04), ⬇ 486.2kiB/s ⬆ 47.3kiB/s
2024-07-02 04:28:53 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597692 (6 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 4.8MiB/s ⬆ 1.0kiB/s
2024-07-02 04:28:58 [Relaychain] ⚙️  Syncing 37.4 bps, target=#21466558 (9 peers), best: #7987627 (0x26cb…adea), finalized #7987200 (0x59f3…ef04), ⬇ 961.2kiB/s ⬆ 92.2kiB/s
2024-07-02 04:28:58 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597693 (6 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 1.5MiB/s ⬆ 0.7kiB/s
2024-07-02 04:29:03 [Relaychain] ⚙️  Syncing 27.6 bps, target=#21466559 (9 peers), best: #7987765 (0x7b98…8404), finalized #7987712 (0xcebb…6a2e), ⬇ 578.9kiB/s ⬆ 51.2kiB/s
2024-07-02 04:29:03 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597693 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 88.2kiB/s ⬆ 0.8kiB/s
2024-07-02 04:29:08 [Relaychain] ⚙️  Syncing 41.4 bps, target=#21466560 (9 peers), best: #7987972 (0x2590…e073), finalized #7987712 (0xcebb…6a2e), ⬇ 705.0kiB/s ⬆ 35.3kiB/s
2024-07-02 04:29:08 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597694 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 0.8kiB/s ⬆ 0.4kiB/s
2024-07-02 04:29:13 [Relaychain] ⚙️  Syncing 37.1 bps, target=#21466561 (9 peers), best: #7988158 (0x336f…1f0b), finalized #7987712 (0xcebb…6a2e), ⬇ 677.0kiB/s ⬆ 36.1kiB/s
2024-07-02 04:29:13 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597694 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 0.7kiB/s ⬆ 0.7kiB/s
2024-07-02 04:29:18 [Relaychain] ⚙️  Syncing 24.6 bps, target=#21466562 (9 peers), best: #7988281 (0x56d1…4049), finalized #7988224 (0x51f6…64d9), ⬇ 153.5kiB/s ⬆ 63.7kiB/s
2024-07-02 04:29:18 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597694 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 0.4kiB/s ⬆ 0.4kiB/s
2024-07-02 04:29:23 [Relaychain] ⚙️  Syncing 31.6 bps, target=#21466562 (9 peers), best: #7988439 (0x5095…06c5), finalized #7988224 (0x51f6…64d9), ⬇ 96.4kiB/s ⬆ 50.5kiB/s
2024-07-02 04:29:23 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597695 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 1.4kiB/s ⬆ 0.4kiB/s
2024-07-02 04:29:28 [Relaychain] ⚙️  Syncing 33.2 bps, target=#21466563 (10 peers), best: #7988605 (0x013f…f05f), finalized #7988224 (0x51f6…64d9), ⬇ 159.1kiB/s ⬆ 81.3kiB/s
2024-07-02 04:29:28 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597695 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 1.1kiB/s ⬆ 0.5kiB/s
2024-07-02 04:29:33 [Relaychain] ⚙️  Syncing 31.2 bps, target=#21466564 (10 peers), best: #7988761 (0x35ba…6bac), finalized #7988736 (0xfd70…d2c8), ⬇ 106.4kiB/s ⬆ 57.1kiB/s
2024-07-02 04:29:33 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597695 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 0.3kiB/s ⬆ 0.3kiB/s
2024-07-02 04:29:38 [Relaychain] ⚙️  Syncing 35.4 bps, target=#21466565 (10 peers), best: #7988938 (0x6ed3…f188), finalized #7988736 (0xfd70…d2c8), ⬇ 62.4kiB/s ⬆ 27.6kiB/s
2024-07-02 04:29:38 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597696 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 0.3kiB/s ⬆ 12 B/s
2024-07-02 04:29:43 [Relaychain] ⚙️  Syncing 38.2 bps, target=#21466566 (10 peers), best: #7989129 (0xbafb…6ece), finalized #7988736 (0xfd70…d2c8), ⬇ 2.9MiB/s ⬆ 60.5kiB/s
2024-07-02 04:29:43 [Parachain] ⚙️  Syncing  0.0 bps, target=#6597696 (7 peers), best: #6562003 (0x9d55…70c7), finalized #60230 (0x826c…01fe), ⬇ 25 B/s ⬆ 25 B/s

@letsbangout
Copy link
Author

Scratch that, I did get another out-of-memory condition, running on 25GB RAM. All other parameters being equal.

It's being invoked with:

root@assethub-phx4:~# root@assethub-phx4:~# cat launch.sh
#!/bin/bash

./polkadot-sdk/target/release/polkadot-parachain --chain asset-hub-polkadot -ldb=trace,state-db=trace

Here are the last 200 lines of the trace:

root@assethub-phx4:~# tail -n 200 ah-another-trace.log
2024-07-11 18:49:54.228 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.228 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Releasing reference for 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db: [Parachain] Inserted uncanonicalized changeset 6593400.0 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce (52 inserted, 53 deleted)
2024-07-11 18:49:54.234 DEBUG tokio-runtime-worker db: [Parachain] DB Commit 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce (6593400), best=true, state=true, existing=false, finalized=false
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x43388e2e5169bebc849183eaab759ca53cbf1bd5f67502832ca22968e15d0b85
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker db: [Parachain] Canonicalize block #6589303 (0x6b3bb006a9cd04da92cc31236b6e262924a2ed7fa44fd2a7220d4a581e849726)
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db: [Parachain] Canonicalizing 0x6b3bb006a9cd04da92cc31236b6e262924a2ed7fa44fd2a7220d4a581e849726
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6b3bb006a9cd04da92cc31236b6e262924a2ed7fa44fd2a7220d4a581e849726
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db: [Parachain] Discarding 1 records
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db: [Parachain] Adding to pruning window: 0x6b3bb006a9cd04da92cc31236b6e262924a2ed7fa44fd2a7220d4a581e849726 (47 inserted, 47 deleted) 
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db: [Parachain] Importing 6589303, base=6589047
2024-07-11 18:49:54.234 TRACE tokio-runtime-worker state-db: [Parachain] Pruning 0x1e4bc38bc5cb1d1e705febdca43e70f4b77088560224b8f690a40a376b4e6de9 (47 deleted)
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker db: [Parachain] DB Commit done 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Discarding unpinned non-canon block: 0x6b3bb006a9cd04da92cc31236b6e262924a2ed7fa44fd2a7220d4a581e849726
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.235 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.236 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.237 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.237 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.237 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.237 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.237 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.237 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.241 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.241 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db::pin: [Parachain] Releasing reference for 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db: [Parachain] Inserted uncanonicalized changeset 6593401.0 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb (48 inserted, 49 deleted)
2024-07-11 18:49:54.242 DEBUG tokio-runtime-worker db: [Parachain] DB Commit 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb (6593401), best=true, state=true, existing=false, finalized=false
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x9fc234cb41790a52501ef62ccd010bd44cd10b4f8f58e84c3b77e229f24fecce
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker db: [Parachain] Canonicalize block #6589304 (0x54b2852748321e7d6ca8b0143e37aa2362dca49bed552728c2fdcd0ecb54a894)
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db: [Parachain] Canonicalizing 0x54b2852748321e7d6ca8b0143e37aa2362dca49bed552728c2fdcd0ecb54a894
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x54b2852748321e7d6ca8b0143e37aa2362dca49bed552728c2fdcd0ecb54a894
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db: [Parachain] Discarding 1 records
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db: [Parachain] Adding to pruning window: 0x54b2852748321e7d6ca8b0143e37aa2362dca49bed552728c2fdcd0ecb54a894 (49 inserted, 49 deleted) 
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db: [Parachain] Importing 6589304, base=6589048
2024-07-11 18:49:54.242 TRACE tokio-runtime-worker state-db: [Parachain] Pruning 0xc9e18268e42d202893fc42fb3641c9d9fa3bedd2b800f5197d02a961168c7376 (49 deleted)
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker db: [Parachain] DB Commit done 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Discarding unpinned non-canon block: 0x54b2852748321e7d6ca8b0143e37aa2362dca49bed552728c2fdcd0ecb54a894
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.243 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.244 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.245 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.245 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.245 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.249 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db::pin: [Parachain] Releasing reference for 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db: [Parachain] Inserted uncanonicalized changeset 6593402.0 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8 (50 inserted, 49 deleted)
2024-07-11 18:49:54.250 DEBUG tokio-runtime-worker db: [Parachain] DB Commit 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8 (6593402), best=true, state=true, existing=false, finalized=false
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xf18a2ff6e221fbdd5ad0e5ebbf83fed21c6082fa2c7841070d839b80f7c1f9fb
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker db: [Parachain] Canonicalize block #6589305 (0xf43d3f057f4c6a685d188c67a729c5ee3087c2b3a4956e60e633f2be5e5d356b)
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db: [Parachain] Canonicalizing 0xf43d3f057f4c6a685d188c67a729c5ee3087c2b3a4956e60e633f2be5e5d356b
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xf43d3f057f4c6a685d188c67a729c5ee3087c2b3a4956e60e633f2be5e5d356b
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db: [Parachain] Discarding 1 records
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db: [Parachain] Adding to pruning window: 0xf43d3f057f4c6a685d188c67a729c5ee3087c2b3a4956e60e633f2be5e5d356b (50 inserted, 49 deleted) 
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db: [Parachain] Importing 6589305, base=6589049
2024-07-11 18:49:54.250 TRACE tokio-runtime-worker state-db: [Parachain] Pruning 0xf71ec50a252444fd2b64a535b6b825572f32fa7416754ad218921d6b74b3a7ba (44 deleted)
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker db: [Parachain] DB Commit done 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Discarding unpinned non-canon block: 0xf43d3f057f4c6a685d188c67a729c5ee3087c2b3a4956e60e633f2be5e5d356b
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.251 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.252 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.253 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.253 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.257 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.257 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.257 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db::pin: [Parachain] Releasing reference for 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db: [Parachain] Inserted uncanonicalized changeset 6593403.0 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7 (50 inserted, 50 deleted)
2024-07-11 18:49:54.258 DEBUG tokio-runtime-worker db: [Parachain] DB Commit 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7 (6593403), best=true, state=true, existing=false, finalized=false
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0x6712ad90e4c745c0b56522511082b452ea106f7c982fc2428723186cb51f02f8
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker db: [Parachain] Canonicalize block #6589306 (0x145903798eedd4e8458615aa2df8b8a59f337e40172b31b25fa0a9071c5fd999)
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db: [Parachain] Canonicalizing 0x145903798eedd4e8458615aa2df8b8a59f337e40172b31b25fa0a9071c5fd999
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0x145903798eedd4e8458615aa2df8b8a59f337e40172b31b25fa0a9071c5fd999
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db: [Parachain] Discarding 1 records
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db: [Parachain] Adding to pruning window: 0x145903798eedd4e8458615aa2df8b8a59f337e40172b31b25fa0a9071c5fd999 (48 inserted, 48 deleted) 
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db: [Parachain] Importing 6589306, base=6589050
2024-07-11 18:49:54.258 TRACE tokio-runtime-worker state-db: [Parachain] Pruning 0x65b5fa4c88420d53fb1f10bca9991db4905d7df2243aa3be9d19afe5c3878bf2 (46 deleted)
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker db: [Parachain] DB Commit done 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Discarding unpinned non-canon block: 0x145903798eedd4e8458615aa2df8b8a59f337e40172b31b25fa0a9071c5fd999
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.259 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.260 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.261 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.261 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.396  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing 64.6 bps, target=#6665383 (8 peers), best: #6593403 (0xa0d3…1fd7), finalized #60230 (0x826c…01fe), ⬇ 12 B/s ⬆ 12 B/s
2024-07-11 18:49:54.425 TRACE tokio-runtime-worker state-db::pin: [Parachain] Unpinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.454 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
2024-07-11 18:49:54.465 TRACE tokio-runtime-worker state-db::pin: [Parachain] Pinned non-canon block: 0xa0d34934e9b019822f132661b361323bdc6b69dc72cd39f81d7330e3fc061fd7
./launch.sh: line 3:  1045 Killed                  ./polkadot-sdk/target/release/polkadot-parachain --chain asset-hub-polkadot -ldb=trace,state-db=trace
root@assethub-phx4:~# root@assethub-phx4:~#

@letsbangout
Copy link
Author

Failed again with 25GB in the VM and a 10GB swap file activated.

The common thread seems to be that the counters on these chains do not increment (Relaychain/best, Parachain/finalized):

2024-07-11 19:56:10.852  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing 110.0 bps, target=#6665709 (9 peers), best: #6633212 (0x0468…3969), finalized #60230 (0x826c…01fe), ⬇ 5.9MiB/s ⬆ 0.4kiB/s
2024-07-11 19:56:11.075  INFO tokio-runtime-worker substrate: [Relaychain] ⚙️  Syncing  0.0 bps, target=#21604380 (10 peers), best: #9790976 (0xef4d…c7b7), finalized #9788416 (0x4ae7…0639), ⬇ 81.1kiB/s ⬆ 451.1kiB/s
2024-07-11 19:56:15.863  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing 102.6 bps, target=#6665709 (9 peers), best: #6633726 (0x2039…fa60), finalized #60230 (0x826c…01fe), ⬇ 1.4MiB/s ⬆ 1.0kiB/s
2024-07-11 19:56:16.084  INFO tokio-runtime-worker substrate: [Relaychain] ⚙️  Syncing  0.0 bps, target=#21604381 (10 peers), best: #9790976 (0xef4d…c7b7), finalized #9788416 (0x4ae7…0639), ⬇ 133.2kiB/s ⬆ 62.1kiB/s
2024-07-11 19:56:20.948  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing 58.0 bps, target=#6665710 (9 peers), best: #6634019 (0xaac4…9cc3), finalized #60230 (0x826c…01fe), ⬇ 0.4kiB/s ⬆ 38 B/s
2024-07-11 19:56:21.244  INFO tokio-runtime-worker substrate: [Relaychain] ⚙️  Syncing  0.0 bps, target=#21604382 (10 peers), best: #9790976 (0xef4d…c7b7), finalized #9788416 (0x4ae7…0639), ⬇ 58.8kiB/s ⬆ 26.5kiB/s
2024-07-11 19:56:25.964  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing 40.2 bps, target=#6665710 (9 peers), best: #6634221 (0x4bc5…7bf0), finalized #60230 (0x826c…01fe), ⬇ 25 B/s ⬆ 25 B/s
2024-07-11 19:56:26.259  INFO tokio-runtime-worker substrate: [Relaychain] ⚙️  Syncing  0.0 bps, target=#21604383 (10 peers), best: #9790976 (0xef4d…c7b7), finalized #9788416 (0x4ae7…0639), ⬇ 129.1kiB/s ⬆ 50.5kiB/s
2024-07-11 19:56:31.087  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing 44.5 bps, target=#6665711 (9 peers), best: #6634449 (0x3d88…3be7), finalized #60230 (0x826c…01fe), ⬇ 1.0kiB/s ⬆ 0.4kiB/s
2024-07-11 19:56:31.299  INFO tokio-runtime-worker substrate: [Relaychain] ⚙️  Syncing  0.0 bps, target=#21604384 (10 peers), best: #9790976 (0xef4d…c7b7), finalized #9788416 (0x4ae7…0639), ⬇ 82.1kiB/s ⬆ 43.1kiB/s
2024-07-11 19:56:36.123  INFO tokio-runtime-worker substrate: [Parachain] ⚙️  Syncing 52.3 bps, target=#6665711 (9 peers), best: #6634713 (0x4d74…7535), finalized #60230 (0x826c…01fe), ⬇ 0.9kiB/s ⬆ 0.7kiB/s
2024-07-11 19:56:36.321  INFO tokio-runtime-worker substrate: [Relaychain] ⚙️  Syncing  0.0 bps, target=#21604384 (10 peers), best: #9790976 (0xef4d…c7b7), finalized #9788416 (0x4ae7…0639), ⬇ 101.9kiB/s ⬆ 34.2kiB/s
[ 4966.291949] Out of memory: Killed process 1429 (polkadot-parach) total-vm:1282048780kB, anon-rss:23949388kB, file-rss:0kB, shmem-rss:152kB, UID:0 pgtables:88680kB oom_score_adj:0

LMK what I can do to provide more useful info @bkchr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known.
Projects
None yet
Development

No branches or pull requests

2 participants