Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TrieDB iteration error: Database missing expected key #663

Closed
tugytur opened this issue Apr 7, 2023 · 12 comments
Closed

TrieDB iteration error: Database missing expected key #663

tugytur opened this issue Apr 7, 2023 · 12 comments

Comments

@tugytur
Copy link
Contributor

tugytur commented Apr 7, 2023

Two our our public archive rpc nodes crashed at the same time with the same error.
Version: polkadot 0.9.41-e203bfb396e

Restarting didn't solve the problem. But downgrading to polkadot 0.9.39-1-298b4aefe1b solved the problem for now

Parameters:

ExecStart=/usr/local/bin/polkadot \
        --name="rpcdot-001" \
        --state-pruning=archive \
        --no-mdns \
        --public-addr=/ip4/X.X.X.X/tcp/30333 \
        --listen-addr=/ip4/X.X.X.X/tcp/30333 \
        --chain=polkadot \
        --database=paritydb \
        --prometheus-port=9615 \
        --rpc-port=9933 \
        --ws-port=9944 \
        --ws-external \
        --no-telemetry \
        --no-hardware-benchmarks \
        --no-private-ip \
        --in-peers=10 \
        --out-peers=10 \
        --max-parallel-downloads=3 \
        --rpc-methods=safe \
        --rpc-cors=all \
        --execution=wasm \
        --wasm-execution=compiled \
        --runtime-cache-size=64 \
        --max-runtime-instances=128 \
        --base-path=/var/blockchain

Logs:

polkadot[2151]: 2023-04-07 16:36:37 ✨ Imported #14987243 (0xfdbd…a251)
polkadot[2151]: 2023-04-07 16:36:38 💤 Idle (20 peers), best: #14987243 (0xfdbd…a251), finalized #14987240 (0xebe4…a8f8), ⬇ 430.4kiB/s ⬆ 559.4kiB/s
polkadot[2151]: 2023-04-07 16:36:38 Accepting new connection 1/100
polkadot[2151]: ====================
polkadot[2151]: Version: 0.9.41-e203bfb396e
polkadot[2151]:    0: sp_panic_handler::set::{{closure}}
polkadot[2151]:    1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
polkadot[2151]:       std::panicking::rust_panic_with_hook
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
polkadot[2151]:    2: std::panicking::begin_panic_handler::{{closure}}
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:579:13
polkadot[2151]:    3: std::sys_common::backtrace::__rust_end_short_backtrace
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
polkadot[2151]:    4: rust_begin_unwind
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
polkadot[2151]:    5: core::panicking::panic_fmt
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
polkadot[2151]:    6: core::result::unwrap_failed
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/result.rs:1790:5
polkadot[2151]:    7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::next_storage_key
polkadot[2151]:    8: sp_io::storage::next_key_version_1
polkadot[2151]:    9: sp_io::storage::ExtStorageNextKeyVersion1::call
polkadot[2151]:   10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
polkadot[2151]:   11: <unknown>
polkadot[2151]:   12: <unknown>
polkadot[2151]:   13: <unknown>
polkadot[2151]:   14: <unknown>
polkadot[2151]:   15: <unknown>
polkadot[2151]:   16: <unknown>
polkadot[2151]:   17: <unknown>
polkadot[2151]:   18: <unknown>
polkadot[2151]:   19: <unknown>
polkadot[2151]:   20: <unknown>
polkadot[2151]:   21: <unknown>
polkadot[2151]:   22: <unknown>
polkadot[2151]:   23: wasmtime_runtime::traphandlers::catch_traps::call_closure
polkadot[2151]:   24: wasmtime_setjmp
polkadot[2151]:   25: sc_executor_wasmtime::runtime::perform_call
polkadot[2151]:   26: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats
polkadot[2151]:   27: sc_executor_common::wasm_runtime::WasmInstance::call_export
polkadot[2151]:   28: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
polkadot[2151]:   29: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
polkadot[2151]:   30: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
polkadot[2151]:   31: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
polkadot[2151]:   32: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
polkadot[2151]:   33: <polkadot_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_api::Core<__SrApiBlock__>>::__runtime_api_internal_call_api_at
polkadot[2151]:   34: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
polkadot[2151]:   35: <sc_consensus_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
polkadot[2151]:   36: <sc_consensus_beefy::import::BeefyBlockImport<Block,BE,Runtime,I> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
polkadot[2151]:   37: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
polkadot[2151]:   38: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+Transaction = Transaction+Error = sp_consensus::error::Error+core::marker::Sync+core::marker::Send> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
polkadot[2151]:   39: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
polkadot[2151]:   40: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
polkadot[2151]:   41: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
polkadot[2151]:   42: tokio::runtime::task::raw::poll
polkadot[2151]:   43: std::sys_common::backtrace::__rust_begin_short_backtrace
polkadot[2151]:   44: core::ops::function::FnOnce::call_once{{vtable.shim}}
polkadot[2151]:   45: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
polkadot[2151]:       <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
polkadot[2151]:       std::sys::unix::thread::Thread::new::thread_start
polkadot[2151]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys/unix/thread.rs:108:17
polkadot[2151]:   46: start_thread
polkadot[2151]:   47: clone
polkadot[2151]: Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "TrieDB iteration error: Database missing expected key: 0xb320f22e6379820fc8e26f5d4523260306a01324663582af921b72b34d1ddbcd"', /usr/local/cargo/git/checkouts/substrate-7e08433d4c370a21/980eb16/primitives/state-machine/src/ext.rs:339
polkadot[2151]: This is a bug. Please report it at:
polkadot[2151]:         https://github.com/paritytech/polkadot/issues/new
polkadot[2151]: 2023-04-07 16:36:42 1 storage transactions are left open by the runtime. Those will be rolled back.
polkadot[2151]: 2023-04-07 16:36:42 1 storage transactions are left open by the runtime. Those will be rolled back.
polkadot[2151]: 2023-04-07 16:36:42 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: host code panicked while being called by the runtime: Externalities not allowed to fail within runtime: "TrieDB iteration error: Database missing expected key: 0xb320f22e6379820fc8e26f5d4523260306a01324663582af921b72b34d1ddbcd"
polkadot[2151]: WASM backtrace:
polkadot[2151]: error while executing at wasm backtrace:
polkadot[2151]:     0: 0x4a3e00 - <unknown>!sp_io::storage::extern_host_function_impls::next_key::h9c69a85986ccdf46
polkadot[2151]:     1: 0x1efc51 - <unknown>!<frame_support::storage::PrefixIterator<T,OnRemoval> as core::iter::traits::iterator::Iterator>::next::h8e6f048c6427d4d6
polkadot[2151]:     2: 0x1579d4 - <unknown>!polkadot_runtime_parachains::inclusion::<impl polkadot_runtime_parachains::inclusion::pallet::Pallet<T>>::collect_pending::h06f9834338807efc
polkadot[2151]:     3: 0x23b2fd - <unknown>!polkadot_runtime_parachains::paras_inherent::pallet::collect_all_freed_cores::hf015393c5b3dd97f
polkadot[2151]:     4: 0x1f7e71 - <unknown>!polkadot_runtime_parachains::paras_inherent::<impl polkadot_runtime_parachains::paras_inherent::pallet::Pallet<T>>::enter_inner::hcf22a357a6807de2
polkadot[2151]:     5: 0x375a8c - <unknown>!frame_support::storage::transactional::with_storage_layer::he8cedd9fef548c8a
polkadot[2151]:     6: 0x674ee - <unknown>!<polkadot_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::ha67e343f8aabe03c
polkadot[2151]:     7: 0x664e5 - <unknown>!<polkadot_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h7347e163ec0a217e
polkadot[2151]:     8: 0x2a602a - <unknown>!<sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply::ha9febc5cffd72e0c
polkadot[2151]:     9: 0x133729 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::ha64cd9785b93ffe3
polkadot[2151]:    10: 0x13294d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h387cdac41e037898
polkadot[2151]:    11: 0x24e7d4 - <unknown>!Core_execute_block
polkadot[2151]: 2023-04-07 16:36:42 💔 Error importing block 0x548d5c20fbc5bfe0178400c1dd44c9b2ef3509cdc703e786fa7b1f4a2606c1c9: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: host code panicked while being called by the runtime: Externalities not allowed to fail within runtime: "TrieDB iteration error: Database missing expected key: 0xb320f22e6379820fc8e26f5d4523260306a01324663582af921b72b34d1ddbcd"
polkadot[2151]: WASM backtrace:
polkadot[2151]: error while executing at wasm backtrace:
polkadot[2151]:     0: 0x4a3e00 - <unknown>!sp_io::storage::extern_host_function_impls::next_key::h9c69a85986ccdf46
polkadot[2151]:     1: 0x1efc51 - <unknown>!<frame_support::storage::PrefixIterator<T,OnRemoval> as core::iter::traits::iterator::Iterator>::next::h8e6f048c6427d4d6
polkadot[2151]:     2: 0x1579d4 - <unknown>!polkadot_runtime_parachains::inclusion::<impl polkadot_runtime_parachains::inclusion::pallet::Pallet<T>>::collect_pending::h06f9834338807efc
polkadot[2151]:     3: 0x23b2fd - <unknown>!polkadot_runtime_parachains::paras_inherent::pallet::collect_all_freed_cores::hf015393c5b3dd97f
polkadot[2151]:     4: 0x1f7e71 - <unknown>!polkadot_runtime_parachains::paras_inherent::<impl polkadot_runtime_parachains::paras_inherent::pallet::Pallet<T>>::enter_inner::hcf22a357a6807de2
polkadot[2151]:     5: 0x375a8c - <unknown>!frame_support::storage::transactional::with_storage_layer::he8cedd9fef548c8a
polkadot[2151]:     6: 0x674ee - <unknown>!<polkadot_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::ha67e343f8aabe03c
polkadot[2151]:     7: 0x664e5 - <unknown>!<polkadot_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h7347e163ec0a217e
polkadot[2151]:     8: 0x2a602a - <unknown>!<sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply::ha9febc5cffd72e0c
polkadot[2151]:     9: 0x133729 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::ha64cd9785b93ffe3
polkadot[2151]:    10: 0x13294d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h387cdac41e037898
polkadot[2151]:    11: 0x24e7d4 - <unknown>!Core_execute_block
polkadot[2151]: ====================
@PierreBesson
Copy link

We are encountering the exact same error log lines on at least one of our Polkadot ParityDB archive nodes:

      polkadot \
        --base-path=/data/ \
        --chain=polkadot \
        --database=paritydb \
        --pruning=archive \
        --prometheus-external \
        --prometheus-port 9615 \
        --rpc-external --ws-external --rpc-methods=safe --rpc-cors=all

@bkchr
Copy link
Member

bkchr commented Apr 10, 2023

This is maybe related to not using the latest paritydb release.

@bkchr
Copy link
Member

bkchr commented Apr 10, 2023

Which should be fixed by 0.9.42

@bkchr bkchr mentioned this issue Apr 11, 2023
Closed
@stakeworld
Copy link
Contributor

@bkchr : I had the same problem on a warp syncing 0.9.41 polkadot node. Restarting does not fix it.

startup: --chain polkadot --name sw-val-dot-02 --validator --port 30311 --rpc-port 9911 --ws-port 9811 --prometheus-port 9611 --prometheus-external --database paritydb

Apr 13 11:07:51 worker02 polkadot[1204]: 2023-04-13 11:07:51 ⚙️  Syncing 110.0 bps, target=#15068261 (40 peers), best: #4078610 (0xe9b1…0b00), finalized #4078592 (0xaa9f…a24e), ⬇ 440.1kiB/s ⬆ 337.0kiB/s
Apr 13 11:07:56 worker02 polkadot[1204]: 2023-04-13 11:07:56 ⚙️  Syncing 98.0 bps, target=#15068261 (40 peers), best: #4079100 (0x2abe…b477), finalized #4078980 (0x36e3…16de), ⬇ 458.8kiB/s ⬆ 363.4kiB/s
Apr 13 11:07:58 worker02 polkadot[1204]: ====================
Apr 13 11:07:58 worker02 polkadot[1204]: Version: 0.9.41-e203bfb396e
Apr 13 11:07:58 worker02 polkadot[1204]:    0: sp_panic_handler::set::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:    1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
Apr 13 11:07:58 worker02 polkadot[1204]:       std::panicking::rust_panic_with_hook
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
Apr 13 11:07:58 worker02 polkadot[1204]:    2: std::panicking::begin_panic_handler::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:579:13
Apr 13 11:07:58 worker02 polkadot[1204]:    3: std::sys_common::backtrace::__rust_end_short_backtrace
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
Apr 13 11:07:58 worker02 polkadot[1204]:    4: rust_begin_unwind
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
Apr 13 11:07:58 worker02 polkadot[1204]:    5: core::panicking::panic_fmt
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
Apr 13 11:07:58 worker02 polkadot[1204]:    6: core::result::unwrap_failed
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/result.rs:1790:5
Apr 13 11:07:58 worker02 polkadot[1204]:    7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
Apr 13 11:07:58 worker02 polkadot[1204]:    8: sp_io::storage::get_version_1
Apr 13 11:07:58 worker02 polkadot[1204]:    9: sp_io::storage::ExtStorageGetVersion1::call
Apr 13 11:07:58 worker02 polkadot[1204]:   10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
Apr 13 11:07:58 worker02 polkadot[1204]:   11: <unknown>
Apr 13 11:07:58 worker02 polkadot[1204]:   12: <unknown>
Apr 13 11:07:58 worker02 polkadot[1204]:   13: <unknown>
Apr 13 11:07:58 worker02 polkadot[1204]:   14: <unknown>
Apr 13 11:07:58 worker02 polkadot[1204]:   15: <unknown>
Apr 13 11:07:58 worker02 polkadot[1204]:   16: <unknown>
Apr 13 11:07:58 worker02 polkadot[1204]:   17: wasmtime_runtime::traphandlers::catch_traps::call_closure
Apr 13 11:07:58 worker02 polkadot[1204]:   18: wasmtime_setjmp
Apr 13 11:07:58 worker02 polkadot[1204]:   19: sc_executor_wasmtime::runtime::perform_call
Apr 13 11:07:58 worker02 polkadot[1204]:   20: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats
Apr 13 11:07:58 worker02 polkadot[1204]:   21: sc_executor_common::wasm_runtime::WasmInstance::call_export
Apr 13 11:07:58 worker02 polkadot[1204]:   22: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:   23: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
Apr 13 11:07:58 worker02 polkadot[1204]:   24: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
Apr 13 11:07:58 worker02 polkadot[1204]:   25: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
Apr 13 11:07:58 worker02 polkadot[1204]:   26: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
Apr 13 11:07:58 worker02 polkadot[1204]:   27: <polkadot_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_api::Core<__SrApiBlock__>>::__runtime_api_internal_call_api_at
Apr 13 11:07:58 worker02 polkadot[1204]:   28: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:   29: <sc_consensus_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:   30: <sc_consensus_beefy::import::BeefyBlockImport<Block,BE,Runtime,I> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:   31: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:   32: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+Transaction = Transaction+Error = sp_consensus::error::Error+core::marker::Sync+core::marker::Send> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:   33: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
Apr 13 11:07:58 worker02 polkadot[1204]:   34: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
Apr 13 11:07:58 worker02 polkadot[1204]:   35: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
Apr 13 11:07:58 worker02 polkadot[1204]:   36: tokio::runtime::task::raw::poll
Apr 13 11:07:58 worker02 polkadot[1204]:   37: std::sys_common::backtrace::__rust_begin_short_backtrace
Apr 13 11:07:58 worker02 polkadot[1204]:   38: core::ops::function::FnOnce::call_once{{vtable.shim}}
Apr 13 11:07:58 worker02 polkadot[1204]:   39: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
Apr 13 11:07:58 worker02 polkadot[1204]:       <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
Apr 13 11:07:58 worker02 polkadot[1204]:       std::sys::unix::thread::Thread::new::thread_start
Apr 13 11:07:58 worker02 polkadot[1204]:              at rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys/unix/thread.rs:108:17
Apr 13 11:07:58 worker02 polkadot[1204]:   40: <unknown>
Apr 13 11:07:58 worker02 polkadot[1204]:   41: <unknown>
Apr 13 11:07:58 worker02 polkadot[1204]: Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x958be42ed37be4a8406d9d5a0560f0d73c80e401fb298e530ce6ac8162f4a889"', /usr/local/cargo/git/checkouts/substrate-7e08433d4c370a21/980eb16/primitives/state-machine/src/ext.rs:192
Apr 13 11:07:58 worker02 polkadot[1204]: This is a bug. Please report it at:
Apr 13 11:07:58 worker02 polkadot[1204]:         https://github.com/paritytech/polkadot/issues/new
Apr 13 11:08:04 worker02 systemd[1]: sw-val-dot-01.service: Main process exited, code=exited, status=1/FAILURE
Apr 13 11:08:04 worker02 systemd[1]: sw-val-dot-01.service: Failed with result 'exit-code'.

@bkchr
Copy link
Member

bkchr commented Apr 13, 2023

@stakeworld would it be possible for you to test latest master?

@stakeworld
Copy link
Contributor

@bkchr Sure, is there a binary downloadable somewere or should I compile?

@bkchr
Copy link
Member

bkchr commented Apr 13, 2023

You would need to compile it yourself with cargo build --profile production. TY!

@PierreBesson
Copy link

It should be downloadable from the Gitlab CI artifacts for a few days: https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/2679262/artifacts/raw/artifacts/polkadot

@stakeworld
Copy link
Contributor

stakeworld commented Apr 13, 2023 •
edited
Loading

@bkchr a master build fixes the problem and the sync resumes normally
@PierreBesson that binary throws a ssl lib error on ubuntu 22.04

@bkchr
Copy link
Member

bkchr commented Apr 14, 2023

@bkchr a master build fixes the problem and the sync resumes normally

Ty for reporting back! And with 0.9.41 it was constantly failing?

@stakeworld
Copy link
Contributor

stakeworld commented Apr 14, 2023 •
edited
Loading

Ty for reporting back! And with 0.9.41 it was constantly failing?

Yes, on 0.9.41 it kept crashing also after a restart, with the master build no more errors, restarted normally and continued the sync

@bkchr
Copy link
Member

bkchr commented Apr 14, 2023

Okay perfect!

@Sophia-Gold Sophia-Gold transferred this issue from paritytech/polkadot Aug 24, 2023
@bkchr bkchr closed this as completed Aug 25, 2023
claravanstaden pushed a commit to Snowfork/polkadot-sdk that referenced this issue Dec 8, 2023
@alistair-singh
Remove renounce ownership (paritytech#663)
1210b36 
* Remove renounce ownership

* script to poll local ethereum for contract events

* added comments
bkchr pushed a commit that referenced this issue Apr 10, 2024
@svyatonik @bkchr
decode call after spec_version check (#663)
acee558
@Anastasiia-Khab Anastasiia-Khab mentioned this issue Nov 21, 2024
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@PierreBesson @bkchr @tugytur @stakeworld