Skip to content
This repository has been archived by the owner on Jul 5, 2023. It is now read-only.

Node failing to synchronize using snapshot #253

Open
Eric-Showers opened this issue Sep 18, 2020 · 12 comments
Open

Node failing to synchronize using snapshot #253

Eric-Showers opened this issue Sep 18, 2020 · 12 comments

Comments

@Eric-Showers
Copy link

Eric-Showers commented Sep 18, 2020 •
edited
Loading

Steps followed:

  1. Installed node on Ubuntu 18.04.3 using this article
  2. Downloaded most recent snapshot from this article and followed instructions
  3. Modified config.toml
  • state_sync_reactor = false
  • state_sync_height = -1
  1. Ran bnbchaind start, which produced this error:

panic: Failed to process committed block (112397623:6EA3AE907079229A36C4F8F12E00FB682BEF23A6349419722C76FA14F13EC329): Wrong Block.Header.AppHash. Expected F9F03780A35803BFE10F60A5814D48ED059016032CD04075D44BA3233B559D98, got ACE158FBC230B0897622D8D8913FC538F2B73A51B9C9627C4A66D5F081AD7C84

goroutine 58419 [running]: github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc00b0469c0) /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/blockchain/v0/reactor.go:404 +0x1c70 created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/blockchain/v0/reactor.go:127 +0x84

  1. Attempted running again, same error:

ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F9F03780A35803BFE10F60A5814D48ED059016032CD04075D44BA3233B559D98, got ACE158FBC230B0897622D8D8913FC538F2B73A51B9C9627C4A66D5F081AD7C84

Was it a mistake to turn off state sync? I need this node to have access to all historical blocks so I assumed that fast sync would be required.
@chainwhisper
Copy link
Contributor

Hello！

Which version of bnbchaiind did you use?

@Eric-Showers
Copy link
Author 

bnbchaind version
Binance Chain Release: v0.8.0;Binance Chain Commit: 422f0a96; Cosmos Release: v0.25.0-binance.23; Tendermint Release: v0.32.3-binance.3;

@chainwhisper
Copy link
Contributor

@Eric-Showers Have you verify if app.toml file is the latest?

@Eric-Showers
Copy link
Author

Yes sir - my app.toml file is identical to the one found here

@chainwhisper
Copy link
Contributor

@Eric-Showers, please verify the sha256sum: 194cbad7e1adcaafc16df94576f0a3b4a26245e4019cef93c51a3183bef71d73

@Eric-Showers
Copy link
Author

Okay I just completed a sha256 hash of the compressed snapshot and it matches that checksum

@alexqrid
Copy link

alexqrid commented Mar 1, 2021 •
edited
Loading

I confirm that the issue still presents. I downloaded Latest snapshot of February 1st and tried to sync from it on linux server. I used the latest version of the binary for linux - 0.8.2.

:~/Binance/node-binary/fullnode/prod/0.8.2/linux# uname -a
Linux server 5.11.0-051100-generic #202102142330 SMP Sun Feb 14 23:33:21 UTC 2021 x86_64 GNU/Linux

But the node crashed.

Logs 
I[2021-03-01|12:41:44.161] replayed all tx. Starting match              module=dex height=142231523
I[2021-03-01|12:41:44.161] No order comes in for the block              module=dexkeeper
I[2021-03-01|12:41:44.161] Relaying block for order book                module=dex height=142231524
I[2021-03-01|12:41:44.161] Canceled Order                               module=dex order="CancelOrderMsg{Sender:FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B, RefId: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B-960881}"
I[2021-03-01|12:41:44.161] Canceled Order                               module=dex order="CancelOrderMsg{Sender:DF6136BC4D4E9756437DBB59F45EC0468B2DDEB1, RefId: DF6136BC4D4E9756437DBB59F45EC0468B2DDEB1-550255}"
I[2021-03-01|12:41:44.161] Canceled Order                               module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211025}"
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2210936}"
I[2021-03-01|12:41:44.162] Added Order                                  module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211035, Symbol: FRM-DE7_BNB}"
I[2021-03-01|12:41:44.162] replayed all tx. Starting match              module=dex height=142231524
I[2021-03-01|12:41:44.162] Relaying block for order book                module=dex height=142231525
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:07285195B9434C05899780381C0E7372C7A03591, RefId: 07285195B9434C05899780381C0E7372C7A03591-340023}"
I[2021-03-01|12:41:44.162] Added Order                                  module=dex order="NewOrderMsg{Sender: 612F1135E2A7A514C4444C526297153E1007E16E, Id: 612F1135E2A7A514C4444C526297153E1007E16E-3662609, Symbol: NEXO-A84_BNB}"
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211018}"
I[2021-03-01|12:41:44.162] Added Order                                  module=dex order="NewOrderMsg{Sender: D089834ACD87C38C46D00E3BCE7160A16B69F002, Id: D089834ACD87C38C46D00E3BCE7160A16B69F002-346350, Symbol: XRP-BF2_BNB}"
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:7CD235E3C90C2D389E295CC6B4E4BDF445671D44, RefId: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44-653784}"
I[2021-03-01|12:41:44.162] Added Order                                  module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874960, Symbol: BULL-BE4_BNB}"
I[2021-03-01|12:41:44.162] replayed all tx. Starting match              module=dex height=142231525
I[2021-03-01|12:41:44.162] Relaying block for order book                module=dex height=142231526
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2210981}"
I[2021-03-01|12:41:44.162] replayed all tx. Starting match              module=dex height=142231526
I[2021-03-01|12:41:44.162] No order comes in for the block              module=dexkeeper
I[2021-03-01|12:41:44.162] Relaying block for order book                module=dex height=142231527
I[2021-03-01|12:41:44.162] Added Order                                  module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211038, Symbol: TWT-8C2_BNB}"
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:5CC9EEA33C86D854E0A3F8353111BC88C3FD0DAF, RefId: 5CC9EEA33C86D854E0A3F8353111BC88C3FD0DAF-73508}"
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:B8606C02542B4B41BFDDE81B9AF7776BF73486BB, RefId: B8606C02542B4B41BFDDE81B9AF7776BF73486BB-890893}"
I[2021-03-01|12:41:44.162] replayed all tx. Starting match              module=dex height=142231527
I[2021-03-01|12:41:44.162] Relaying block for order book                module=dex height=142231528
I[2021-03-01|12:41:44.162] replayed all tx. Starting match              module=dex height=142231528
I[2021-03-01|12:41:44.162] No order comes in for the block              module=dexkeeper
I[2021-03-01|12:41:44.162] Relaying block for order book                module=dex height=142231529
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:612F1135E2A7A514C4444C526297153E1007E16E, RefId: 612F1135
E2A7A514C4444C526297153E1007E16E-3662552}"
I[2021-03-01|12:41:44.162] Added Order                                  module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198D
E83FDDE4BC0BDF3F3A5621CF6A8-2211039, Symbol: SHR-DB6_BNB}"
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF5
9198DE83FDDE4BC0BDF3F3A5621CF6A8-2210982}"
I[2021-03-01|12:41:44.162] Canceled Order                               module=dex order="CancelOrderMsg{Sender:2C3ACBDA0A3DA42FCE4AC4D6750BC371C0BFC176, RefId: 2C3ACBDA
0A3DA42FCE4AC4D6750BC371C0BFC176-710093}"
I[2021-03-01|12:41:44.162] Added Order                                  module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3AD
C9CFF3EF31ADC7EDD288F5E208E-27874961, Symbol: MITH-C76_BNB}"  
I[2021-03-01|12:41:44.162] Added Order                                  module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198D
E83FDDE4BC0BDF3F3A5621CF6A8-2211041, Symbol: NOW-E68_BNB}"
I[2021-03-01|12:41:44.162] replayed all tx. Starting match              module=dex height=142231529
I[2021-03-01|12:41:44.163] Relaying block for order book                module=dex height=142231530
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198D
E83FDDE4BC0BDF3F3A5621CF6A8-2211042, Symbol: TWT-8C2_BNB}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: DF6136BC4D4E9756437DBB59F45EC0468B2DDEB1, Id: DF6136BC4D4E9
756437DBB59F45EC0468B2DDEB1-550263, Symbol: MITX-CAA_BNB}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:1EB0FCA540E10FE79867804B3BF5034BFC9D2A12, RefId: 1EB0FCA540E10FE79867804B3BF5034BFC9D2A12-2134187}"
I[2021-03-01|12:41:44.163] replayed all tx. Starting match              module=dex height=142231530
I[2021-03-01|12:41:44.163] Relaying block for order book                module=dex height=142231531
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874962, Symbol: BULL-BE4_BNB}"  
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2210987}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2211044, Symbol: CBIX-3C9_BNB}"   
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: B9E7F47CB15FA05A6900818F142B09213C265758, Id: B9E7F47CB15FA05A6900818F142B09213C265758-750644, Symbol: FRM-DE7_BNB}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 612F1135E2A7A514C4444C526297153E1007E16E, Id: 612F1135E2A7A514C4444C526297153E1007E16E-3662611, Symbol: CAN-677_BNB}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874963, Symbol: ETHBULL-D33_BNB}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A, RefId: 4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A-319059}"
I[2021-03-01|12:41:44.163] replayed all tx. Starting match              module=dex height=142231531
I[2021-03-01|12:41:44.163] Relaying block for order book                module=dex height=142231532
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:5AEBC595B8C8EDBFF616D9CBF291BE12E46C9AE8, RefId: 5AEBC595B8C8EDBFF616D9CBF291BE12E46C9AE8-620018}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A, Id: 4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A-319080, Symbol: CHZ-ECD_BNB}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B, Id: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B-960883, Symbol: NEXO-A84_BNB}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:AB3283B32936A6F2BBD7942F93716D96B1EE3A0E, RefId: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E-1573554}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:BBE4CDB5EFC7AF0E92EA467A03169370E6249BE8, RefId: BBE4CDB5EFC7AF0E92EA467A03169370E6249BE8-360780}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:7CD235E3C90C2D389E295CC6B4E4BDF445671D44, RefId: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44-653795}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E, Id: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E-1573556, Symbol: BTTB-D31_BNB}"   
I[2021-03-01|12:41:44.163] replayed all tx. Starting match              module=dex height=142231532
I[2021-03-01|12:41:44.163] Relaying block for order book                module=dex height=142231533
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:8452761BA0E6E214AF58CA493294263B81AE5A32, RefId: 8452761BA0E6E214AF58CA493294263B81AE5A32-6482184}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 8452761BA0E6E214AF58CA493294263B81AE5A32, Id: 8452761BA0E6E214AF58CA493294263B81AE5A32-6482206, Symbol: DOS-120_BNB}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, RefId: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874952}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44, Id: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44-653800, Symbol: XRP-BF2_BUSD-BD1}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: ABF1725319F1ACFC83382D2D084D75B97EBEC306, Id: ABF1725319F1ACFC83382D2D084D75B97EBEC306-625052, Symbol: XRP-BF2_BUSD-BD1}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, RefId: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874962}"
I[2021-03-01|12:41:44.164] Canceled Order                               module=dex order="CancelOrderMsg{Sender:0F674A63B7B3E956CF0A206A6195AF3090C0649E, RefId: 0F674A63B7B3E956CF0A206A6195AF3090C0649E-467060}"
I[2021-03-01|12:41:44.164] Added Order                                  module=dex order="NewOrderMsg{Sender: B8606C02542B4B41BFDDE81B9AF7776BF73486BB, Id: B8606C02542B4B41BFDDE81B9AF7776BF73486BB-890895, Symbol: CAS-167_BNB}"
I[2021-03-01|12:41:44.164] Canceled Order                               module=dex order="CancelOrderMsg{Sender:37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, RefId: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8-2210991}"
I[2021-03-01|12:41:44.164] replayed all tx. Starting match              module=dex height=142231533
I[2021-03-01|12:41:44.164] Relaying block for order book                module=dex height=142231534
I[2021-03-01|12:41:44.164] Canceled Order                               module=dex order="CancelOrderMsg{Sender:BC6E0EC1C60C6FBBC1AFB0A51F15013C9988E756, RefId: BC6E0EC1C60C6FBBC1AFB0A51F15013C9988E756-506353}"
I[2021-03-01|12:41:44.164] Added Order                                  module=dex order="NewOrderMsg{Sender: 37DD8CF59198DE83FDDE4BC0BDF3F3A5621CF6A8, Id: 37DD8CF59198D
E83FDDE4BC0BDF3F3A5621CF6A8-2211044, Symbol: CBIX-3C9_BNB}"   
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: B9E7F47CB15FA05A6900818F142B09213C265758, Id: B9E7F47CB15FA
05A6900818F142B09213C265758-750644, Symbol: FRM-DE7_BNB}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 612F1135E2A7A514C4444C526297153E1007E16E, Id: 612F1135E2A7A
514C4444C526297153E1007E16E-3662611, Symbol: CAN-677_BNB}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, Id: 1468EE412C3AD
C9CFF3EF31ADC7EDD288F5E208E-27874963, Symbol: ETHBULL-D33_BNB}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A, RefId: 4BD42EEC
CE4ADCF3EEC8FA67598F83DD6CACD36A-319059}"
I[2021-03-01|12:41:44.163] replayed all tx. Starting match              module=dex height=142231531
I[2021-03-01|12:41:44.163] Relaying block for order book                module=dex height=142231532
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:5AEBC595B8C8EDBFF616D9CBF291BE12E46C9AE8, RefId: 5AEBC595
B8C8EDBFF616D9CBF291BE12E46C9AE8-620018}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 4BD42EECCE4ADCF3EEC8FA67598F83DD6CACD36A, Id: 4BD42EECCE4AD
CF3EEC8FA67598F83DD6CACD36A-319080, Symbol: CHZ-ECD_BNB}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B, Id: FEFD69FB31E0350CB1F504C1D25C7FD21F0AA09B-960883, Symbol: NEXO-A84_BNB}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:AB3283B32936A6F2BBD7942F93716D96B1EE3A0E, RefId: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E-1573554}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:BBE4CDB5EFC7AF0E92EA467A03169370E6249BE8, RefId: BBE4CDB5EFC7AF0E92EA467A03169370E6249BE8-360780}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:7CD235E3C90C2D389E295CC6B4E4BDF445671D44, RefId: 7CD235E3C90C2D389E295CC6B4E4BDF445671D44-653795}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E, Id: AB3283B32936A6F2BBD7942F93716D96B1EE3A0E-1573556, Symbol: BTTB-D31_BNB}"   
I[2021-03-01|12:41:44.163] replayed all tx. Starting match              module=dex height=142231532
I[2021-03-01|12:41:44.163] Relaying block for order book                module=dex height=142231533
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:8452761BA0E6E214AF58CA493294263B81AE5A32, RefId: 8452761BA0E6E214AF58CA493294263B81AE5A32-6482184}"
I[2021-03-01|12:41:44.163] Added Order                                  module=dex order="NewOrderMsg{Sender: 8452761BA0E6E214AF58CA493294263B81AE5A32, Id: 8452761BA0E6E214AF58CA493294263B81AE5A32-6482206, Symbol: DOS-120_BNB}"
I[2021-03-01|12:41:44.163] Canceled Order                               module=dex order="CancelOrderMsg{Sender:1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E, RefId: 1468EE412C3ADC9CFF3EF31ADC7EDD288F5E208E-27874952}"
....
I[2021-03-01|12:41:44.179] Added Order                                  module=dex order="NewOrderMsg{Sender: 5CC9EEA33C86D854E0A3F8353111BC88C3FD0DAF, Id: 5CC9EEA33C86D
854E0A3F8353111BC88C3FD0DAF-73516, Symbol: DOS-120_BNB}"
I[2021-03-01|12:41:44.179] replayed all tx. Starting match              module=dex height=142231612
I[2021-03-01|12:41:46.255] Located day to load breathe block height     module=dexkeeper epochDay="\ufffdH\u0000\u0000\u0000\u0000\u0000\u0000"
I[2021-03-01|12:41:46.255] Loaded breathe block height                  module=dexkeeper height=142180985
I[2021-03-01|12:41:46.255] get last breathe block height                module=main height=142180985
I[2021-03-01|12:41:46.255] reload latest snapshot                       module=statesync height=142180985
I[2021-03-01|12:41:48.457] ABCI Handshake App Info                      module=consensus height=142231612 hash=6AC28FEA5CC0BDD1D7F84D6A705F2C2243A318C612B8DE0EF0E73F63B9
586F24 software-version= protocol-version=0
I[2021-03-01|12:41:48.457] ABCI Replay Blocks                           module=consensus appHeight=142231612 storeHeight=142231613 stateHeight=142231612
I[2021-03-01|12:41:48.457] Replay last block using real app             module=consensus
I[2021-03-01|12:41:48.463] symbols to match                             module=dexkeeper symbols="[BEAR-14C_BNB MATIC-84A_BNB]"
I[2021-03-01|12:41:48.472] Sync params proposals.                       module=paramHub
I[2021-03-01|12:41:48.498] start to collect publish information         module=pub height=142231613
I[2021-03-01|12:41:48.515] start to publish                             module=pub height=142231613 blockTime=1612418258924370991 numOfTrades=0 numOfOrders=4 numOfProposals=0 numOfSideProposals=0 numOfStakeUpdates=0 numOfAccounts=5
I[2021-03-01|12:41:48.515] Executed block                               module=consensus height=142231613 validTxs=4 invalidTxs=0
I[2021-03-01|12:41:48.540] Committed state                              module=consensus height=142231613 txs=4 appHash=775B775DF3348FA7242FF9735E07E64C5E53486375184F8E16EA4525B6CA4DDE
I[2021-03-01|12:41:48.541] Completed ABCI Handshake - Tendermint and App are synced module=consensus appHeight=142231612 appHash=6AC28FEA5CC0BDD1D7F84D6A705F2C2243A318C612B8DE0EF0E73F63B9586F24
I[2021-03-01|12:41:48.541] This node is not a validator                 module=consensus addr=FB4560C8F3982AA272BF10FEAE472024D2397C78 pubKey=PubKeyEd25519{A94507950684BB9352761BDB09148CA7232C699B8CB3ADD94F2C76967FA45452}
I[2021-03-01|12:41:48.563] Starting BlockchainReactor                   module=blockchain impl=BlockchainReactor
I[2021-03-01|12:41:48.563] Starting Consensus                           module=consensus impl=ConsensusReactor
I[2021-03-01|12:41:48.563] ConsensusReactor                             module=consensus fastSync=true
I[2021-03-01|12:41:48.563] Starting StateSyncReactor                    module=statesync impl=StateSyncReactor
I[2021-03-01|12:41:48.563] Starting StatePool                           module=statesync impl=StatePool
I[2021-03-01|12:41:48.563] broadcast state status request               module=statesync
E[2021-03-01|12:41:48.658] failed to load immutable tree                module=statesync err="version does not exist" store="KVStoreKey{0xc000218020, acc}"
I[2021-03-01|12:41:48.752] added a peer to state sync reactor           module=statesync peer=dbec1c96777dc757db67945cbc41e3e793b94261
E[2021-03-01|12:41:48.825] dialing failed (attempts: 1): auth failure: conn.ID (af89cf5aa3aff6d8b3f60043b8706dc458293b7f) dialed ID (aeed2359fde99ee784e0407e8e062be4b555a360) mismatch module=pex [email protected]:27146
E[2021-03-01|12:41:48.833] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:48.833] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:48.833] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:48.833] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:48.833] connection with [email protected]:27147 has been established or dialed module=pex [email protected]:27147
E[2021-03-01|12:41:48.833] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
I[2021-03-01|12:41:48.946] added a peer to state sync reactor           module=statesync peer=ca1245471ff1f2224e1076dfd94d7d2d66077cf5
I[2021-03-01|12:41:48.947] added a peer to state sync reactor           module=statesync peer=7d2a3c3727f46a6d04321656ecdf2f75d1d9c0dc
E[2021-03-01|12:41:48.953] dialing failed (attempts: 1): dial tcp 3.92.86.128:27146: connect: connection refused module=pex [email protected]:27146
E[2021-03-01|12:41:48.963] dialing failed (attempts: 1): dial tcp 35.208.230.254:27146: connect: connection refused module=pex [email protected]:27146
I[2021-03-01|12:41:49.049] added a peer to state sync reactor           module=statesync peer=48dada912ff10f04dec0db7900a78750384d06fd
I[2021-03-01|12:41:49.074] added a peer to state sync reactor           module=statesync peer=e2c323d27867bad2298a5452490157436e70e698
I[2021-03-01|12:41:49.081] added a peer to state sync reactor           module=statesync peer=cfb5c1f7c10dfd99219158f5491ce65227bfd072
E[2021-03-01|12:41:49.094] dialing failed (attempts: 1): auth failure: secret conn failed: EOF module=pex [email protected]:27146
I[2021-03-01|12:41:49.097] added a peer to state sync reactor           module=statesync peer=b323708b0bd429a95154ff0a287f0f42f432cb62
E[2021-03-01|12:41:49.130] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:49.130] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:49.130] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:49.130] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:49.130] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:49.130] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:49.130] connection with [email protected]:27146 has been established or dialed module=pex [email protected]:27146
E[2021-03-01|12:41:49.168] connection with [email protected]:27146 has been established or dialed module=pex addr=7d2a3c3727f46a6d043
[email protected]:27146
E[2021-03-01|12:41:49.168] Stopping peer for error                      module=p2p peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}" 
err=EOF
I[2021-03-01|12:41:49.170] added a peer to state sync reactor           module=statesync peer=0fd066de431d13e5613aba53b3ed25c5a7a0b67c
I[2021-03-01|12:41:49.174] added a peer to state sync reactor           module=statesync peer=782303c9060d46211225662fdd1dd411c638263a
I[2021-03-01|12:41:49.177] Stopping gossipDataRoutine for peer          module=consensus peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 
out}"
I[2021-03-01|12:41:49.177] Stopping gossipVotesRoutine for peer         module=consensus peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}"
I[2021-03-01|12:41:49.203] added a peer to state sync reactor           module=statesync peer=289f680c0c78465dbf1d86c2260dd032d4541a4c
I[2021-03-01|12:41:49.205] added a peer to state sync reactor           module=statesync peer=229ec3d0e55be99dee2194142d0ffc33e923a30d
I[2021-03-01|12:41:49.208] added a peer to state sync reactor           module=statesync peer=e2acf608686f830386e8648d6fc2f50c3316aa88
E[2021-03-01|12:41:49.235] dialing failed (attempts: 1): incompatible: Peer is on a different network. Got Binance-Chain-Ganges, expected Binance-Chain-Tigris module=pex [email protected]:27146
E[2021-03-01|12:41:49.249] dialing failed (attempts: 1): dial tcp 52.200.132.60:27146: connect: connection refused module=pex [email protected]:27146
E[2021-03-01|12:41:49.256] dialing failed (attempts: 1): auth failure: conn.ID (98579505572cc85488caa403bebb62e5b61d68e0) dialed ID (bb721fe089e83d42bcb8f629f8efe0e4af888fde) mismatch module=pex [email protected]:27146
E[2021-03-01|12:41:49.332] Stopping peer for error                      module=p2p peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}" err=EOF
I[2021-03-01|12:41:49.333] Stopping gossipVotesRoutine for peer         module=consensus peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}"
I[2021-03-01|12:41:49.333] Stopping gossipDataRoutine for peer          module=consensus peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}"
E[2021-03-01|12:41:49.414] dialing failed (attempts: 1): auth failure: secret conn failed: read tcp 142.93.45.202:51970->35.169.158.175:27146: read: connection reset by peer module=pex [email protected]:27146
I[2021-03-01|12:41:49.453] added a peer to state sync reactor           module=statesync peer=b7affdf084115fd7e7cdbc04686ef8c218e1801c
E[2021-03-01|12:41:49.491] dialing failed (attempts: 1): incompatible: Peer is on a different network. Got Binance-Chain-Ganges, expected Binance-Chain-Tigris module=pex [email protected]:27146
E[2021-03-01|12:41:49.513] dialing failed (attempts: 1): incompatible: Peer is on a different network. Got Binance-Chain-Ganges, expected Binance-Chain-Tigris module=pex [email protected]:27146
I[2021-03-01|12:41:49.548] added a peer to state sync reactor           module=statesync peer=5c96d4f0174dda7757bcda11a57ecc91851a90f0
E[2021-03-01|12:41:49.563] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex [email protected]:27147
E[2021-03-01|12:41:49.563] dialing failed (attempts: 1): dial tcp 35.159.10.214:27146: i/o timeout module=pex [email protected]:27146
I[2021-03-01|12:41:49.817] added a peer to state sync reactor           module=statesync peer=98579505572cc85488caa403bebb62e5b61d68e0
E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex [email protected]:27147
E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex [email protected]:27147
E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 35.187.203.129:27147: i/o timeout module=pex [email protected]:27147
E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 66.228.60.219:26656: i/o timeout module=pex [email protected]:26656
E[2021-03-01|12:41:49.833] dialing failed (attempts: 1): dial tcp 99.81.122.88:27146: i/o timeout module=pex [email protected]:27146
E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex [email protected]:27147
E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 18.139.29.55:27146: i/o timeout module=pex [email protected]:27146
E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 95.211.89.68:27146: i/o timeout module=pex [email protected]:27146
E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 82.65.68.233:27147: i/o timeout module=pex [email protected]:27147
E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex [email protected]:27147
I[2021-03-01|12:41:51.097] Stopping queryMaj23Routine for peer          module=consensus peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}"
I[2021-03-01|12:41:51.203] Stopping queryMaj23Routine for peer          module=consensus peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}"
E[2021-03-01|12:41:51.953] dialing failed (attempts: 1): auth failure: secret conn failed: read tcp 142.93.45.202:36030->18.208.56.183:27146: i/o timeout module=pex [email protected]:27146
I[2021-03-01|12:41:58.563] decided not init pool                        module=statesync manifests=0 tried=1
I[2021-03-01|12:41:58.563] broadcast state status request               module=statesync
I[2021-03-01|12:42:08.563] got enough peers to start state sync         module=statesync manifestHash="[100 117 109 109 121 104 97 115 104 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]" numOfPeers=13
I[2021-03-01|12:42:08.563] all peers we connected have no manifest, start fast sync directly module=statesync
panic: runtime error: invalid memory address or nil pointer dereference
E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 82.65.68.233:27147: i/o timeout module=pex [email protected]:27147
E[2021-03-01|12:41:50.130] dialing failed (attempts: 1): dial tcp 34.92.4.164:27147: i/o timeout module=pex [email protected]:27147
I[2021-03-01|12:41:51.097] Stopping queryMaj23Routine for peer          module=consensus peer="Peer{MConn{34.247.169.233:27146} b323708b0bd429a95154ff0a287f0f42f432cb62 out}"
I[2021-03-01|12:41:51.203] Stopping queryMaj23Routine for peer          module=consensus peer="Peer{MConn{34.227.165.88:27146} 289f680c0c78465dbf1d86c2260dd032d4541a4c out}"
E[2021-03-01|12:41:51.953] dialing failed (attempts: 1): auth failure: secret conn failed: read tcp 142.93.45.202:36030->18.208.56.183:27146: i/o timeout module=pex [email protected]:27146
I[2021-03-01|12:41:58.563] decided not init pool                        module=statesync manifests=0 tried=1
I[2021-03-01|12:41:58.563] broadcast state status request               module=statesync
I[2021-03-01|12:42:08.563] got enough peers to start state sync         module=statesync manifestHash="[100 117 109 109 121 104 97 115 104 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]" numOfPeers=13
I[2021-03-01|12:42:08.563] all peers we connected have no manifest, start fast sync directly module=statesync
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xd42ef2]

goroutine 155414 [running]:
github.com/tendermint/tendermint/snapshot.(*StateReactor).startFastSync(0xc03743cd00)
        /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/snapshot/reactor.go:237 +0x222
github.com/tendermint/tendermint/snapshot.(*StateReactor).poolRoutine(0xc03743cd00)
        /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/snapshot/reactor.go:343 +0x26e
created by github.com/tendermint/tendermint/snapshot.(*StateReactor).OnStart
        /root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/snapshot/reactor.go:126 +0x6d
Can someone help me @huangsuyu ? I want to run an archive node but don't want to wait until the sync from scratch will finish, just need to sync from a snapshot.

@alexqrid
Copy link

alexqrid commented Mar 6, 2021

Any updates on this? @huangsuyu
Why does binance actually make the snapshots if they are useless as far as node can sync from them?

@chainwhisper
Copy link
Contributor

@alexqrid Checking! i will come back to you asap

@chainwhisper
Copy link
Contributor

Okay I just completed a sha256 hash of the compressed snapshot and it matches that checksum

can you check if you set open file limit to 65535?

@alexqrid
Copy link

alexqrid commented Mar 9, 2021

can you check if you set open file limit to 65535?

Yes I did. Open file limit was set to 65535. The problem is with wrong block hash as you can see from the logs.

Logs 
I[2021-03-09|15:30:45.586] start to collect publish information         module=pub height=142231940
I[2021-03-09|15:30:45.601] start to publish                             module=pub height=142231940 blockTime=1612418401567922666 numOfTrades=0 numOfOrders=7 numOfProposals=0 numOfSideProposals=0 numOfStakeUpdates=0 numOfAccounts=9
I[2021-03-09|15:30:45.601] Executed block                               module=state height=142231940 validTxs=8 invalidTxs=0
I[2021-03-09|15:30:45.613] Committed state                              module=state height=142231940 txs=8 appHash=8728829543CA547C713CEEEB1CAEA89447F7B11E1104460212099EA636B1C880
I[2021-03-09|15:30:45.620] failed to process NewOrderMsg                module=main oid=DB5E68DFAD2F8CFBBFACC9C29266B95617A49FBC-375131
I[2021-03-09|15:30:45.620] symbols to match                             module=dexkeeper symbols=[RAVEN-F66_BNB]
I[2021-03-09|15:30:45.621] Sync params proposals.                       module=paramHub 
I[2021-03-09|15:30:45.628] start to collect publish information         module=pub height=142231941
E[2021-03-09|15:30:45.642] failed to locate order change in OrderChangesMap module=pub orderChange="id: DB5E68DFAD2F8CFBBFACC9C29266B95617A49FBC-375131, tpe: FailedBlocking"
I[2021-03-09|15:30:45.646] start to publish                             module=pub height=142231941 blockTime=1612418401925314661 numOfTrades=0 numOfOrders=3 numOfProposals=0 numOfSideProposals=0 numOfStakeUpdates=0 numOfAccounts=3
I[2021-03-09|15:30:45.646] Executed block                               module=state height=142231941 validTxs=2 invalidTxs=1
I[2021-03-09|15:30:45.657] Committed state                              module=state height=142231941 txs=3 appHash=45E848914F26AE91F0F228C5225216B4879EE7FF976A26ECB49FF04B3DB52F02
panic: Failed to process committed block (142231942:9C1D7035617ACB4C3437C060F420419C283C2C5C49C5AA47083900FB52F851ED): Wrong Block.Header.AppHash.  Expected 45E848914F26AE91F0F228C5225216B4879EE7FF976A26ECB49FF04B3DB52F02, got A3C8008FC6857F897CABA74860A7C8A1FFA7474739E8BA85E2721A3CC017E436

goroutine 155332 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000b10b60)
	/root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/blockchain/v0/reactor.go:404 +0x194b
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
	/root/bnbchain/pkg/mod/github.com/binance-chain/[email protected]/blockchain/v0/reactor.go:127 +0x84

@kenorb
Copy link

kenorb commented Nov 13, 2021

See: cosmos/cosmos-sdk#10281

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kenorb @Eric-Showers @alexqrid @chainwhisper