Skip to content
This repository has been archived by the owner on May 13, 2022. It is now read-only.

Panic in SLOAD when reloading the chain after it died or during normal chain operation #626

Closed
2 tasks
ratranqu opened this issue Jul 18, 2017 · 13 comments
Closed
2 tasks

Comments

@ratranqu
Copy link
Contributor

Please include in your bug report:

  • burrow version (docker image tag or branch if built from source)
    develop branch

Panic on SLOAD when rebuilding the chain or during normal operation of the chain (it just happens).

  • steps to reproduction
    burrow serve --work-dir /home/ubuntu/decpub/single -d
    If issue is a feature request, tell us why this feature is useful.
    Last op and stack trace below:
    --
(4) (00000000) 0000000000000000000000007B02183F9397EB10AF226693385C1C900409229D (code=18053) gas: 999996903 (d) 3A067CA6
(pc) 0   (op) PUSH1          (st) 0     => 0x0000000000000000000000000000000000000000000000000000000000000060
(pc) 2   (op) PUSH1          (st) 1     => 0x0000000000000000000000000000000000000000000000000000000000000040
(pc) 4   (op) MSTORE         (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000060 @ 0x40
(pc) 5   (op) CALLDATASIZE   (st) 0     => 4
(pc) 6   (op) ISZERO         (st) 1      == 0 = 0
(pc) 7   (op) PUSH2          (st) 1     => 0x00000000000000000000000000000000000000000000000000000000000000ED
(pc) 10  (op) JUMPI          (st) 2     ~> false
(pc) 11  (op) PUSH1          (st) 0     => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 13  (op) CALLDATALOAD   (st) 1     => 0x3A067CA600000000000000000000000000000000000000000000000000000000
(pc) 14  (op) PUSH29         (st) 1     => 0x0000000100000000000000000000000000000000000000000000000000000000
(pc) 44  (op) SWAP1          (st) 2     => [2] 3A067CA600000000000000000000000000000000000000000000000000000000
(pc) 45  (op) DIV            (st) 2     26245606592116269608546148900486738430516241565468967273606210530203626635264 / 26959946667150639794667015087019630673637144422540572481103610249216 = 973503654 (000000000000000000000000000000000000000000000000000000003A067CA6)
(pc) 46  (op) DUP1           (st) 1     => [1] 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 47  (op) PUSH4          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000012065FE0
(pc) 52  (op) EQ             (st) 3     0000000000000000000000000000000000000000000000000000000012065FE0 == 000000000000000000000000000000000000000000000000000000003A067CA6 = 0
(pc) 53  (op) PUSH2          (st) 2     => 0x00000000000000000000000000000000000000000000000000000000000000FF
(pc) 56  (op) JUMPI          (st) 3     ~> false
(pc) 57  (op) DUP1           (st) 1     => [1] 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 58  (op) PUSH4          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000014EC51FB
(pc) 63  (op) EQ             (st) 3     0000000000000000000000000000000000000000000000000000000014EC51FB == 000000000000000000000000000000000000000000000000000000003A067CA6 = 0
(pc) 64  (op) PUSH2          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000127
(pc) 67  (op) JUMPI          (st) 3     ~> false
(pc) 68  (op) DUP1           (st) 1     => [1] 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 69  (op) PUSH4          (st) 2     => 0x000000000000000000000000000000000000000000000000000000001965BC3B
(pc) 74  (op) EQ             (st) 3     000000000000000000000000000000000000000000000000000000001965BC3B == 000000000000000000000000000000000000000000000000000000003A067CA6 = 0
(pc) 75  (op) PUSH2          (st) 2     => 0x000000000000000000000000000000000000000000000000000000000000014F
(pc) 78  (op) JUMPI          (st) 3     ~> false
(pc) 79  (op) DUP1           (st) 1     => [1] 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 80  (op) PUSH4          (st) 2     => 0x000000000000000000000000000000000000000000000000000000001C6957BD
(pc) 85  (op) EQ             (st) 3     000000000000000000000000000000000000000000000000000000001C6957BD == 000000000000000000000000000000000000000000000000000000003A067CA6 = 0
(pc) 86  (op) PUSH2          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000182
(pc) 89  (op) JUMPI          (st) 3     ~> false
(pc) 90  (op) DUP1           (st) 1     => [1] 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 91  (op) PUSH4          (st) 2     => 0x00000000000000000000000000000000000000000000000000000000204357BA
(pc) 96  (op) EQ             (st) 3     00000000000000000000000000000000000000000000000000000000204357BA == 000000000000000000000000000000000000000000000000000000003A067CA6 = 0
(pc) 97  (op) PUSH2          (st) 2     => 0x00000000000000000000000000000000000000000000000000000000000001C9
(pc) 100 (op) JUMPI          (st) 3     ~> false
(pc) 101 (op) DUP1           (st) 1     => [1] 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 102 (op) PUSH4          (st) 2     => 0x00000000000000000000000000000000000000000000000000000000216E7275
(pc) 107 (op) EQ             (st) 3     00000000000000000000000000000000000000000000000000000000216E7275 == 000000000000000000000000000000000000000000000000000000003A067CA6 = 0
(pc) 108 (op) PUSH2          (st) 2     => 0x00000000000000000000000000000000000000000000000000000000000001FC
(pc) 111 (op) JUMPI          (st) 3     ~> false
(pc) 112 (op) DUP1           (st) 1     => [1] 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 113 (op) PUSH4          (st) 2     => 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 118 (op) EQ             (st) 3     000000000000000000000000000000000000000000000000000000003A067CA6 == 000000000000000000000000000000000000000000000000000000003A067CA6 = 1
(pc) 119 (op) PUSH2          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000224
(pc) 122 (op) JUMPI          (st) 3     ~> 548
(pc) 548 (op) JUMPDEST       (st) 1    
(pc) 549 (op) CALLVALUE      (st) 1     => 0
(pc) 550 (op) PUSH2          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000002
(pc) 553 (op) JUMPI          (st) 3     ~> false
(pc) 554 (op) PUSH2          (st) 1     => 0x0000000000000000000000000000000000000000000000000000000000000236
(pc) 557 (op) PUSH1          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000004
(pc) 559 (op) DUP1           (st) 3     => [1] 0x0000000000000000000000000000000000000000000000000000000000000004
(pc) 560 (op) POP            (st) 4     => 0x0000000000000000000000000000000000000000000000000000000000000004
(pc) 561 (op) POP            (st) 3     => 0x0000000000000000000000000000000000000000000000000000000000000004
(pc) 562 (op) PUSH2          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000B7E
(pc) 565 (op) JUMP           (st) 3     ~> 2942
(pc) 2942 (op) JUMPDEST       (st) 2    
(pc) 2943 (op) PUSH1          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2945 (op) PUSH1          (st) 3     => 0x0000000000000000000000000000000000000000000000000000000000000003
(pc) 2947 (op) PUSH1          (st) 4     => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2949 (op) SWAP1          (st) 5     => [2] 0000000000000000000000000000000000000000000000000000000000000003
(pc) 2950 (op) SLOAD          (st) 5     {0x0000000000000000000000000000000000000000000000000000000000000003 : 0x0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB}
(pc) 2951 (op) SWAP1          (st) 5     => [2] 0000000000000000000000000000000000000000000000000000000000000000
(pc) 2952 (op) PUSH2          (st) 5     => 0x0000000000000000000000000000000000000000000000000000000000000100
(pc) 2955 (op) EXP            (st) 6     256 ** 0 = 1 (0000000000000000000000000000000000000000000000000000000000000001)
(pc) 2956 (op) SWAP1          (st) 5     => [2] 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB
(pc) 2957 (op) DIV            (st) 5     697540357585017921034752128164944730699623392475 / 1 = 697540357585017921034752128164944730699623392475 (0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB)
(pc) 2958 (op) PUSH20         (st) 4     => 0x000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
(pc) 2979 (op) AND            (st) 5     000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF & 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB = 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB
(pc) 2980 (op) SWAP1          (st) 4     => [2] 0000000000000000000000000000000000000000000000000000000000000000
(pc) 2981 (op) POP            (st) 4     => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2982 (op) PUSH2          (st) 3     => 0x0000000000000000000000000000000000000000000000000000000000000BAA
(pc) 2985 (op) JUMP           (st) 4     ~> 2986
(pc) 2986 (op) JUMPDEST       (st) 3    
(pc) 2987 (op) SWAP1          (st) 3     => [2] 0000000000000000000000000000000000000000000000000000000000000236
(pc) 2988 (op) JUMP           (st) 3     ~> 566
(pc) 566 (op) JUMPDEST       (st) 2    
(pc) 567 (op) PUSH1          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000040
(pc) 569 (op) MLOAD          (st) 3     => 0x0000000000000000000000000000000000000000000000000000000000000060 @ 0x40
(pc) 570 (op) DUP1           (st) 3     => [1] 0x0000000000000000000000000000000000000000000000000000000000000060
(pc) 571 (op) DUP3           (st) 4     => [3] 0x0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB
(pc) 572 (op) PUSH20         (st) 5     => 0x000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
(pc) 593 (op) AND            (st) 6     000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF & 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB = 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB
(pc) 594 (op) DUP2           (st) 5     => [2] 0x0000000000000000000000000000000000000000000000000000000000000060
(pc) 595 (op) MSTORE         (st) 6     => 0x0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB @ 0x60
(pc) 596 (op) PUSH1          (st) 4     => 0x0000000000000000000000000000000000000000000000000000000000000020
(pc) 598 (op) ADD            (st) 5     32 + 96 = 128 (0000000000000000000000000000000000000000000000000000000000000080)
(pc) 599 (op) SWAP2          (st) 4     => [3] 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB
(pc) 600 (op) POP            (st) 4     => 0x0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB
(pc) 601 (op) POP            (st) 3     => 0x0000000000000000000000000000000000000000000000000000000000000060
(pc) 602 (op) PUSH1          (st) 2     => 0x0000000000000000000000000000000000000000000000000000000000000040
(pc) 604 (op) MLOAD          (st) 3     => 0x0000000000000000000000000000000000000000000000000000000000000060 @ 0x40
(pc) 605 (op) DUP1           (st) 3     => [1] 0x0000000000000000000000000000000000000000000000000000000000000060
(pc) 606 (op) SWAP2          (st) 4     => [3] 0000000000000000000000000000000000000000000000000000000000000080
(pc) 607 (op) SUB            (st) 4     128 - 96 = 32 (0000000000000000000000000000000000000000000000000000000000000020)
(pc) 608 (op) SWAP1          (st) 3     => [2] 0000000000000000000000000000000000000000000000000000000000000060
(pc) 609 (op) RETURN         (st) 3     => [96, 32] (32) 0x0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB
resume 0000000000000000000000008C47FC114D5AD90872C4A10BB0128C6707913EC8 (0xc421f74118)
(pc) 2143 (op) ISZERO         (st) 9      == 0 = 0
(pc) 2144 (op) PUSH2          (st) 9     => 0x0000000000000000000000000000000000000000000000000000000000000002
(pc) 2147 (op) JUMPI          (st) 10    ~> false
(pc) 2148 (op) POP            (st) 8     => 0x0000000000000000000000000000000000000000000000000000000000000064
(pc) 2149 (op) POP            (st) 7     => 0x000000000000000000000000000000000000000000000000000000003A067CA6
(pc) 2150 (op) POP            (st) 6     => 0x0000000000000000000000007B02183F9397EB10AF226693385C1C900409229D
(pc) 2151 (op) PUSH1          (st) 5     => 0x0000000000000000000000000000000000000000000000000000000000000040
(pc) 2153 (op) MLOAD          (st) 6     => 0x0000000000000000000000000000000000000000000000000000000000000060 @ 0x40
(pc) 2154 (op) DUP1           (st) 6     => [1] 0x0000000000000000000000000000000000000000000000000000000000000060
(pc) 2155 (op) MLOAD          (st) 7     => 0x0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB @ 0x60
(pc) 2156 (op) SWAP1          (st) 7     => [2] 0000000000000000000000000000000000000000000000000000000000000060
(pc) 2157 (op) PUSH1          (st) 7     => 0x0000000000000000000000000000000000000000000000000000000000000020
(pc) 2159 (op) ADD            (st) 8     32 + 96 = 128 (0000000000000000000000000000000000000000000000000000000000000080)
(pc) 2160 (op) POP            (st) 7     => 0x0000000000000000000000000000000000000000000000000000000000000080
(pc) 2161 (op) PUSH20         (st) 6     => 0x000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
(pc) 2182 (op) AND            (st) 7     000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF & 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB = 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB
(pc) 2183 (op) EQ             (st) 6     0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB == 0000000000000000000000007A2ECA9A87849E167F63632EAF98F3316B8564DB = 1
(pc) 2184 (op) ISZERO         (st) 5      == 0 = 0
(pc) 2185 (op) ISZERO         (st) 5      == 0 = 1
(pc) 2186 (op) PUSH2          (st) 5     => 0x0000000000000000000000000000000000000000000000000000000000000892
(pc) 2189 (op) JUMPI          (st) 6     ~> 2194
(pc) 2194 (op) JUMPDEST       (st) 4    
(pc) 2195 (op) PUSH1          (st) 4     => 0x0000000000000000000000000000000000000000000000000000000000000001
(pc) 2197 (op) PUSH1          (st) 5     => 0x0000000000000000000000000000000000000000000000000000000000000008
(pc) 2199 (op) PUSH1          (st) 6     => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2201 (op) DUP3           (st) 7     => [3] 0x0000000000000000000000000000000000000000000000000000000000000001
(pc) 2202 (op) DUP3           (st) 8     => [3] 0x0000000000000000000000000000000000000000000000000000000000000008
(pc) 2203 (op) DUP3           (st) 9     => [3] 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2204 (op) POP            (st) 10    => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2205 (op) SLOAD          (st) 9     {0x0000000000000000000000000000000000000000000000000000000000000008 : 0x0000000000000000000000000000000000000000000000000000000000000007}
(pc) 2206 (op) ADD            (st) 9     7 + 1 = 8 (0000000000000000000000000000000000000000000000000000000000000008)
(pc) 2207 (op) SWAP3          (st) 8     => [4] 0000000000000000000000000000000000000000000000000000000000000001
(pc) 2208 (op) POP            (st) 8     => 0x0000000000000000000000000000000000000000000000000000000000000001
(pc) 2209 (op) POP            (st) 7     => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2210 (op) DUP2           (st) 6     => [2] 0x0000000000000000000000000000000000000000000000000000000000000008
(pc) 2211 (op) SWAP1          (st) 7     => [2] 0000000000000000000000000000000000000000000000000000000000000008
(pc) 2212 (op) SSTORE         (st) 7     {0x0000000000000000000000000000000000000000000000000000000000000008 : 0x0000000000000000000000000000000000000000000000000000000000000008}
(pc) 2213 (op) POP            (st) 5     => 0x0000000000000000000000000000000000000000000000000000000000000008
(pc) 2214 (op) PUSH1          (st) 4     => 0x0000000000000000000000000000000000000000000000000000000000000005
(pc) 2216 (op) PUSH1          (st) 5     => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2218 (op) POP            (st) 6     => 0x0000000000000000000000000000000000000000000000000000000000000000
(pc) 2219 (op) DUP1           (st) 5     => [1] 0x0000000000000000000000000000000000000000000000000000000000000005
(pc) 2220 (op) SLOAD          (st) 6    panic: Panicked on a Sanity Check: Value missing for key 4440B2DB04149DB6636438C9744A6ACBDEF40C7A

goroutine 1 [running]:
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-common.PanicSanity(0xb5a3a0, 0xc4215a6b50)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-common/errors.go:26 +0xdd
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle.(*nodeDB).GetNode(0xc42367d8b0, 0xc421ac9f60, 0xc420f58d40, 0x14, 0x14, 0x0)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle/iavl_tree.go:238 +0x504
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle.(*IAVLNode).getLeftNode(0xc421adaa00, 0xc421ac9f60, 0x20)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle/iavl_node.go:356 +0x6c
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle.(*IAVLNode).get(0xc421adaa00, 0xc421ac9f60, 0xc420f58c80, 0x20, 0x20, 0x20, 0x0, 0xc42044c3e0, 0x14, 0x14)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle/iavl_node.go:125 +0x1bb
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle.(*IAVLNode).get(0xc421ada1e0, 0xc421ac9f60, 0xc420f58c80, 0x20, 0x20, 0x20, 0xffffffffffffffff, 0xc42044c220, 0x14, 0x14)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle/iavl_node.go:128 +0x29a
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle.(*IAVLNode).get(0xc421ada140, 0xc421ac9f60, 0xc420f58c80, 0x20, 0x20, 0x20, 0xffffffffffffffff, 0xc42044c0e0, 0x14, 0x14)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle/iavl_node.go:125 +0x1f7
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle.(*IAVLNode).get(0xc421ada0a0, 0xc421ac9f60, 0xc420f58c80, 0x20, 0x20, 0x20, 0xffffffffffffffff, 0xc420f58c80, 0x0, 0x11bbb00)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle/iavl_node.go:125 +0x1f7
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle.(*IAVLNode).get(0xc421ada000, 0xc421ac9f60, 0xc420f58c80, 0x20, 0x20, 0x4115f8, 0x20, 0xc14fa0, 0xc42005c801, 0xc420f58c80)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle/iavl_node.go:125 +0x1f7
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle.(*IAVLTree).Get(0xc421ac9f60, 0xc420f58c80, 0x20, 0x20, 0xc42005c998, 0x46e384, 0x46b905, 0x1, 0x1)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-merkle/iavl_tree.go:144 +0x5a
github.com/hyperledger/burrow/manager/burrow-mint/state.(*BlockCache).GetStorage(0xc4201211a0, 0x0, 0x11fc478c00000000, 0xba1c47208d95a4d, 0xc83e9107678c12b0, 0x0, 0x0, 0x0, 0x500000000000000, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/state/block_cache.go:117 +0x263
github.com/hyperledger/burrow/manager/burrow-mint/state.(*TxCache).GetStorage(0xc4207aa000, 0x0, 0x11fc478c00000000, 0xba1c47208d95a4d, 0xc83e9107678c12b0, 0x0, 0x0, 0x0, 0x500000000000000, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/state/tx_cache.go:124 +0x19b
github.com/hyperledger/burrow/manager/burrow-mint/evm.(*VM).call(0xc4230d6020, 0xc420d15d00, 0xc420d15b80, 0xc42012cf00, 0x22e6, 0x22e6, 0xc424fac150, 0x24, 0x24, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/evm/vm.go:685 +0x131d9
github.com/hyperledger/burrow/manager/burrow-mint/evm.(*VM).Call(0xc4230d6020, 0xc420d15d00, 0xc420d15b80, 0xc42012cf00, 0x22e6, 0x22e6, 0xc424fac150, 0x24, 0x24, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/evm/vm.go:155 +0x23c
github.com/hyperledger/burrow/manager/burrow-mint/evm.(*VM).call(0xc4230d6020, 0xc420d15880, 0xc420d15d00, 0xc420481500, 0x29e7, 0x29e7, 0xc420481500, 0x29e7, 0x29e7, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/evm/vm.go:887 +0x19504
github.com/hyperledger/burrow/manager/burrow-mint/evm.(*VM).Call(0xc4230d6020, 0xc420d15880, 0xc420d15d00, 0xc420481500, 0x29e7, 0x29e7, 0xc420481500, 0x29e7, 0x29e7, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/evm/vm.go:155 +0x23c
github.com/hyperledger/burrow/manager/burrow-mint/evm.(*VM).call(0xc4230d6020, 0xc420d15800, 0xc420d15880, 0xc420463800, 0x4685, 0x4685, 0xc421c9c2d0, 0xe4, 0xe4, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/evm/vm.go:802 +0x1798c
github.com/hyperledger/burrow/manager/burrow-mint/evm.(*VM).Call(0xc4230d6020, 0xc420d15800, 0xc420d15880, 0xc420463800, 0x4685, 0x4685, 0xc421c9c2d0, 0xe4, 0xe4, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/evm/vm.go:155 +0x23c
github.com/hyperledger/burrow/manager/burrow-mint/state.ExecTx(0xc4201211a0, 0x1166cc0, 0xc4201071d0, 0xc423680401, 0x1167dc0, 0xc420121440, 0x1173bc0, 0xc4238b7a40, 0xc421bfd460, 0x20)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/state/execution.go:522 +0x3a52
github.com/hyperledger/burrow/manager/burrow-mint.(*BurrowMint).DeliverTx(0xc4204bae10, 0xc420147a00, 0x190, 0x190, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/ubuntu/src/github.com/hyperledger/burrow/manager/burrow-mint/burrow-mint.go:120 +0x2b7
github.com/hyperledger/burrow/vendor/github.com/tendermint/abci/client.(*localClient).DeliverTxAsync(0xc4200587e0, 0xc420147a00, 0x190, 0x190, 0x0)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/abci/client/local_client.go:74 +0x85
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).DeliverTxAsync(0xc4216246d0, 0xc420147a00, 0x190, 0x190, 0xc42268cbd0)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:73 +0x51
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x0, 0x0, 0x1174640, 0xc4216246d0, 0xc420978a00, 0xc423492640, 0x1166300, 0xc4238b77d0)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/state/execution.go:93 +0x764
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/state.(*State).ValExecBlock(0xc42005bba0, 0x0, 0x0, 0x1174640, 0xc4216246d0, 0xc420978a00, 0x11d7748, 0xc42378a770, 0x40edc8)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/state/execution.go:29 +0x82
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/state.(*State).ApplyBlock(0xc42005bba0, 0x0, 0x0, 0x1174640, 0xc4216246d0, 0xc420978a00, 0x1, 0xc420979040, 0x14, 0x14, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/state/execution.go:212 +0x73
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(0xc420155e00, 0x32467, 0x1174640, 0xc4216246d0, 0x14, 0x20, 0x0, 0x0, 0xc42005ea00)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/consensus/replay.go:360 +0x173
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(0xc420155e00, 0x1175ec0, 0xc4200ee090, 0x0, 0x32467, 0x1, 0xc420102bb0, 0xc4200ee1b0, 0x0, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/consensus/replay.go:346 +0x323
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc420155e00, 0x0, 0x0, 0x0, 0x0, 0x1175ec0, 0xc4200ee090, 0x0, 0x0, 0x0, ...)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/consensus/replay.go:296 +0x766
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc420155e00, 0x1175ec0, 0xc4200ee090, 0x0, 0x0)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/consensus/replay.go:235 +0x451
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/proxy.(*multiAppConn).OnStart(0xc4200ee090, 0xc42014d540, 0x15)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/proxy/multi_app_conn.go:99 +0x212
github.com/hyperledger/burrow/vendor/github.com/tendermint/go-common.(*BaseService).Start(0xc4200ee090, 0x0, 0x0, 0x0)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/go-common/service.go:99 +0x2c2
github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/node.NewNode(0x11767a0, 0xc42000e3e8, 0xc4204466e0, 0x1167fc0, 0xc421623340, 0x2)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/tendermint/tendermint/node/node.go:84 +0x592
github.com/hyperledger/burrow/consensus/tendermint.NewTendermint(0xc420142200, 0x1175740, 0xc4204bae10, 0x1173bc0, 0xc420121080, 0xb5a3a0, 0xc420150280, 0xb5a3a0)
	/home/ubuntu/src/github.com/hyperledger/burrow/consensus/tendermint/tendermint.go:120 +0xda2
github.com/hyperledger/burrow/consensus.LoadConsensusEngineInPipe(0xc420142200, 0x1176840, 0xc42005eb00, 0x24, 0x0)
	/home/ubuntu/src/github.com/hyperledger/burrow/consensus/consensus.go:38 +0x191
github.com/hyperledger/burrow/core.NewCore(0xc4204c9e80, 0x6, 0xc420142200, 0xc420142280, 0x1173bc0, 0xc420135dd0, 0x4, 0x0, 0x0)
	/home/ubuntu/src/github.com/hyperledger/burrow/core/core.go:63 +0x302
github.com/hyperledger/burrow/cmd.NewCoreFromDo(0xc4204d0840, 0x7fff8e0897bb, 0x1a, 0xc78aab)
	/home/ubuntu/src/github.com/hyperledger/burrow/cmd/serve.go:159 +0xaf1
github.com/hyperledger/burrow/cmd.ServeRunner.func1(0xc42008dd40, 0xc4204dce70, 0x0, 0x3)
	/home/ubuntu/src/github.com/hyperledger/burrow/cmd/serve.go:177 +0xab
github.com/hyperledger/burrow/vendor/github.com/spf13/cobra.(*Command).execute(0xc42008dd40, 0xc4204dcd50, 0x3, 0x3, 0xc42008dd40, 0xc4204dcd50)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/spf13/cobra/command.go:635 +0x23a
github.com/hyperledger/burrow/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x11b2c00, 0x1, 0x1, 0xc42008dd40)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/spf13/cobra/command.go:710 +0x339
github.com/hyperledger/burrow/vendor/github.com/spf13/cobra.(*Command).Execute(0x11b2c00, 0x0, 0xc4204dcc00)
	/home/ubuntu/src/github.com/hyperledger/burrow/vendor/github.com/spf13/cobra/command.go:669 +0x2b
github.com/hyperledger/burrow/cmd.Execute()
	/home/ubuntu/src/github.com/hyperledger/burrow/cmd/burrow.go:46 +0x3a5
main.main()
	/home/ubuntu/src/github.com/hyperledger/burrow/cmd/burrow/main.go:22 +0x20
@ratranqu ratranqu changed the title Panic in SLOAD Panic in SLOAD when reloading the chain after it died or during normal chain operation Jul 18, 2017
@ratranqu
Copy link
Contributor Author

Following @benjaminbollen's suggestion, I removed this commit: 288737d .

It seems to make the problems go away. Not really conclusive at this stage, but strong indication that this commit introduces a regression.

@silasdavis
Copy link
Contributor

@ratranqu are you able to share the solidity that caused this? Or a minimal reproduction in solidity?

@silasdavis
Copy link
Contributor

silasdavis commented Aug 4, 2017

Also are you running any transactions that are calling SELFDESTRUCT around the same time? I'm expecting to see that somewhere.

It looks like we have an attempt to access the storage of a destructed contract. Off the top of my head I don't think that makes sense in terms of the EVM semantics, though I could be wrong. That being the case we may be seeing a non-deterministic reordering of transactions during the commit phase or something like that, though I need to dig a bit more.

@silasdavis
Copy link
Contributor

Are you consistently seeing this issue with 288737d and not without? From the IAVL tree perspective I suspect that a node is getting orphaned and removed by one tree that shares a node db with another. Again we'll really need a working test case to look into this further.

@silasdavis
Copy link
Contributor

silasdavis commented Aug 6, 2017

It's worth noting that the panic originates outside of the BlockCache in go-merkle. go-merkle doesn't expect a dangling node pointer to ever occur (whether than is reasonable on go-merkle's account given it allows IAVLTrees to share a node database is another matter), so it seems like this issue might be caused by change to caching behaviour, but it should not really be happening anyway.

@benjaminbollen
Copy link
Contributor

There are two factors that play here: the updates to go-merkle and the updates to BlockCache; as described in go-merkle, the likely solution is to construct a new BlockCache after committing to the database

@ratranqu
Copy link
Contributor Author

ratranqu commented Aug 8, 2017

thx guys for looking into it. @silasdavis: there are no explicit self-destruct in the solidity code, and yes I am consistently seeing this issue with 288737d and not without.

@silasdavis
Copy link
Contributor

silasdavis commented Aug 8, 2017

That may be a solution, but I don't yet see how this is being triggered. I would like to fix against a clear failing test case. Are you able to share some solidity code that caused this so we can reproduce?

@silasdavis
Copy link
Contributor

silasdavis commented Aug 18, 2017

@ratranqu any chance you could provide us with some solidity/usage that reproduces this issue? We really don't have enough to go on just from stacktrace without being able to get into the same initial state and with knowing what kind of requests are hitting the instance. A minimal test case would be much appreciate. Or failing that if you could share some code in private and we could try to come up with one.

This was referenced Aug 24, 2017
@silasdavis
Copy link
Contributor

@ratranqu bump, can you provide any more detailed reproduction on this?

@ratranqu
Copy link
Contributor Author

ratranqu commented Sep 12, 2017 via email

@silasdavis
Copy link
Contributor

silasdavis commented Sep 12, 2017

I can appreciate the time issue, if you could share the state that would be great (and probably enough for me to progress with it). If you are able to share it here then you should be able to drag and drop a .zip/.gz into the comment box and it will upload. Failing that you could send it to me at .iosilas@monax (rotate left 3).

@compleatang
Copy link
Contributor

this has all changed dramatically. closing for now.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

No branches or pull requests

4 participants