Segmentation fault (Core Dump)

[davkastheranger19]

Node just crashed with the error "Segmentation Fault (Core Dump)"
Syslog show the error:
coin-staker[1778]: segfault at 8 ip 0000555c3514283d sp 00007f2798821ba0 error 4 in pocketcoind[555c34c37000+b61000]
Code: 48 89 ef e8 65 67 af ff 49 8b 37 48 89 ef e8 1a 0b 04 00 48 8b 44 24 40 4c 8d bc 24 80 00 00 00 48 8d 7c 24 70 4c 89 7c 24 70 <48> 8b 50 08 48 8b 30 48 89 7c 24 28 48 01 f2 e8 af 21 fe ff 48 8b

Kernel Log shows the same
coin-staker[1778]: segfault at 8 ip 0000555c3514283d sp 00007f2798821ba0 error 4 in pocketcoind[555c34c37000+b61000]
Code: 48 89 ef e8 65 67 af ff 49 8b 37 48 89 ef e8 1a 0b 04 00 48 8b 44 24 40 4c 8d bc 24 80 00 00 00 48 8d 7c 24 70 4c 89 7c 24 70 <48> 8b 50 08 48 8b 30 48 89 7c 24 28 48 01 f2 e8 af 21 fe ff 48 8b

Apport log. Not sure if this is useful.
ERROR: apport (pid 696492) Sat Nov 19 11:29:59 2022: called for pid 1651, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 696492) Sat Nov 19 11:29:59 2022: executable: /usr/local/bin/pocketcoind (command line "pocketcoind")
ERROR: apport (pid 696492) Sat Nov 19 11:29:59 2022: executable does not belong to a package, ignoring

Running Ubuntu Server 22.04 LTS
Pocketnet Core 0.20.28
I have two nodes and both dumped at the same time as did a dozen others according to the Node Owners Chat.

[americanpatriotdave]

~# cat /var/log/kern.log

Nov 19 16:30:00 pocketnet-node07 kernel: [177315.826960] show_signal_msg: 21 callbacks suppressed
Nov 19 16:30:00 pocketnet-node07 kernel: [177315.827074] coin-staker[4384]: segfault at 8 ip 000055d020c2d83d sp 00007fb991520c40 error 4 in pocketcoind[55d020722000+b61000]
Nov 19 16:30:00 pocketnet-node07 kernel: [177315.827129] Code: 48 89 ef e8 65 67 af ff 49 8b 37 48 89 ef e8 1a 0b 04 00 48 8b 44 24 40 4c 8d bc 24 80 00 00 00 48 8d 7c 24 70 4c 89 7c 24 70 <48> 8b 50 08 48 8b 30 48 89 7c 24 28 48 01 f2 e8 af 21 fe ff 48 8b

Hope this helps.

[andyoknen]

Could you give the output result of the command?

pocketcoin-cli getmempoolinfo

[andyoknen]

It also wouldn't hurt to use the debug.log part during node startup to understand how high the blockchain is now

[americanpatriotdave]

pocketcoin-cli getmempoolinfo

~# pocketcoin-cli getmempoolinfo
{
"size": {
"memory": 68,
"sqlite": 1610
},
"bytes": 20323,
"usage": 83088,
"maxmempool": 300000000,
"mempoolminfee": 0.00001000,
"minrelaytxfee": 0.00001000
}

2022-11-19T16:30:00Z Available coins count 361 BestHeader: 1970366 079b5a7d2972cd1f0c4cbb4138da0be4

[davkastheranger19]

NODE 1
{
"size": {
"memory": 11,
"sqlite": 1135
},
"bytes": 4412,
"usage": 14880,
"maxmempool": 300000000,
"mempoolminfee": 0.00001000,
"minrelaytxfee": 0.00001000
}

NODE 2
{
"size": {
"memory": 12,
"sqlite": 1405
},
"bytes": 4844,
"usage": 16304,
"maxmempool": 300000000,
"mempoolminfee": 0.00001000,
"minrelaytxfee": 0.00001000
}

[andyoknen]

Is the error reproducible? I can assemble a node with additional logging that can help find the cause. I can't reproduce the problem on my nodes..

[andyoknen]

https://github.com/pocketnetteam/pocketnet.core/actions/runs/3507683807

This build is based on 0.20.28 and includes additional logging of the staking process in which an error occurs.

To enable add this line to pocketcoin.conf

debug=miner

[americanpatriotdave]

Running the test build with the debug setting as requested.
installed the .deb package over the stock v0.20.28

Node up and running with all kinds of new (and expected) debug.log entries.
Will keep you posted.

[americanpatriotdave]

Is the error reproducible? I can assemble a node with additional logging that can help find the cause. I can't reproduce the problem on my nodes..

The error is not reproducible. I was able to just bounce pocketcoind and it was running fine again.

[andyoknen]

The error is not reproducible. I was able to just bounce pocketcoind and it was running fine again.

I will wait - detailed log is needed to catch a transaction that causes failure

[nienburg]

I'm running the docker hub container and it crashed with it, as well as again today. going by the netstakeweight it seemed to be quite a lot of nodes.

[dmaltsiniotis]

I'm also having this issue. The processes segfaulted on Linux a few minutes ago.

Nov 21 11:37:55 debian kernel: [3454459.655333] coin-staker[3776963]: segfault at 8 ip 000055de2e72f28c sp 00007fb5f66c2450 error 4 in pocketcoind[55de2e200000+164d000]
Nov 21 11:37:55 debian kernel: [3454459.655342] Code: 89 df e8 37 e4 b6 ff 49 8b 34 24 48 89 df e8 5b 31 03 00 4c 8d 64 24 50 48 8b 44 24 20 49 8d 54 24 10 4c 89 e7 48 89 54 24 50 <48> 8b 50 08 48 8b 30 48 01 f2 e8 75 8d fd ff 48 8b 7c 24 28 48 85

~$ pocketcoind --version
0000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Pocketnet Core Daemon version 0.20.28

I'm not using the debug build posted above yet, I will switch to it in hopes of catching the bad transaction.

[davkastheranger19]

Using the debug build and both nodes crashed with the same transaction
2022-11-21T17:37:54Z Testing Pocket transaction completed. New block size: 1
2022-11-21T17:37:54Z Select transaction 0620a28387fa9e340edb8a1a9b430b7046c7dbfd61adc8c88cf17aae2f23f26b for new block at height 1973312
2022-11-21T17:37:54Z Checking transaction 0620a28387fa9e340edb8a1a9b430b7046c7dbfd61adc8c88cf17aae2f23f26b with result 0 completed

[americanpatriotdave]

crash-debug-20221121-1737Z.log

[nienburg]

When can we expect a new release that solves the crashing issues?

[andyoknen]

I am preparing a release for release - I think it will be today or tomorrow at most