CRIT Failed to truncate extra state histories err=EOF

[SimonSMH1015]

System information

Geth version: geth 1.13.0
CL client & version: e.g. [email protected]
OS & Version: Linux

Expected behaviour

After looking the new feature provided by version 1.13.0, I updated my node. I leaved only the ancient blocks and restarted my geth with --state.scheme=path. It tookl me around 6 hour to resynced blocks and generated snapshot.

To test the Semi-instant shutdowns, I forced shutdown my server without closing geth. Then I got the Error after I lauched the geth :-)

INFO [09-13|17:38:43.874] Starting Geth on Ethereum mainnet...
INFO [09-13|17:38:43.874] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [09-13|17:38:43.876] Maximum peer count                       ETH=50 LES=0 total=50
INFO [09-13|17:38:43.877] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [09-13|17:38:44.034] Using pebble as the backing database
INFO [09-13|17:38:44.034] Allocated cache and file handles         database=/mnt/eth/geth/geth/chaindata cache=2.00GiB handles=32767
INFO [09-13|17:38:44.705] Opened ancient database                  database=/mnt/eth/geth/geth/chaindata/ancient/chain readonly=true
INFO [09-13|17:38:44.705] State scheme set by user                 scheme=path
INFO [09-13|17:38:44.710] Set global gas cap                       cap=50,000,000
INFO [09-13|17:38:44.716] Initializing the KZG library             backend=gokzg
INFO [09-13|17:38:44.759] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [09-13|17:38:44.838] Using pebble as the backing database
INFO [09-13|17:38:44.838] Allocated cache and file handles         database=/mnt/eth/geth/geth/chaindata cache=2.00GiB handles=32767
INFO [09-13|17:38:45.737] Opened ancient database                  database=/mnt/eth/geth/geth/chaindata/ancient/chain readonly=false
INFO [09-13|17:38:45.739] Initialising Ethereum protocol           network=1 dbversion=8
WARN [09-13|17:38:45.739] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
INFO [09-13|17:38:45.739] Failed to load journal, discard it       err="journal not found"
INFO [09-13|17:38:45.741] Opened ancient database                  database=/mnt/eth/geth/geth/chaindata/ancient/state readonly=false
CRIT [09-13|17:38:46.774] Failed to truncate extra state histories err=EOF

Now I have to synced form 0 as leave the ancient block data still leading to this problem.

[karalabe]

The error seems to be with the ancient/state thing, which is only used by pathdb. If you resync, you could also delete that and leave ancient/chain. That will save a lot. Will look into what's happening here. My guess is that it's some very stupid simple small check.

[rjl493456442]

Do you still have the corrupted database which can reproduce this issue? Or do you keep the ancient folder by any chance?

[rjl493456442]

My hunch is the state ancientstore is somehow corrupted by losing some data chunk, but it's really hard to say without enough debug information. I will try to expose more error messages and hopefully it can be reproduced.

[SimonSMH1015]

Do you still have the corrupted database which can reproduce this issue? Or do you keep the ancient folder by any chance?

Sorry, I did not keep that. My node is now resyncing. But Im happy to do it again and tried to reproduce after synced.

[rjl493456442]

#28115

Would be nice to use this patch to reproduce, with a bit debug information, thank you!

[rjl493456442]

Please keep a bit logs before shutdown as well, easier to know what's the previous status.

[SimonSMH1015]

geth.log

I ve reproduced this ,and will kept the broken db as you may need more infos @rjl493456442

[rjl493456442]

Can you please ls -l the state ancient folder?

e.g. :~/xxx/geth/chaindata/ancient/state$ ls -l

[rjl493456442]

The thing I have discovered via the log is:

a) The index file is zero size after restart, which truncates the head data file respectively

WARN [09-14|21:09:33.203] Truncating dangling head database=/mnt/eth/geth/geth/chaindata/ancient/state table=history.meta indexed=0 stored=413,253

b) The EOF error shouldn't occur, even with a corrupted ancient database. The only exception is the index refers to some non-existent data which results in the EOF read.

[ryny24]

Yesterday I did a resync with "--state.scheme=path" and now my GETH won't start, with this error. What do I do??

[rjl493456442]

@ryny24 can you share a bit more logs? You also reboot the machine without stopping geth?

[ryny24]

I did not reboot unless there was a power issue I'm not aware of, this machine is off-site, I only stopped GETH after the constant restart phone alerts woke up my wife lol. Here is the last of the log before it tries again.

INFO [09-15|03:14:49.764] Set global gas cap                       cap=50,000,000
INFO [09-15|03:14:49.765] Initializing the KZG library             backend=gokzg
INFO [09-15|03:14:49.919] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [09-15|03:14:50.055] Using pebble as the backing database
INFO [09-15|03:14:50.055] Allocated cache and file handles         database=/home/ryan/.ethereum/geth/chaindata cache=2.00GiB handles=26
2,144
INFO [09-15|03:14:52.116] Opened ancient database                  database=/home/ryan/.ethereum/geth/chaindata/ancient/chain readonly=f
alse
INFO [09-15|03:14:52.120] Initialising Ethereum protocol           network=1 dbversion=8
WARN [09-15|03:14:52.120] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
INFO [09-15|03:14:52.121] Failed to load journal, discard it       err="journal not found"
INFO [09-15|03:14:52.122] Opened ancient database                  database=/home/ryan/.ethereum/geth/chaindata/ancient/state readonly=f
alse
CRIT [09-15|03:14:52.200] Failed to truncate extra state histories err=EOF

[ryny24]

Here is my state folder. I hope this is enough for now, I'm going back to bed.

total 180
-rw-r--r-- 1 ryan ryan     0 Sep 15 03:10 account.data.0000.cdat
-rw-r--r-- 1 ryan ryan 25872 Sep 15 03:07 account.data.cidx
-rw-r--r-- 1 ryan ryan     3 Sep 14 12:26 account.data.meta
-rw-r--r-- 1 ryan ryan     0 Sep 15 03:10 account.index.0000.cdat
-rw-r--r-- 1 ryan ryan 25872 Sep 15 03:07 account.index.cidx
-rw-r--r-- 1 ryan ryan     3 Sep 14 12:26 account.index.meta
-rw------- 1 ryan ryan     0 Sep 14 12:26 FLOCK
-rw-r--r-- 1 ryan ryan     0 Sep 15 03:10 history.meta.0000.rdat
-rw-r--r-- 1 ryan ryan     3 Sep 14 12:26 history.meta.meta
-rw-r--r-- 1 ryan ryan 25872 Sep 15 03:07 history.meta.ridx
-rw-r--r-- 1 ryan ryan     0 Sep 15 03:10 storage.data.0000.cdat
-rw-r--r-- 1 ryan ryan 25872 Sep 15 03:07 storage.data.cidx
-rw-r--r-- 1 ryan ryan     3 Sep 14 12:26 storage.data.meta
-rw-r--r-- 1 ryan ryan     0 Sep 15 03:10 storage.index.0000.cdat
-rw-r--r-- 1 ryan ryan 25872 Sep 15 03:07 storage.index.cidx
-rw-r--r-- 1 ryan ryan     3 Sep 14 12:26 storage.index.meta

[ryny24]

Oh yes, it looks there was a power issue that caused GETH to shutdown unclean. Can I just delete the ancient only and just resync that?

[rjl493456442]

What's the OS you are using?

[rjl493456442]

Can you ls -l the parent folder of ancient/state?

[ryny24]

What's the OS you are using?

Debian GNU/Linux 11 (bullseye)

[ryny24]

the parent folder of ancient/state

The the parent folder of ancient/state would be chaindata, correct?

There are 110,975 items in chaindata. I'll share a google doc instead of flooding this thread.

Thank you.

[ryny24]

I tried to remove the ancient by doing "geth removedb" and selecting N for state, then Y. But I'm still getting the same error.

[rjl493456442]

I am sorry, I referred to ancient folder.

[ryny24]

Sorry, I removed everything and started a full sync.

[rjl493456442]

Hi @SimonSMH1015

Could you please attempt to reproduce the issue once more using this specific branch: #28115?

Based on the log you've provided, my suspicion is that the last item in the index file might be corrupted. Unfortunately, I'm unable to replicate it locally. I'm looking to gather more information, although I am not sure if it's available for you.

[SimonSMH1015]

Hi @SimonSMH1015

Could you please attempt to reproduce the issue once more using this specific branch: #28115?

Based on the log you've provided, my suspicion is that the last item in the index file might be corrupted. Unfortunately, I'm unable to replicate it locally. I'm looking to gather more information, although I am not sure if it's available for you.

Sure， I would like to resynced again and tried to reproduce the issue once more. No worries

[rjl493456442]

@SimonSMH1015 Can you repro it?

[SimonSMH1015]

@SimonSMH1015 Can you repro it?

Not yet. Trying reproduce it on goerli might be a little long. I shall test this again on Sepolia at the same time

[Titan-Node]

Hi all,
cc @rjl493456442
Getting the same issue here: logs.txt

Checked my system reboot logs, looks like my server rebooted 12 hours ago while asleep. Woke up to this error.

Let me know if there is any troubleshooting I can do to help solve this issue.
Thanks

[Titan-Node]

@rjl493456442
I've ran your https://github.com/rjl493456442/go-ethereum/tree/state-freezer-errormsg branch.
Here are the logs:
geth state-freezer-errormsg logs.txt

Also ran https://github.com/rjl493456442/go-ethereum/tree/add-freezer-errormsg branch.
Logs here:
geth add-freezer-errormsg logs.txt

[rjl493456442]

Thank you for the information.

The thing can be confirmed is that the index file of freezer is corrupted:

• The last index item in index file is (0, 0)

I am not sure why it can happen, but leave the comment here for further investigation.

[Titan-Node]

@rjl493456442 is there any way to recover the database or is this a mandatory resync?

Also if you need any further testing let me know and I can keep the node in this state.

Just switched in Erigon in the meantime.

[ajmccl]

I've got the same after a power outage. Resyncing now.

[sharkya1]

Same issue here. sudo geth removedb didn't help. I had to manually delete geth/chaindata/ancient/state/*

[unimos50]

Also getting the same error after moving to 1.13.4 (Pebble DB):
Part of the log:

eth-docker-execution-1  | INFO [11-01|09:59:15.788] Initialising Ethereum protocol           network=1 dbversion=8
eth-docker-execution-1  | WARN [11-01|09:59:15.788] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
eth-docker-execution-1  | INFO [11-01|09:59:15.788] Failed to load journal, discard it       err="journal not found"
eth-docker-execution-1  | INFO [11-01|09:59:15.800] Opened ancient database                  database=/var/lib/goethereum/geth/chaindata/ancient/state readonly=false
eth-docker-execution-1  | CRIT [11-01|09:59:17.879] Failed to truncate extra state histories err=EOF
eth-docker-execution-1  | INFO [11-01|10:00:18.598] Starting pprof server                    addr=http://0.0.0.0:6060/debug/pprof
eth-docker-execution-1  | INFO [11-01|10:00:18.607] Starting Geth on Ethereum mainnet... 
eth-docker-execution-1  | INFO [11-01|10:00:18.607] Bumping default cache on mainnet         provided=1024 updated=4096
eth-docker-execution-1  | INFO [11-01|10:00:18.607] Enabling metrics collection 
eth-docker-execution-1  | INFO [11-01|10:00:18.609] Maximum peer count                       ETH=50 LES=0 total=50
eth-docker-execution-1  | INFO [11-01|10:00:18.610] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
eth-docker-execution-1  | INFO [11-01|10:00:18.613] Set global gas cap                       cap=50,000,000
eth-docker-execution-1  | INFO [11-01|10:00:18.623] Initializing the KZG library             backend=gokzg
eth-docker-execution-1  | INFO [11-01|10:00:18.650] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
eth-docker-execution-1  | INFO [11-01|10:00:18.751] Using pebble as the backing database 
eth-docker-execution-1  | INFO [11-01|10:00:18.751] Allocated cache and file handles         database=/var/lib/goethereum/geth/chaindata cache=2.00GiB handles=524,288
eth-docker-execution-1  | INFO [11-01|10:00:20.163] Opened ancient database                  database=/var/lib/goethereum/geth/chaindata/ancient/chain readonly=false
eth-docker-execution-1  | INFO [11-01|10:00:20.164] State scheme set to already existing     scheme=path
eth-docker-execution-1  | INFO [11-01|10:00:20.166] Initialising Ethereum protocol           network=1 dbversion=8
eth-docker-execution-1  | WARN [11-01|10:00:20.166] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
eth-docker-execution-1  | INFO [11-01|10:00:20.167] Failed to load journal, discard it       err="journal not found"
eth-docker-execution-1  | INFO [11-01|10:00:20.178] Opened ancient database                  database=/var/lib/goethereum/geth/chaindata/ancient/state readonly=false
eth-docker-execution-1  | CRIT [11-01|10:00:22.323] Failed to truncate extra state histories err=EOF
eth-docker-execution-1  | INFO [11-01|10:01:23.011] Starting pprof server                    addr=http://0.0.0.0:6060/debug/pprof
eth-docker-execution-1  | INFO [11-01|10:01:23.020] Starting Geth on Ethereum mainnet... 
eth-docker-execution-1  | INFO [11-01|10:01:23.020] Bumping default cache on mainnet         provided=1024 updated=4096
eth-docker-execution-1  | INFO [11-01|10:01:23.020] Enabling metrics collection 
eth-docker-execution-1  | INFO [11-01|10:01:23.022] Maximum peer count                       ETH=50 LES=0 total=50
eth-docker-execution-1  | INFO [11-01|10:01:23.023] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
eth-docker-execution-1  | INFO [11-01|10:01:23.026] Set global gas cap                       cap=50,000,000
eth-docker-execution-1  | INFO [11-01|10:01:23.036] Initializing the KZG library             backend=gokzg
eth-docker-execution-1  | INFO [11-01|10:01:23.069] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
eth-docker-execution-1  | INFO [11-01|10:01:23.173] Using pebble as the backing database 
eth-docker-execution-1  | INFO [11-01|10:01:23.173] Allocated cache and file handles         database=/var/lib/goethereum/geth/chaindata cache=2.00GiB handles=524,288
eth-docker-execution-1  | INFO [11-01|10:01:24.512] Opened ancient database                  database=/var/lib/goethereum/geth/chaindata/ancient/chain readonly=false
eth-docker-execution-1  | INFO [11-01|10:01:24.513] State scheme set to already existing     scheme=path
eth-docker-execution-1  | INFO [11-01|10:01:24.535] Initialising Ethereum protocol           network=1 dbversion=8
eth-docker-execution-1  | WARN [11-01|10:01:24.535] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
eth-docker-execution-1  | INFO [11-01|10:01:24.535] Failed to load journal, discard it       err="journal not found"
eth-docker-execution-1  | INFO [11-01|10:01:24.547] Opened ancient database                  database=/var/lib/goethereum/geth/chaindata/ancient/state readonly=false
eth-docker-execution-1  | CRIT [11-01|10:01:24.616] Failed to truncate extra state histories err=EOF

[rjl493456442]

just to confirm, you also have the poweroff issue to unclean shutdown Geth right?

What's the version you use before the poweroff?
What's the version you use after the poweroff to restart Geth for the first time and corresponding logs?

[nonsense]

@rjl493456442 I hit the same error with a new synced node from scratch running v1.13.2 stable with:

geth --state.scheme=path

It happened due to unclean shutdown of the node, without stopping geth.

I haven't done anything to the node and can give you full root access if you want to debug.

Add your SSH keys to Github and I can send you SSH details if interested.

[DigiDr]

i've just experienced the same thing on an unclean shutdown (server reset) on the latest stable.

[rjl493456442]

@nonsense Yes please, it will be helpful!

[rjl493456442]

Should be fixed by #28483(will make a release very soon), close it now. Feel free to reopen this issue or another new one in case it's not resolved.

Appreciate for the reports from you guys.

[Piradoxlanieve]

For some reason my machine suddenly turned off and geth won't start.
I think it's the same problem.

Nov 10 17:50:15 nodotest geth[1518]: INFO [11-10|17:50:15.477] Failed to load journal, discard it       err="unmatched journal want f16bbaa8e42cf88930f7df7c8080cecb4a13ae0757c04b8c39f9d00f35363991 got 1f94926fcd2911d778cc314f7d54f62ea12>
Nov 10 17:50:15 nodotest geth[1518]: INFO [11-10|17:50:15.498] Opened ancient database                  database=/mnt/ssd/nodoeth1/geth/chaindata/ancient/state readonly=false
Nov 10 17:50:15 nodotest geth[1518]: CRIT [11-10|17:50:15.578] Failed to truncate extra state histories err=EOF
Nov 10 17:50:15 nodotest systemd[1]: eth1.service: Main process exited, code=exited, status=1/FAILURE
Nov 10 17:50:15 nodotest systemd[1]: eth1.service: Failed with result 'exit-code'.
Nov 10 17:50:15 nodotest systemd[1]: eth1.service: Consumed 2.970s CPU time.
Nov 10 17:50:18 nodotest systemd[1]: eth1.service: Scheduled restart job, restart counter is at 43.
Nov 10 17:50:18 nodotest systemd[1]: Stopped Geth Execution Layer Client service for Mainnet.
Nov 10 17:50:18 nodotest systemd[1]: eth1.service: Consumed 2.970s CPU time.
Nov 10 17:50:18 nodotest systemd[1]: Started Geth Execution Layer Client service for Mainnet.
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.743] Starting pprof server                    addr=http://127.0.0.1:6059/debug/pprof
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.752] Starting Geth on Ethereum mainnet...
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.753] Bumping default cache on mainnet         provided=1024 updated=4096
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.753] Enabling metrics collection
Nov 10 17:50:18 nodotest geth[1525]: WARN [11-10|17:50:18.753] --metrics.port specified without --metrics.addr, metrics server will not start.
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.755] Maximum peer count                       ETH=50 LES=0 total=50
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.757] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Nov 10 17:50:18 nodotest geth[1525]: WARN [11-10|17:50:18.759] Sanitizing cache to Go's GC limits       provided=4096 updated=2645
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.760] Set global gas cap                       cap=50,000,000
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.778] Initializing the KZG library             backend=gokzg
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.897] Allocated trie memory caches             clean=396.00MiB dirty=661.00MiB
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.990] Using pebble as the backing database
Nov 10 17:50:18 nodotest geth[1525]: INFO [11-10|17:50:18.990] Allocated cache and file handles         database=/mnt/ssd/nodoeth1/geth/chaindata cache=1.29GiB handles=262,144
Nov 10 17:50:20 nodotest geth[1525]: INFO [11-10|17:50:20.588] Opened ancient database                  database=/mnt/ssd/nodoeth1/geth/chaindata/ancient/chain readonly=false
Nov 10 17:50:20 nodotest geth[1525]: INFO [11-10|17:50:20.589] State scheme set by user                 scheme=path
Nov 10 17:50:20 nodotest geth[1525]: INFO [11-10|17:50:20.591] Initialising Ethereum protocol           network=1 dbversion=8
Nov 10 17:50:20 nodotest geth[1525]: WARN [11-10|17:50:20.591] Sanitizing invalid node buffer size      provided=661.00MiB updated=256.00MiB
Nov 10 17:50:21 nodotest geth[1525]: INFO [11-10|17:50:21.162] Failed to load journal, discard it       err="unmatched journal want f16bbaa8e42cf88930f7df7c8080cecb4a13ae0757c04b8c39f9d00f35363991 got 1f94926fcd2911d778cc314f7d54f62ea12>
Nov 10 17:50:21 nodotest geth[1525]: INFO [11-10|17:50:21.172] Opened ancient database                  database=/mnt/ssd/nodoeth1/geth/chaindata/ancient/state readonly=false
Nov 10 17:50:21 nodotest geth[1525]: CRIT [11-10|17:50:21.239] Failed to truncate extra state histories err=EOF
Nov 10 17:50:21 nodotest systemd[1]: eth1.service: Main process exited, code=exited, status=1/FAILURE
Nov 10 17:50:21 nodotest systemd[1]: eth1.service: Failed with result 'exit-code'.
Nov 10 17:50:21 nodotest systemd[1]: eth1.service: Consumed 3.261s CPU time.
Nov 10 17:50:24 nodotest systemd[1]: eth1.service: Scheduled restart job, restart counter is at 44.
Nov 10 17:50:24 nodotest systemd[1]: Stopped Geth Execution Layer Client service for Mainnet.
Nov 10 17:50:24 nodotest systemd[1]: eth1.service: Consumed 3.261s CPU time.
Nov 10 17:50:24 nodotest systemd[1]: Started Geth Execution Layer Client service for Mainnet.
Nov 10 17:50:24 nodotest geth[1532]: INFO [11-10|17:50:24.495] Starting pprof server                    addr=http://127.0.0.1:6059/debug/pprof
Nov 10 17:50:24 nodotest geth[1532]: INFO [11-10|17:50:24.502] Starting Geth on Ethereum mainnet...
Nov 10 17:50:24 nodotest geth[1532]: INFO [11-10|17:50:24.503] Bumping default cache on mainnet         provided=1024 updated=4096
Nov 10 17:50:24 nodotest geth[1532]: INFO [11-10|17:50:24.503] Enabling metrics collection

[Titan-Node]

@rjl493456442 my ubuntu server rebooted again, same error on 1.13.0

Upgraded to 1.13.5 as per your pull request #28483

Now getting this error on trying to recover the old db.

Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.497] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.501] Starting Geth on Ethereum mainnet...
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.502] Bumping default cache on mainnet         provided=1024 updated=4096
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.502] Enabling metrics collection
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.502] Maximum peer count                       ETH=50 LES=0 total=50
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.503] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.504] Set global gas cap                       cap=50,000,000
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.509] Initializing the KZG library             backend=gokzg
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.535] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.602] Using pebble as the backing database
Nov 15 19:30:36 thunderfury geth[181143]: INFO [11-15|19:30:36.602] Allocated cache and file handles         database=/media/ssd/geth/geth/chaindata cache=2.00GiB handles=262,144
Nov 15 19:30:37 thunderfury geth[181143]: INFO [11-15|19:30:37.417] Opened ancient database                  database=/media/ssd/geth/geth/chaindata/ancient/chain readonly=false
Nov 15 19:30:37 thunderfury geth[181143]: INFO [11-15|19:30:37.418] State scheme set by user                 scheme=path
Nov 15 19:30:37 thunderfury geth[181143]: INFO [11-15|19:30:37.437] Initialising Ethereum protocol           network=1 dbversion=8
Nov 15 19:30:37 thunderfury geth[181143]: WARN [11-15|19:30:37.437] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
Nov 15 19:30:37 thunderfury geth[181143]: INFO [11-15|19:30:37.548] Failed to load journal, discard it       err="unmatched journal want 7c0af404d2dfa0ed4b6257dbb533c4aa4e0c9684715d446f7ebf12cafd0b7aff got d1f932d16ab295c82f5d21ea2bdaf26dd3c64f35d3dc0f9068d56d766195dfc2"
Nov 15 19:30:37 thunderfury geth[181143]: ERROR[11-15|19:30:37.549] Corrupted index file detected            lastOffset=0 indexes=194,985
Nov 15 19:30:37 thunderfury geth[181143]: ERROR[11-15|19:30:37.549] Corrupted index file detected            lastOffset=0 indexes=194,985
Nov 15 19:30:37 thunderfury geth[181143]: ERROR[11-15|19:30:37.550] Corrupted index file detected            lastOffset=0 indexes=194,985
Nov 15 19:30:37 thunderfury geth[181143]: ERROR[11-15|19:30:37.550] Corrupted index file detected            lastOffset=0 indexes=194,985
Nov 15 19:30:37 thunderfury geth[181143]: ERROR[11-15|19:30:37.550] Corrupted index file detected            lastOffset=0 indexes=194,985
Nov 15 19:30:37 thunderfury geth[181143]: INFO [11-15|19:30:37.553] Opened ancient database                  database=/media/ssd/geth/geth/chaindata/ancient/state readonly=false
Nov 15 19:30:37 thunderfury geth[181143]: CRIT [11-15|19:30:37.577] Failed to truncate extra state histories err="EOF, fileid: 0, start: 14231715, length: 4280735581"

Does 1.13.5 need to re-sync in order for this fix to work?

Thanks for your help!

[karalabe]

1.13.5 can recover from some crashes, as long as the data is still there. But if the crash restarted with the 1.13.4 or older code, the data might have already been wiped, making recovery not possible unfortunately.

[Titan-Node]

@karalabe thank you for the quick reply!
Great Ill re-sync with 1.13.5 and hopefully put this mess behind us :)