Remove pubkey cache file

[michaelsproul]

Description

The pubkey cache file is unnecessary when we have a perfectly good database available. I think it would save complexity and reduce the likelihood of error if it were removed and rolled into the database. Its separate handling has been the source of bugs, e.g. #1680

This also links into the locking timeout issue #1096.

[michaelsproul]

From Discord:

CRIT Failed to start beacon node             reason: Unable to open persisted pubkey cache: ValidatorPubkeyCacheFileError("InconsistentIndex { expected: Some(79843), found: 0 }")

Reportedly occurred after OOM.

[paulhauner]

The pubkey cache file is unnecessary when we have a perfectly good database available.

I'm not sure it's trivial to rebuild the cache from any database. We don't just need all the keys in the head state, we need to go and find the state with the largest state.validators.len() and add all the validators from there, since we assume that the pubkey cache includes every single validator we've ever seen.

[michaelsproul]

I just mean that we store a table in the DB that is exactly the same as the current pubkey cache, so we get DB writes rather than hand-rolled ones.

A user on Discord reported corruption after a power outage:

Hi I followed  Esat's guide.  I had all 3 services connected and synced.  But,  after simulated power loss on my server the Lighthouse Beacon service won't start: 

Unable to open persisted pubkey cache: ValidatorPubkeyCacheFileError("Ssz(InvalidByteLength { len: 32, expected: 56 })")

[apalade]

I am experiencing the same problem after a power outage.

Dec 20 15:24:26 pala lighthouse[50671]: Dec 20 15:24:26.109 CRIT Failed to start beacon node reason: Unable to open persisted pubkey cache: ValidatorPubkeyCacheFileError("InconsistentIndex { expected: Some(48577), found: 0 }")

Questions:

1. the only way out of this right now is to resync the database?
2. would it be safe to use a pubkey_cache.ssz file from an older backup instead of a full resync?

[michaelsproul]

I've sent you the file on Discord, but just in case you check GH first:

1. the only way out of this right now is to resync the database?

Yeah :\

2. would it be safe to use a pubkey_cache.ssz file from an older backup instead of a full resync?

An earlier backup might work if you get lucky, but a backup from another node is more reliable.

[michaelsproul]

Another instance on Discord today after power failure:

CRIT Failed to start beacon node             reason: Unable to open persisted pubkey cache: ValidatorPubkeyCacheFileError("InconsistentIndex { expected: Some(100388), found: 0 }")