Hash mismatch on mainnet-beta

[sakridge]

Problem

Proposed Solution

Debug and fix

[sakridge]

cc @ryoqun @carllin

[sakridge]

Related to #14948 ?

[mvines]

Do we know what node(s) are generating the vote hash mismatch? betaVcnkBhHKaWx9o6LoSYrGaoDCskQLm94cUVWqDLS perhaps?

I'm running the new SPL Token program and that node, since Friday-ish, and there have been multiple times now that betaV.. has been generating vote hash mismatches due to behaviour differences between the old and new SPL Token program. cc: solana-labs/solana-program-library#1231

[sakridge]

betaVcnkBhHKaWx9o6LoSYrGaoDCskQLm94cUVWqDLS is responsible for some.

There are others though:
solana-validator.log:[2021-02-15T15:20:40.200817248Z WARN solana_vote_program::vote_state] HS8DF8wP3A8qSHPnzbE6pSvXub4FeZQ6xMNSJuqArAHS dropped vote Vote { slots: [65446328, 65446329, 65446330], hash: 7pvANx9uzekMd4QgQgAceavYAfqdoFmHitC2mGRjZQUP, timestamp: Some(1613402438) } failed to match hash 7pvANx9uzekMd4QgQgAceavYAfqdoFmHitC2mGRjZQUP EdJJSvqkb1HQxkunr3LRRWj75GrcKWHpttejTaJxFqi6

solana-validator.log.1:[2021-02-14T20:45:59.313236194Z WARN solana_vote_program::vote_state] HZzEML2w5Qs6cccMd7o57Vx2sQUuuTaiecEcMU2xceSK dropped vote Vote { slots: [65345240, 65345241, 65345242, 65345243], hash: 7LNYzy6nwN1RBhLnvn8wD7otDqCDtHKg8xSLMgJFjo3W, timestamp: Some(1613335552) } failed to match hash 7LNYzy6nwN1RBhLnvn8wD7otDqCDtHKg8xSLMgJFjo3W zpA2Y2BDS1EwaZxWX1GqRqDyLucXA6FVn6MmWASeavo

solana-validator.log.2:[2021-02-14T00:29:58.362771390Z WARN solana_vote_program::vote_state] 8JJAFng66sS7Lo7fX5Mi3GoBhE4k9Hjz1SWMLK18Er5h dropped vote Vote { slots: [65231999, 65232000, 65232001], hash: 3soTqcq41NqTHKjsaFj8TAWpiA6rtR527qE9NnYrUcvg, timestamp: Some(1613262596) } failed to match hash 3soTqcq41NqTHKjsaFj8TAWpiA6rtR527qE9NnYrUcvg EDDBVi4H7RstYWr5B6BP3XwsFLgt8XKiTNQ9yH5hhtH4

[sakridge]

Last gossip messages for those 3:
135.181.163.160 | 6260 | HS8DF8wP3A8qSHPnzbE6pSvXub4FeZQ6xMNSJuqArAHS | 1.4.27 | 8001 | 8004 | 8005 | 8002 | 8003 |
135.181.137.252 | 2233 | HZzEML2w5Qs6cccMd7o57Vx2sQUuuTaiecEcMU2xceSK | 1.4.27 | 8000 | 8003 | 8004 | 8001 | 8002 |
37.120.245.134 | 491 | 8JJAFng66sS7Lo7fX5Mi3GoBhE4k9Hjz1SWMLK18Er5h |

All on 1.4, so doesn't seem to be a 1.4 vs. 1.5 issue.

[mvines]

Stale issue

[github-actions]

This issue has been automatically locked since there has not been any activity in past 7 days after it was closed. Please open a new issue for related bugs.