Fix TxSearch with block-by-block indexing

[avious00]

Fix Neutron Validator Indexing for TxSearch with block-by-block indexing

Problem

We need to design a sustainable method for Cosmos indexing, TxSearch is unreliable.

• it can give back junk responses depending on the timeout of the RPC server
• we require an archive node to index older messages
• sometimes TxSearch isn’t public because it’s such an expensive RPC
• Results in neutron validators getting stuck or not being able to index older messages, so we have gaps in checkpoints

Solution

• We'll go with block-by-block indexing
• The original get_event_log function from Mitosis can serve as an example. Might need some prettier logic
• It doesn't appear a single RPC will give us both tx hashes and the events from those txs
• Following Mitosis's work, we need to use the /block endpoint to get tx hashes, and then the /block_results endpoint to get the events for txs. The latter endpoint just returns an array where each element contains the events of the tx in the corresponding element in the array of txs from /block, but no tx hash. Also /block doesn't actually return the TX hash - it returns the raw tx that you then need to hash to get the tx hash.
  • References to this:
    • proof that the order of txs in /block and /block_results are the same Is the order of tx in block and blockResults the same? tendermint/tendermint#3866
    • Getting the hash from the raw tx Return transactions in blocks cosmos/cosmos-sdk#3729 (comment)
  • Example RPCs:
    • /block https://rpc-magnetix.neutron-1.neutron.org/block?height=6162253
    • /block_results https://rpc-magnetix.neutron-1.neutron.org/block_results?height=6162253
    • tx in question in that block (note this is a message getting processed, not dispatched) https://www.mintscan.io/neutron/tx/336CE709DA9BEC52C1C9A956267611115B5A80C26334AE154DD85A8EFB0E4E6F?height=6162253
• There may be an alternative approach to avoid two RPCs where we'd still use TxSearch but request just a specific height. There's logic in the TxSearch server if only a specific height is requested and not a block range to be more efficient, but because TxSearch tends to be blocked by many RPC providers and iiuc it's still pretty inefficient even in this case, I think it's best to just opt for the /block and /block_results approach and move away from TxSearch entirely.

Additional context

• Notion proposal
• internal Discord thread
• Discord thread with recap Cosmos community recommendations

Related issues

• Scoping: https://github.com/hyperlane-xyz/issues/issues/976
• Broader indexing redesign: Spec out an agent indexing refactor epic #3224
• Related/blocking: https://github.com/hyperlane-xyz/issues/issues/924

Tasks

Preview Give feedback

1. deploy cosmos fallback provider to RC, and debug RPC inconsistencies between Injective and Neutron #3302
   +0