🔷 [Epic] Achieve 300BPS historical backfill performance

[Ishatt]

Block rate should be on par (300 BPS) with near lake framework JS for indexing all historical blocks. The benchmark is to have an indexer that reads every block, constructs a near primitives block object and does not store anything in the database.

In addition, historical backfill should work for all contract filters (including "*").

Child Issues

Preview Give feedback

1. Handoff between real-time and historical processing #209
   1 of 1
   +0
2. Investigate Latency Contributors #484
   +0
   
3. Store Hasura Credentials Fetch #581
   component: Runner +1
   
4. Remove unnecessary Runner logs #587
   +0
   
5. Add Instrumentation to Runner #530
   component: Runner +1
   
6. Cache Provisioning Status #603
   +0
   
7. Enable PgBouncer for Production #605
   +0
   
8. Remove Unnecessary Status Writes #604
   +0
   
9. Prepare code for setting Status and Last Processed BlockHeights through Postgres #588
   +0
   
10. Support Transactions for Indexer Code Execution #340
    component: Runner +1
    
11. QueryAPI: Bitmap approach to store block heights #275
    2 of 2
    component: Streamer +1
    
12. Backfill bitmap indexer with databricks data #531
    +0
    
13. Utilize Bitmap Indexer for Historical Backfill #740
    4 of 4
    component: Streamer +1
    
14. Migrating prefetch to block streamer #595
    +0
    
15. Reduce duplicate Lake requests across dedicated streams #419
    component: Streamer +1
    
16. Batch commands with Pipelines #206
    +0
    
17. Scale indexers horizontally across multiple processes #203
    component: Registry +1
18. Use Consumer Groups to process stream messages #202
    +0
19. Provision metadata tables #651
    +0
    
20. Write statuses to metadata table and change the UI #652
    +0