Optimize contract engine for concurrency

[aeyakovenko]

Multi threaded banks need to atomically unlock accounts and linearize the blobs they produce. Otherwise there may be an error replaying the ledger as the blobs get out of order. Some of this work is addressed in #1368

This is what is implemented in #1368

1. PoHRecorder breaks up transactions into chunks that could fit into a blob
2. banking stage for mt banks call poh_recoder.record(txs: Vec)
3. Blob gets full, PoHRecorder atomically calls

• call bank.unlock_accounts(txs: Vec)
• poh.record(Blob's hash)
• sender.send(Entry for Blob)
• call bank.register_entry_id for the Entry only

4. concurrently banking stage calls poh_recorder.tick(), which atomically does the following:

• also emit the poh Tick
• call bank.register_entry_id for the Tick entry
• sender.send(Tick)

5. bank.process_entry only calls bank.register_entry_id for every entry

This is the proposed solution

1. PoHRecorder holds a pending blob.
2. banking stage for mt banks call poh_recoder.record(txs: Vec)
3. Blob gets full, PoHRecorder atomically calls

• call bank.unlock_accounts(txs: Vec)
• poh.record(Blob's hash)
• sender.send(Entry for Blob)

4. concurrently banking stage calls poh_recorder.tick(), which atomically does the following:

• if sender.send has not been called in more then 1 tick (100ms), do the same work as in poh.record with the current partial blob blob
• also emit the Tick
• call bank.register_entry_id for the Tick entry only
• sender.send(Tick)

5. bank.process_entry only calls bank.register_entry_id for the Tick

@garious @rob-solana (captured the todo's left from #1368)

[rob-solana]

how does this batching proposal differ from what's done today? do multiple batches in the blob have to be identified as blocks that can be handled in parallel?

[aeyakovenko]

@rob-solana what the code does right now is construct entries (allocation + copy), then stick them into blobs (a copy). instead of writing into blobs then sending them out to IO. since you are not binning at the IO boundary, you have no idea how to construct the entry in such a way that it fits in whatever space is left in the blob.

[rob-solana]

this issue and the mt-bank PR don't look in sync anymore. are they?

[garious]

No, one of @aeyakovenko's comments on that PR said that he considered it out of scope.

[mvines]

@aeyakovenko, are we optimized? Anything more that needs to be done here?

[aeyakovenko]

woot! optimized!