amqp_compressed_proof_plugin: Send compressed action merkle proofs over AMQP

[spoonincode]

Change Description

The amqp_compressed_proof_plugin creates “compressed” action merkle proofs over a list of filtered action receivers and their actions once they become irreversible. The proofs are compressed in that they contain as few leafs of the merkle tree as necessary to match the filtered actions with the action_mroot. Additionally, the input stream is constructed in a manner that attempts to be efficient in validation within a smart contract — hashing can be done with minimal copying and working through the proof requires very little temporary memory.

Configuration of the plugin is a mouthful to say the least. amqp_compressed_proof_plugin can generate multiple proofs per block sending them to different AMQP servers and/or exchanges. This is desirable if different actions need to be handled by different downstream consumers on the message bus. But there is no easy way in appbase’s configuration system to define a “group” other then a repeating string parsed in a particular manner. That is what compressed-proof option is here, a repeating string with 4 fields describing a single proof to be created and sent in the form of

NAME=FILTER=SERVER=EXCHANGE

Where,

• NAME: A unique name for this proof. Used for naming the persistence files created for this publisher. I elected to make this manually configurable instead of auto-generated so that users could tweak the exchange, filter list, or server without losing the persistent unconfirmed AMQP queue that goes with each entry.
• FILTER: A comma separated list of receivers to include in the proof.
• SERVER: URI for the AMQP server
• EXCHANGE: exchange name to publish to

An example valid configuration may be, for example

# pusblish all actions on eosio and eosio.token to the exchange named 'wheelhouse' on the local amqp server
compressed-proof = proof1=eosio,eosio.token=amqp://=wheelhouse
# Additionally publish actions on eosio via another exchange named 'stack'
compressed-proof = justSendEOSIOHere=eosio=amqp://=stack

Parts of the implementation refer to a “command stream” when generating or parsing the proof. This terminology comes about because the serialized data structure is effectively three commands that operate a stack. After the final input from the command stream, the stack should have a single entry that is the same as the action_mroot.

At the front of the serialized proof is a boolean indicating if the proof needs to be computed with action return value semantics or not. I believe there may be a desire to include the block header as well. Downstream components consuming the proof may want to generate additional EOSIO transactions based on the actions in the proof. Block header will be required in that use case for tapos fields. I’m not really sure how to handle this yet… maybe the block header should simply be slapped on the front too.

There may also be a desire to increase the filter granularity by filtering on action name.

Typically you would combine this plugin with #9122: downstream consumers will only operate on the compressed action merkle proof if other witnesses have attested to the action_mroot being correct.

As part of the unit tests, a validator that can parse the compressed command stream is implemented via abieos as an example of a reasonably efficient implementation that could work in a smart contract.

Consensus Changes

• Consensus Changes

API Changes

• API Changes

Documentation Additions

• Documentation Additions

[swatanabe-b1]

Do we always need to wait for irreversibility or can we just follow the database read mode and let the user decide?

[spoonincode]

We don't here and neither does the witness plugin either. Is relying on the DB read mode really a good indication vs a separate config item though?

[swatanabe-b1]

The witness plugin is somewhat different. Changing the point where it signs changes the meaning of the signature. Signing before a block becomes irreversible is potentially dangerous.

[swatanabe-b1]

For the compressed proof plugin, decoupling the proof generation from on_accepted_block requires you to maintain significant extra state. I don't thing it's worthwhile to make it configurable separately from the database read mode.

[spoonincode]

This branch was rebased to reliable_amqp_publisher-dev and force updated. There were no changes to any commit during the rebase.

[swatanabe-b1]

It looks like this needs to be rebased.

[spoonincode]

yes, was debating doing that before or after merging #9183. I can do it now

[spoonincode]

Branch has been rebased & force pushed. Not something we typically like to do but the base branch had been rebased to completely eliminate some commits that may have muddled boxed vs develop differences. No changes to previous commits before this comment were made but one commit was dropped because it was completely empty now. (github base switched back and forth to update github display)