Feature flag for use of committing client in cometbft

[JimLarson]

What is the Problem Being Solved?

Our chain spends more time executing blocks than other chains do. We suffered from reduced RPC throughput. As a solution, we forked tendermint v0.34.12 and added a CommittingClient type of connection which used read-write locks for concurrency control instead of the pure mutex locks of the LocalClient. Commit.

However, the committing client is a dangerous feature, and a suspect in an ongoing production incident. For this incident and future ones, we'd like an easy way to disable committing client. This is not a consensus-breaking change, so a feature flag seems like the right approach.

Description of the Design

In cosmos-sdk/server/start.go we have the agd server subcommand and the call to construct the committing client. This is the nexus to add a flag and switch on it to perhaps use a local client instead. Using a boolean brings up arguments of which orientation to use ("--dont-not-disable-feature=false"), whereas --consensus-client-type={local,committing} (defaulting to committing) is more self-documenting and future-proof.

As we're targeting a patch release (or just pointing validators to a commit hash), we'll land this change on the Agoric branch of agoric-labs/cosmos-sdk, cherry-pick it onto Agoric-v0.46.16.2.x, tag v0.46.16-alpha.agoric.2.2, then make a version of agoric-sdk using that tag.

Security Considerations

N/A

Scaling Considerations

Local client was originally much slower than committing client, but we have not run performance tests recently.

If local client is still notably slower, operators may need to adjust provisioning accordingly, or segregate certain kinds of traffic.

Test Plan

Unit tests bring it to parity with other tests. Manual testing confirmed.

Upgrade Considerations

N/A

[JimLarson]

It's been noted that having a new cosmos-sdk tag will change error stack traces. If a swingset call into Cosmos gets and error that returns a stack trace, and if that stack trace is eventually put into consensus state, then this becomes a potential consensus-breaking change.

[JimLarson]

Note that without further ceremony, the flag can be set by adding the following line to the top-level section in config/app.toml:

abci-client-type = "committing"