limit contract insertion/deletion batching on backpressure

[S11001001]

#3936 uses a limit on batching websocket output under the assumption that clients would rather download their deltas in bite-sized chunks than very large ones. It was inspired by #3192, which also batches transactions to be added to the query store, albeit with no limit.

While #10353 updates #3192 to support arbitrarily large batches as input to the database, we might as well defensively limit ourselves to the largest batch likely to yield any meaningful RDBMS roundtrip reduction time, using a similar technique to #3936. This will also feed more backpressure to the participant server, possibly reducing its own load in high-load scenarios.

CHANGELOG_BEGIN
- [JSON API] While updating the contract table for a query, if the DB appears to be slow,
  JSON API will slow down its own inserts and deletes at some point rather than construct
  ever-larger INSERT and DELETE batch commands.
  See `issue #11589 <https://github.com/digital-asset/daml/pull/11589>`__.
CHANGELOG_END

Pull Request Checklist

• Read and understand the contribution guidelines
• Include appropriate tests
• Set a descriptive title and thorough description
• Add a reference to the issue this PR will solve, if appropriate
• Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
• Normal production system change, include purpose of change in description
• If you mean to change the status of a component, please make sure you keep the Component Status page up to date.

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

[stefanobaghino-da]

@S11001001 Should this be considered for back-porting to prevent issues for users running older versions?

[stefanobaghino-da]

Should this be configurable, rather than hard-coded?

[S11001001]

No, I don't think so. The benefit comes from setting a [reasonably large] limit, more than what that limit is.

[akshayshirahatti-da]

@S11001001 how did we come up with this maxCost value , also can you remind me of the semantics of delete and why it may entail a table scan ?

[S11001001]

We came up with it by me taking the number 200 from the client output and adding 50 to it. It's a buffer size; it doesn’t really matter what the number is so much as that there is a number and it is reasonably "large" so that the buffer doesn't grow without bound but still gives you good-sized "chunks".

As for the deletes, it depends on how the DB implements delete from contracts where contract_id in (id1, id2, ...).

[S11001001]

Should this be considered for back-porting to prevent issues for users running older versions?

I wrote it with that possibility in mind (else, the conflation function would be in fetch-contracts).

[akshayshirahatti-da]

looks good 👍