Extend JSON API metrics

[cocreature]

copying my comment so it doesn’t get lost

• Concurrent requests: This should probably be split into 3 or maybe 4 metrics: concurrent command submissions, concurrent queries, concurrent websocket connections and maybe one for party & package allocation.
• Total duration of requests. This is what you already added. I think we should at least split that in command submissions and queries and lastly party/package management. Not quite sure what we actually measure for websocket connections in this PR. > The total duration there doesn’t seem all that interesting given that they run forever until the client closes the connection.
• More fine grained durations. A few examples:
  • Validation & conversion of the incoming JSON values.
  • For command submissions the duration of the request to the ledger.
  • Conversion on the outgoing side.
  • Database operations at least split into reads & writes but probably more fine-grained is helpful.

This is not intended to be an exhaustive list and I expect we’ll expand it over time but I think with this we should have a good starting point.

I recommend to not tackle everything in one PR and instead tackle them separately (Some of them might be checked already, haven’t confirmed).

https://github.com/digital-asset/daml/pull/9888#pullrequestreview-674921466
<!--

This issue tracker is only for technical issues related to the Daml SDK.

If you suspect that your issue has a security impact, do not use this issue tracker. In that case, please contact [email protected] and follow our responsible disclosure policy: https://digitalasset.com/security/ .

For general questions about Daml and/or support requests you should use our forum: https://discuss.daml.com .

-->

[realvictorprm]

I split this up into seperate issues to also refine specific things if necessary and also to track the seperate effort. Always good to have an issue to close with a PR 😄

[realvictorprm]

Conversion on the outgoing side.
With this I assume the time spend on constructing the response payload?