diagnose: display detailed transaction time

[cfzjywxk]

Enhancement

In the performance overview panel, the color tuning method is introduced, the idea is that if the overall database time could be broken down as clearly as possible then it's straightforward to tell what's the main bottleneck and what should be done to optimize the performance.

From the perspective of OLTP performance tuning, the most common entry is the throughput and latency of transactions. Theoretically, there could be formulas that

database time = client numbers * (txn duration + out-txn client idle time)
txn duration = stmt duration * stmt number + in-txn client idle time

From this, we could add a color-based database time graph to display the combinations of user database time (note it's quite important to distinguish the user-related requests and tasks and internal background jobs and tasks otherwise there could be much noise )from the perspective of transaction duration including both the total statistics and average statistics top-down. For example, the time details could be displayed like this:

• database time, example question: How much time the TiDB is busy working processing user requests, and how much time is spent waiting for anything?
  • transaction time, example question: How does the client out-txn idle affect the throughput or tps?
    • statement time, example question: How does the client in-txn idle affect the throughput or tps?
      • build and execution time, example question: What's the current bottleneck, the tidb-server or tikv-server? What's the impact of the tail latency on the throughput or tps, could it be quantified?
        • if the current bottleneck is tidb, could the impact be quantified, and what should be done to get improvements?
        • if the current bottleneck is tikv, could the impact be quantified, and what should be done to get improvements?

With this, it could be much easier to compare two benchmark results and to verify what's the performance difference and why is that.

[cfzjywxk]

/cc @zyguan