diagnose: try to differentiate between internal and external requests

[cfzjywxk]

Enhancement

Currently the tidb-server could trigger several different types of internal requests, they could be triggered by some background tasks such as auto analyze, statistics feedback, ddl backfill, gc resolve lock, gc unsafe destroy range, etc. Also, the entries are different some of them may use ExecRestrictedSQL and some may use raw transaction interfaces. These requests would be turned into kv requests just like the user kv requests, but there's no way to differentiate them conveniently.

As these internal requests could use resources and affect the performance when the resources are limited on the cloud, it's difficult to do the performance tuning if the current bottleneck is introduced by some ongoing background tasks. To make the performance diagnosis easier, we could add separate tags for these internal kv requests and try to display how many resources are used by them so that it could be much easier to tell if the internal tasks affect the cluster performance a lot.

The detailed work plan is:

• Add a flag for the internal request in the RPC context of kvproto.
• Use this flag in requests generated by all the internal SQL execution or raw transaction command execution.
• Add or adjust corresponding monitor metrics in both tidb and tikv panels. For the tikv panel, we could distinguish the resources(the thread worker, read/write flow) used by internal requests and their impact on performance.

Note this is just to enhance the path from tidb to tikv, there're still some background tasks like peer move, data compaction in the tikv side, these internal tasks could be discussed in another document.

[cfzjywxk]

For the internal request tracking enhancement, I've created a tracking issue here.
/cc @you06 @sticnarf @youjiali1995

[BornChanger]

Please be kindly reminded, there is a related pr. Make sure we are on the same page and no dup work.

[sticnarf]

Please be kindly reminded, there is a related pr. Make sure we are on the same page and no dup work.

Thanks for the reminder.

This issue is much more systematic work, trying to labeling all requests with their sources. Maybe we will reimplement #34861 with more unified mechanisms.

[you06]

Maybe we will reimplement #34861 with more unified mechanisms.

#34861 pushed InRestrictedSQL flag into DAG requests only, we can move such flag into the KV context and separate request_source into request_source and request_scope, but it makes other TiKV user confused because they don't have the knowledge about "internal" requests.