store/tikv: recycle idle connection in tikv client

[tiancaiamao]

What problem does this PR solve?

If a tikv addr has been idle for a while, recycle its connection.

Fix goroutine leak when tikv is offline:

goroutine 4619 [chan receive, 1145 minutes]:
github.com/pingcap/tidb/store/tikv.fetchAllPendingRequests(0xc00245ac60, 0x80, 0xc0026e2e20, 0xc0026e2e08)
/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client.go:339 +0x42
github.com/pingcap/tidb/store/tikv.(*connArray).batchSendLoop(0xc00245acc0, 0x10, 0xa, 0x3, 0xc0000479b8, 0x3, 0x24e, 0x80, 0xc8, 0x0, ...)
/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client.go:434 +0x4db
created by github.com/pingcap/tidb/store/tikv.(*connArray).Init
/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/store/tikv/client.go:293 +0xb05

The leak happens when tikv is offline, and the background batch loop goroutine never exit, even the connection is idle.

What is changed and how it works?

In the batch send loop, add a idle detect timer, if the batch commands channel doesn't receive message,
mark this connArray as idle and notify rpcClient to recycle the connArray.

Check List

Tests

• Integration test

Unit test doesn't use tikv client, so it must be tested in the integration test

Side effects

• Increased code complexity

Related changes

• Need to cherry-pick to the release branch

Should cherry pick to 3.0

[tiancaiamao]

PTAL @hicqu @lysu

[codecov]

Codecov Report

Merging #10616 into master will decrease coverage by 0.0384%.
The diff coverage is 25%.

@@               Coverage Diff                @@
##             master     #10616        +/-   ##
================================================
- Coverage   77.7226%   77.6842%   -0.0385%     
================================================
  Files           413        413                
  Lines         87515      87548        +33     
================================================
- Hits          68019      68011         -8     
- Misses        14351      14383        +32     
- Partials       5145       5154         +9

[codecov]

Codecov Report

Merging #10616 into master will decrease coverage by 0.0182%.
The diff coverage is 25%.

@@               Coverage Diff                @@
##             master     #10616        +/-   ##
================================================
- Coverage   77.7027%   77.6844%   -0.0183%     
================================================
  Files           413        413                
  Lines         87522      87540        +18     
================================================
- Hits          68007      68005         -2     
- Misses        14361      14383        +22     
+ Partials       5154       5152         -2

[tiancaiamao]

/run-all-tests

[hicqu]

LGTM. Thank you!

[disksing]

What if batch message is disabled?

[tiancaiamao]

The recycle logic never run, just as before. @disksing
https://github.com/pingcap/tidb/pull/10616/files#diff-deb6c7bcc2ba593559c11d2e92f8d8dcR621

[disksing]

LGTM