fix yurthub memory leak

[JameKeal]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage

/kind bug

What this PR does / why we need it:

When yurthub connect with pool-coordinator etcd failed, and keep try again, the goroutine of grpc is increase all the while, and memory of yurthub also increase, and kubelet will restart it when is happen OOM.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

[openyurt-bot]

@JameKeal: GitHub didn't allow me to assign the following users: your_reviewer.

Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage

/kind bug

What this PR does / why we need it:

When yurthub connect with pool-coordinator etcd failed, and keep try again, the goroutine of grpc is increase all the while, and memory of yurthub also increase, and kubelet will restart it when is happen OOM.

goroutine profile: total 1312
756 @ 0x439896 0x449632 0x1467c65 0x46a9a1
#	0x1467c64	google.golang.org/grpc.(*ccBalancerWrapper).watcher+0xa4	/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:71

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
2.5% Duplication

[codecov]

Codecov Report

Merging #1501 (381b5f6) into master (6c04014) will decrease coverage by 0.02%.
The diff coverage is 35.71%.

@@            Coverage Diff             @@
##           master    #1501      +/-   ##
==========================================
- Coverage   50.74%   50.73%   -0.02%     
==========================================
  Files         131      131              
  Lines       15576    15586      +10     
==========================================
+ Hits         7904     7907       +3     
- Misses       6951     6958       +7     
  Partials      721      721              

Flag	Coverage Δ
unittests	50.73% <35.71%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/yurthub/poolcoordinator/coordinator.go	11.99% <0.00%> (-0.05%)	⬇️
pkg/yurthub/storage/etcd/storage.go	79.71% <0.00%> (-1.18%)	⬇️
pkg/yurthub/storage/etcd/keycache.go	77.87% <100.00%> (+0.28%)	⬆️

[rambohe-ch]

@Congrool PTAL

[Congrool]

/lgtm

Thanks!

[Congrool]

/lgtm

Thanks!

[openyurt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JameKeal, rambohe-ch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rambohe-ch]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rambohe-ch]

/backport release-v1.3

[github-actions]

Successfully created backport PR for release-v1.3:

• [Backport release-v1.3] fix yurthub memory leak #1562