Use child context in watchAgents to avoid goroutine leak

[pingsutw]

Tracking issue

#3936

Why are the changes needed?

Rebuilding gRPC connections in Agents leaks goroutines
As part of the investigation for this issue we discovered Agents is periodically rebuilding gRPC connections and not closing them because the context used is that of the overall flyte binary. This results in very long-lived, unused gRPC connections and a goroutine leak that steadily increases memory utilization until crash.

What changes were proposed in this pull request?

Use child context in the watcher and cancel the child context every time. Canceling this context will also remove the grpc connection here.

How was this patch tested?

flyte start --config flyte-local-test.yaml
wget -O goroutine.out  http://localhost:10254/debug/pprof/goroutine\?debug\=1 && cat goroutine.out | grep "goroutine profile:"

Validated by starting single-binary with agents enabled and querying golang pprof with wget -O goroutine.out http://localhost:10254/debug/pprof/goroutine?debug=1 to view the number of goroutines blocking on the gRPC CallbackSerializer. (the total number of goroutines should not increase)

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

NA

Docs link

NA

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 36.71%. Comparing base (56b6d6d) to head (9877501).
Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
...yteplugins/go/tasks/plugins/webapi/agent/plugin.go	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5888      +/-   ##
==========================================
- Coverage   36.71%   36.71%   -0.01%     
==========================================
  Files        1304     1304              
  Lines      130081   130083       +2     
==========================================
- Hits        47764    47758       -6     
- Misses      78147    78155       +8     
  Partials     4170     4170              

Flag	Coverage Δ
unittests-datacatalog	51.58% <ø> (ø)
unittests-flyteadmin	54.39% <ø> (-0.03%)	⬇️
unittests-flytecopilot	11.73% <ø> (ø)
unittests-flytectl	62.40% <ø> (ø)
unittests-flyteidl	6.89% <ø> (ø)
unittests-flyteplugins	53.61% <0.00%> (-0.01%)	⬇️
unittests-flytepropeller	42.84% <ø> (ø)
unittests-flytestdlib	54.78% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Future-Outlier]

just test it.