Wrapping k8s client with write filter and cache reader

[hamersaw]

Tracking issue

fixes #4730

Why are the changes needed?

The previous attempt (#4733) mistakenly put he cache lookup into a loop until it was populated. This caused large CPU spikes.

What changes were proposed in this pull request?

This PR bypasses the k8sClient internal caching logic and instead just wraps an uncached k8s client with a cache lookup for writes and a write through filter on create / delete operations.

How was this patch tested?

local

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

NA

Docs link

NA

[codecov]

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (ca73251) 58.18% compared to head (44ec90d) 58.19%.
Report is 2 commits behind head on master.

Files	Patch %	Lines
flytepropeller/pkg/controller/executors/kube.go	56.41%	17 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4752   +/-   ##
=======================================
  Coverage   58.18%   58.19%           
=======================================
  Files         626      626           
  Lines       53833    53826    -7     
=======================================
  Hits        31322    31322           
+ Misses      20003    19996    -7     
  Partials     2508     2508           

Flag	Coverage Δ
unittests	58.19% <56.41%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[honnix]

This looks good to me. I was struggling with unit test for this and I guess you might be the same. Not sure how to properly test it without some refactoring in place.