[Core feature] serve k8s controller-runtime manager metrics in flytepropeller

[hamersaw]

Motivation: Why do you think this is important?

Currently FlytePropeller uses a k8s controller-runtime manager to aid in k8s informers, caches, etc. This tool includes a metrics endpoint which serves prometheus metrics. In this PR that metrics endpoint was disabled to enable Flyte deployments in environments where host networking is required on webhooks.

Goal: What should the final outcome look like, ideally?

FlytePropeller already includes a prometheus endpoint to serve Flyte specific prometheus metrics. We can include the registry from the k8s controller-runtime manager in this endpoint and serve them alongside the Flyte metrics.

Describe alternatives you've considered

Leaving this as is means that these verbose metrics are unavailable.

Propose: Link/Inline OR Additional context

This is where the k8s controller-runtime manager uses the metrics listener to serve prometheus metrics - https://github.com/kubernetes-sigs/controller-runtime/blob/8ad090e0663e046c5a9dad68a572af0e30626c85/pkg/manager/internal.go#L325-L338

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes