Allow custom network policies for Grafana, Prometheus, and AlertManager

[nfoucha]

Is your feature request related to a problem? Please describe.

When deploying a uds package that requires connectivity to/from the other components in the monitoring stack (e.g. Thanos) there currently isn't a way to supply custom network policies for these components. This blocks required communication channels and prevents the packaged application from working correctly.

Describe the solution you'd like

• Given a package that requires network connectivity to/from monitoring stack components
• When an override for uds-core includes custom network policies for the relevant components
• Then the custom network policies are deployed in the appropriate namespaces and managed as part of the uds-core deployment

Describe alternatives you've considered

Patching the kubernetes cluster to add/modify/remove network policies after the deployment is complete.

Additional context

In the specific example use-case, Thanos acts as long-term storage for the Prometheus stack components so it needs to scrape prometheus (which is possible currently with the overly permissive network policies). Thanos also has it's own set of alerts and a rules engine that integrates with AlertManager so it needs bi-directional connectivity. Lastly, Thanos can act as a (Prometheus) datasource in Grafana, so Grafana needs an egress policy to communicate with Thanos.

[mjnagel]

I think custom netpols could make sense here but also would be good to try and find some standard things we could directly support to make the config even easier. Here's a few ideas:

• Grafana
  • "Extra datasources" values to enable connection to in-cluster things, CIDRs, or anywhere (this came up recently with something like a cloudwatch datasource
  • Maybe a specific toggle for thanos if we want to directly support that one
• Prometheus
  • Thanos toggle to enable a netpol with the right values for access to "uds-package-thanos"
  • Maybe some custom policy access
• Alertmanager
  • Definitely need a customized egress policy that allows egress to send alerts somewhere
  • Potentially a specific egress policy for mattermost?
  • Custom ingress policy to support "additional alert sources"

TLDR is I would much rather try to identify specific usecases that we can turn into a simple toggle - like "I have thanos" - versus someone needing to specify a big block of YAML and hope they don't make it overly permissive.