feat: add ability to add custom netpols for prometheus-stack package

[joelmccoy]

Description

Adds a value to allow custom netpols for the prometheus-stack package. Following a similar pattern that we use for vector. Additionally added some documentation on how to override and add custom netpols to prometheus stack and vector.

Related Issue

Fixes #996

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Other (security config, docs update, etc)

Checklist before merging

• Test, docs, adr added or updated as needed
• Contributor Guide followed

[noahpb]

Based on the discussion here, I'm not sure if this is the preferred approach that we'd like to take for creating additional Network Policies - we have some more design to do here. Given that we currently don't have a solution, I think this is a reasonable approach that can work in the interim. cc @mjnagel and @UnicornChance.

[joelmccoy]

@noahpb Good callout. Missed that issue. It sounds like the preferred approach would be to minimize the customizability of these netpols. The main use case I am trying to support is to allow alertmanager to egress to some place. This place is going to be different for each customer/environment. Would it be preferred to have a value that is more specific to where alertmanager can egress to?

A simple value like alertmanagerEgress: [] which provides a list of ip ranges / netpol targets that it can go to?

[mjnagel]

@noahpb / @joelmccoy yeah my main thoughts with that mention on the other issue are that we should try to capture specific use cases where we can. I don't think we necessarily should lock down all customizability, but where there are known needs to support it would be great to explicitly have a rule for those.

A specific value for alertmanager egress might make sense in this case, with conditionals for inCluster, cidr, or fallback for anywhere (in the future ideal state we could support specific hosts, i.e. slack.com or something). I also know that linked issue called out a few other needs for Thanos support and other alert sources so I could also see an argument to allow any custom policies.

Two other comments on this:

• It might be good to try and document this somewhere that surfaces on the docs site, like a "networking configuration" page that gives an example + shows the value to override.
• I don't like the custom value name personally. I know this is a pattern emerging with SWF and other apps but without looking at the template/having the tribal knowledge of what that does the name custom means nothing. I think we should be a lot more explicit with the name, i.e. customNetworking or additionalNetworkAllow which more directly connect to the field in the Package CR.

[joelmccoy]

Good feedback and clarity. Do we have a preference on how flexible we want this too be? For the alertmanager use case, I am thinking of two routes we could take at this point:

1. Add a field like customNetworking or additionalNetworkAllow and then give access to the end user to configure the network properties directly?
   or..
2. Do we take a less flexible approach and add a field called something like alertmanagerEgressAllow which is only allowed to customize where alertmanager can send alerts.

Happy to document example overrides in the docs for whatever route we prefer

[mjnagel]

@joelmccoy I think option 1 for now. This provides the flexibility that people need and supports unknown end user needs. I think if/when we identify specific possible network targets we should add conditional templates for those specific pieces, but will likely still want this block to support other unknown needs.

[joelmccoy]

Based on feedback added in an additionalNetworkAllow value for the prometheus stack. This is the same pattern currently used for vector. Additionally added some documentation to demonstrate how to add additional network allowances for both Prometheus Stack and Vector.

[noahpb]

LGTM - Thanks for adding a doc! :)

[mjnagel]

Will rebase and merge once #1020 merges.