Generate config/*/prometheus.yml from template

[stephen-soltesz]

In order to use promtool check-config and to add per-cluster labels to federated deployments, we need the ability to generate configs from a template.

[stephen-soltesz]

The more general problem here is that we are currently maintaining three separate config trees which a high degree of overlap and redunancy for each gcp project.

It should be possible using Helm (https://docs.helm.sh/using_helm/) or a similar template system to externalize the parts that vary across projects. Today that is largely only resource settings or new features that should eventually be pushed forward to production.

[stephen-soltesz]

Helm is not suitable as are many of the k8s template tools currently available. Helm for example, uses a custom template language whose beginning template uses conventions like explicit indent directives:

    spec:
      containers:
        - name: {{ .Chart.Name }}
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- if .Values.nodeSelector }}
      nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
    {{- end }}

IMO, this does not make k8s configurations more manageable.

About two years ago discussion started in the k8s community about natively supporting templating in kubernetes configuration (kubernetes/kubernetes#11492). After two years, the issue appears to have been resolved in this thread (kubernetes/kubernetes#23896 (comment)) where it is left as an exercise to the users to find the right tool for their use case.

https://github.com/kopeio/kexpand is simple enough and uses the familar {{}} substitution syntax that we're already using in templates for scraper, scraper-sync and mlabconfig.py.

[stephen-soltesz]

Complete:

#98
#99
#100
#101