Support for environment variables in provisioning

[xlson]

Original feature request: grafana/grafana-docker#163

Make it possible to reference environment variables in datasource and dashboard provisioning files to make provisioning more dynamic when working with several similar environments.

[xlson]

Not entirely sure if this is a case we should support or not, but it's a recent enough feature request in the grafana-docker repo that I didn't want to just close it without moving it here first. This could make provisioning more dynamic but also potentially more error-prone.

[maxime-jeanson]

+1

[CharlesEdouardCady]

I need this feature too. When using Grafana from Docker, a workaround is to run envsubst on the YAML file in the ENTRYPOINT command, just before running run.sh.

[dtshepherd]

I need this as well so I can inject datasource secrets (the password) from kubernetes environment variable secrets.

I don’t even have a good work-around for the grafana helm chart right now...

[dtshepherd]

@CharlesEdouardCady Did you just build a custom grafana docker image to apply the envsubst change?

[CharlesEdouardCady]

@dtshepherd Yes, because we basically customized the entrypoint to something like:

ENTRYPOINT envsubst < /etc/grafana/provisioning/datasources/all.yml > conf && mv conf /etc/grafana/provisioning/datasources/all.yml \
         && su grafana \
         && sh /run.sh

It works for what we need it for (get database credentials).

Our full Dockerfile looks like this:

FROM debian:stretch-slim

ENV PATH=/usr/share/grafana/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    GF_PATHS_CONFIG="/etc/grafana/grafana.ini" \
    GF_PATHS_DATA="/var/lib/grafana" \
    GF_PATHS_HOME="/usr/share/grafana" \
    GF_PATHS_LOGS="/var/log/grafana" \
    GF_PATHS_PLUGINS="/var/lib/grafana/plugins" \
    GF_PATHS_PROVISIONING="/etc/grafana/provisioning"

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
                    tar \
                    sqlite \
                    libfontconfig \
                    curl \
                    ca-certificates \
                    gettext \
 && mkdir -p "$GF_PATHS_HOME" \
 && curl "https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.1.3.linux-x64.tar.gz " | tar xfvz - --strip-components=1 -C "$GF_PATHS_HOME" \
 && apt-get autoremove -y \
 && rm -rf /var/lib/apt/lists/* \
 && groupadd -r -g 472 grafana \
 && useradd -r -u 472 -g grafana grafana \
 && mkdir -p "$GF_PATHS_PROVISIONING/datasources" \
             "$GF_PATHS_PROVISIONING/dashboards" \
             "$GF_PATHS_LOGS" \
             "$GF_PATHS_PLUGINS" \
             "$GF_PATHS_DATA" \
 && cp "$GF_PATHS_HOME/conf/sample.ini" "$GF_PATHS_CONFIG" \
 && chown -R grafana:grafana "$GF_PATHS_DATA" "$GF_PATHS_LOGS" "$GF_PATHS_PLUGINS" \
 && chmod 777 "$GF_PATHS_DATA" "$GF_PATHS_LOGS" "$GF_PATHS_PLUGINS"

COPY ./run.sh /run.sh

USER grafana

RUN grafana-cli plugins install grafana-worldmap-panel \
 && grafana-cli plugins install natel-plotly-panel\
 && grafana-cli plugins install grafana-piechart-panel\
 && grafana-cli plugins install grafana-simple-json-datasource

ADD provisioning /etc/grafana/provisioning
ADD config.ini $GF_PATHS_CONFIG
ADD dashboards $GF_PATHS_DATA/dashboards

USER root

ENTRYPOINT envsubst < /etc/grafana/provisioning/datasources/all.yml > conf && mv conf /etc/grafana/provisioning/datasources/all.yml \
        && su grafana \
        && sh /run.sh

[a-cordier]

+1

[bergquist]

dupe of #12368 ?

If the purpose is to get the token from k8s I don't think it should be stored as an ENV variable.

[xlson]

@bergquist agreed.

[dtshepherd]

@bergquist @xlson It's very common to get secrets from environment variables in k8s. For example, the grafana chart uses environment variables for secrets and allows you to inject addition secrets through environment variables: https://github.com/helm/charts/blob/master/stable/grafana/templates/deployment.yaml#L162-L200

I'm guessing if #12368 was implemented, we can get by, but it would require more tooling for our secret deployment than using the existing environment variables. It seems these two issues are different feature requests, but maybe could be solved by the same PR...

[bergquist]

Your right. Since ENV var is supported now we should support that in provisioning configs as well.

I would like to add support for reading all passwords/secrets from files and recommend that later on. But that's another issue. Thank you for the feedback.

[jeff-cook]

I'm running into the same issue, where I need to deploy the same Grafana image for different sites. The only difference is the url and password of the provisioned datasource.

Thank you

[Xopherus]

+1 to this. I've been able to get around it by treating dashboards as templates that I can inject variables at build time. ENV vars would be a good improvement.

[tomnason]

Would like this as well.

[hecjhs]

+1

[mrooding]

+1 to this. I'm willing to submit a PR that implements this feature. @bergquist anything you want me to consider before starting on it?

[bergquist]

I was just thinking about this issue on the bus... :)

I dont see any obvious way of solving this issue. Supporting ENV vars in grafana.ini is easier since there can only be one property per grafana instance. Ex http_port can be overridden with GF_HTTP_PORT. For provisioning, however, there can be multiple data sources that uses the same property so GF_DATASOURCE_PASSWORD would not work since it would set the password for all provisioned datasources.

Replace vars when parsing config files using go templating
It would look something like this

datasources:
  - name: gdev-graphite
    type: graphite
    access: proxy
    password: {{NAME_OF_THE_ENV_VAR}}
    url: http://localhost:8080

This however might cause problems if a field contains {{}} so I would like to avoid it since it can cause very frustrating situations for users.

Add support for overriding fields
Ex

datasources:
  - name: gdev-graphite
    type: graphite
    access: proxy
    password: 
    password_from_env: CUSTOM_ENV_VAR
    password_from_file: /var/token 
    url: http://localhost:8080

I think this would be better if we could implement a custom parser. So we can reuse it for all fields.
How ever... In some cases it would be disirable to read the value from file on every request instead when parsing the config file to support rotating tokens. But I guess that only means that those fields should not use the custom parser and store the path to the file/name of the env var instead.

If anyone else have a good idea for how to implement this? Feedback would be very appricated :)

[mrooding]

I may be oversimplifying things but is having rudimentary template parsing using either {{}} or ${NAME_OF_THE_ENV_VAR} such a bad thing? It is common practice in a lot of different tools. Kubernetes being one of them. You could have a fallback to not replacing the notation if the corresponding env var could not be found to solve the issue of frustrating users if they happen to have that specific notation in a value. However, I think most of us are so used to this style of templating that it should not cause major issues.

I fully agree that it should definitely be implemented as a custom parser which is able to scan any and every property for an environment variable. In my use case, I'd definitely also like to inject properties like url and user.

The template parsing solution, however, does not immediately solve reading from a file. I feel it's less desirable to have 1 password property that can either be a value, an environment variable, or a path. To me, it makes the most sense separating this from the default password property. This is assuming that we would not want to support having a path inside an environment variable though. If that was to be supported, we'd have to have environment parsing on every property and as a second step, detect whether the value is a path.

[danielsetreus]

For provisioning, however, there can be multiple data sources that uses the same property so GF_DATASOURCE_PASSWORD would not work since it would set the password for all provisioned datasources

Without knowing everything there is to know about data sources; is it really feasibly that two different sources would want to use the same env vars but with different values? My best example is to provision Prometheus data source. To be the best way to do it would be ${MY_ENV_VAR_FOR_PROMETHEUS_URL} in the yaml file. The scenario here is running the same Grafana image in multiple Kubernetes clusters.

[alexvaut]

+1

[Dan-Dickey]

+1