Provisioning: Interpolate env vars in provisioning files #16499

aocenas · 2019-04-10T20:47:53Z

Allows $ENV_VAR or ${ENV_VAR} syntax in provisioning files. Uses https://golang.org/pkg/os/#Expand for the interpolation so does not allow any other syntax at the moment.

The interpolation is implemented as special values that can be used in the struct into which the data should be unmarshalled. Example:

type Data struct {
  Field StringValue `yaml:"field"` // StringValue instead of string
}

This allows better flexibility and can be reused, while specifying custom unmarshaling logic for each config object would have been tedious especially as you would have to reimplement lots of mapping logic that yaml lib gives you for free with this approach.

Other approach considered would be to do the interpolation directly on yaml string when read from file and only afterwards unmarshal it. This would allow using env vars also for field names or even interpolating a bigger pieces of the document which we did not want to allow.

TODO:

notifier configs?
~~env vars in dashboard json files?~~ not doing in this PR
check some advanced yaml syntax and add tests for it.
some manual testing for the notifications

aocenas · 2019-04-10T20:49:31Z

pkg/services/provisioning/values/values_test.go

+
+			Convey("Should unmarshal variable nesting", func() {
+				doc := `
+                 val:


Yaml does not seem to allow tabs and so these must be spaces which then creates odd indent.

marefr · 2019-04-11T08:44:31Z

This looks like an interesting approach and a good start. Great that you've opened a PR early.

Think @bergquist is more suitable reviewing this though given that he has built the provisioning feature.

aocenas · 2019-04-11T11:31:06Z

pkg/services/provisioning/notifiers/types.go

@@ -1,30 +1,61 @@
 package notifiers

-import "github.com/grafana/grafana/pkg/components/simplejson"


Small refactor with the types here to align it with other provisioning types and makes the change to Value much more contained. Basically we have ConfigVx type that can be versioned and yaml file is unmarshalled into it. Afterwards there is Config type that all ConfigVx are mapped into which separates the multiple possible config versions from internal representation. This also means that changes to the config unmarshalling are only done here and does not permeate other parts of the code that needs to work with the data.

aocenas · 2019-04-11T12:14:47Z

@bergquist This should work right now. Not sure about the dashboards json files whether they should have interpolation or not, but even they do we can do it in separate PR.

bergquist

Great work! Could you please update the docs in the PR as well?

pkg/services/provisioning/values/values_test.go

pkg/services/provisioning/values/values.go

aocenas · 2019-04-15T10:28:34Z

docs/sources/administration/provisioning.md

@@ -30,33 +30,19 @@ Checkout the [configuration](/installation/configuration) page for more informat

 ### Using Environment Variables

-All options in the configuration file (listed below) can be overridden
-using environment variables using the syntax:
+It is possible to use environment variable interpolation in all 3 provisioning config types. Allowed syntax


@bergquist Added the documentation here. I removed the old text because it was talking about using env vars for main configuration.ini and was duplicate from installation/configuration.md so it seemed out of place and confusing here.

bergquist

Looks good. I think it would make sense to add tests for this in https://github.com/grafana/grafana/blob/master/pkg/services/provisioning/dashboards/config_reader_test.go#L16 as well. What do you think?

aocenas · 2019-04-17T15:03:41Z

@bergquist I added some env vars into the test configs kinda randomly. I think that would be enough right? Or do you think a separate test case for each would be better?

bergquist

Great work!

I think this is ready to be merged :)

aocenas · 2019-04-18T08:02:06Z

@bergquist Will wait a bit with merging this for #16559 as it should be easier to solve the merge conflicts that way.

aocenas · 2019-04-24T12:56:05Z

@bergquist There are some small changes after merge with master. I added Raw attribute to values that holds the uninterpolated value for a better warning here.

bergquist

Raw makes total sense. Great work :)

…into streaming-test-data-alt * 'streaming-test-data-alt' of github.com:ryantxu/grafana: PanelChrome: delay state update while other panel is in fullscreen/edit TestData: don't reuse unsubbed stream workers PanelQueryState: give code a bit of air (space) PanelQuery: Minor refactoring sqlstore: use column name in order by (grafana#16583) user friendly guide (grafana#16743) Provisioning: Interpolate env vars in provisioning files (grafana#16499) admin: add more stats about roles (grafana#16667) Feature: Migrate Legend components to grafana/ui (grafana#16468) playlist: fix loading dashboards by tag (grafana#16727) CloudWatch: Use default alias if there is no alias for metrics (grafana#16732) Provisioning: Support FolderUid in Dashboard Provisioning Config (grafana#16559)

* grafana/master: GraphLegendEditor: use stats picker rather than switches (grafana#16759) Feature: add cron setting for the ldap settings (grafana#16673) Build: Disables gosec until identified performance problems (grafana#16764) Chore: bump jQuery to 3.4.0 including prototype pollution vulnerability fix (grafana#16761) elasticsearch: add 7.x version support (grafana#16646) Provisioning: Add API endpoint to reload provisioning configs (grafana#16579) Config: Show user-friendly error message instead of stack trace (grafana#16564) Chore: Lowered implicit anys limit to 5954 Feature: Enable React based options editors for Datasource plugins (grafana#16748) sqlstore: use column name in order by (grafana#16583) user friendly guide (grafana#16743) Provisioning: Interpolate env vars in provisioning files (grafana#16499) admin: add more stats about roles (grafana#16667) Feature: Migrate Legend components to grafana/ui (grafana#16468) playlist: fix loading dashboards by tag (grafana#16727) CloudWatch: Use default alias if there is no alias for metrics (grafana#16732) Provisioning: Support FolderUid in Dashboard Provisioning Config (grafana#16559)

…MetricPanelCtrl * grafana/master: (34 commits) Streaming: support streaming and a javascript test datasource (grafana#16729) GraphLegendEditor: use stats picker rather than switches (grafana#16759) Feature: add cron setting for the ldap settings (grafana#16673) Build: Disables gosec until identified performance problems (grafana#16764) Chore: bump jQuery to 3.4.0 including prototype pollution vulnerability fix (grafana#16761) elasticsearch: add 7.x version support (grafana#16646) Provisioning: Add API endpoint to reload provisioning configs (grafana#16579) Config: Show user-friendly error message instead of stack trace (grafana#16564) Chore: Lowered implicit anys limit to 5954 Feature: Enable React based options editors for Datasource plugins (grafana#16748) sqlstore: use column name in order by (grafana#16583) user friendly guide (grafana#16743) Provisioning: Interpolate env vars in provisioning files (grafana#16499) admin: add more stats about roles (grafana#16667) Feature: Migrate Legend components to grafana/ui (grafana#16468) playlist: fix loading dashboards by tag (grafana#16727) CloudWatch: Use default alias if there is no alias for metrics (grafana#16732) Provisioning: Support FolderUid in Dashboard Provisioning Config (grafana#16559) Refactor: Make SelectOptionItem a generic type to enable select value typing (grafana#16718) docs: fix upgrade instructions ...

* grafana/master: Streaming: support streaming and a javascript test datasource (grafana#16729) GraphLegendEditor: use stats picker rather than switches (grafana#16759) Feature: add cron setting for the ldap settings (grafana#16673) Build: Disables gosec until identified performance problems (grafana#16764) Chore: bump jQuery to 3.4.0 including prototype pollution vulnerability fix (grafana#16761) elasticsearch: add 7.x version support (grafana#16646) Provisioning: Add API endpoint to reload provisioning configs (grafana#16579) Config: Show user-friendly error message instead of stack trace (grafana#16564) Chore: Lowered implicit anys limit to 5954 Feature: Enable React based options editors for Datasource plugins (grafana#16748) sqlstore: use column name in order by (grafana#16583) user friendly guide (grafana#16743) Provisioning: Interpolate env vars in provisioning files (grafana#16499) admin: add more stats about roles (grafana#16667) Feature: Migrate Legend components to grafana/ui (grafana#16468) playlist: fix loading dashboards by tag (grafana#16727) CloudWatch: Use default alias if there is no alias for metrics (grafana#16732) Provisioning: Support FolderUid in Dashboard Provisioning Config (grafana#16559)

* grafana/master: (31 commits) Streaming: support streaming and a javascript test datasource (grafana#16729) GraphLegendEditor: use stats picker rather than switches (grafana#16759) Feature: add cron setting for the ldap settings (grafana#16673) Build: Disables gosec until identified performance problems (grafana#16764) Chore: bump jQuery to 3.4.0 including prototype pollution vulnerability fix (grafana#16761) elasticsearch: add 7.x version support (grafana#16646) Provisioning: Add API endpoint to reload provisioning configs (grafana#16579) Config: Show user-friendly error message instead of stack trace (grafana#16564) Chore: Lowered implicit anys limit to 5954 Feature: Enable React based options editors for Datasource plugins (grafana#16748) sqlstore: use column name in order by (grafana#16583) user friendly guide (grafana#16743) Provisioning: Interpolate env vars in provisioning files (grafana#16499) admin: add more stats about roles (grafana#16667) Feature: Migrate Legend components to grafana/ui (grafana#16468) playlist: fix loading dashboards by tag (grafana#16727) CloudWatch: Use default alias if there is no alias for metrics (grafana#16732) Provisioning: Support FolderUid in Dashboard Provisioning Config (grafana#16559) Refactor: Make SelectOptionItem a generic type to enable select value typing (grafana#16718) docs: fix upgrade instructions ...

Add value types with custom unmarshalling logic

4d71b76

aocenas commented Apr 10, 2019

View reviewed changes

marefr requested a review from bergquist April 11, 2019 08:44

aocenas added 4 commits April 11, 2019 11:50

Add env support for notifications config

a60ecfb

Use env vars in json data tests for values

6c39553

Add some more complexities to value tests

d158a19

Update comment with example usage

bbd3517

aocenas commented Apr 11, 2019

View reviewed changes

aocenas marked this pull request as ready for review April 11, 2019 12:13

aocenas added add to changelog area/backend area/provisioning labels Apr 11, 2019

bergquist reviewed Apr 11, 2019

View reviewed changes

pkg/services/provisioning/values/values_test.go Show resolved Hide resolved

pkg/services/provisioning/values/values.go Outdated Show resolved Hide resolved

bergquist mentioned this pull request Apr 11, 2019

Add support for reading data source provisioner secrets from files #16315

Closed

aocenas added 2 commits April 12, 2019 10:22

Set env directly in the tests, removing patching

07eef00

Update documentation

53f1104

aocenas commented Apr 15, 2019

View reviewed changes

bergquist reviewed Apr 16, 2019

View reviewed changes

bergquist added this to the 6.2 milestone Apr 16, 2019

Add env var to the file reader tests

6cb3d64

bergquist approved these changes Apr 18, 2019

View reviewed changes

aocenas mentioned this pull request Apr 23, 2019

Provisioning: Support FolderUid in Dashboard Provisioning Config #16559

Merged

4 tasks

aocenas added 4 commits April 24, 2019 11:50

Add raw value

1d2c8bb

Merge branch 'master' into env-vars-provisioning

f694595

Post merge fixes

9c5af53

Add comment

950e015

bergquist approved these changes Apr 24, 2019

View reviewed changes

aocenas merged commit fcebd71 into master Apr 24, 2019

aocenas deleted the env-vars-provisioning branch April 24, 2019 13:39

marefr mentioned this pull request Jul 9, 2019

Provisioned values containing $ character is replaced with empty string #17986

Closed

dpajin mentioned this pull request Feb 28, 2020

Provisioning: Support integer env variables in yaml files #20347

Open

mieseprem mentioned this pull request Jun 8, 2020

Dashboard provisioning using environment variables is not working #25443

Closed

senkevinli mentioned this pull request Nov 20, 2020

Logging service uavaustin/orchestra#52

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provisioning: Interpolate env vars in provisioning files #16499

Provisioning: Interpolate env vars in provisioning files #16499

aocenas commented Apr 10, 2019 •

edited

Loading

aocenas Apr 10, 2019

marefr commented Apr 11, 2019

aocenas Apr 11, 2019

aocenas commented Apr 11, 2019

bergquist left a comment

aocenas Apr 15, 2019

bergquist left a comment •

edited

Loading

aocenas commented Apr 17, 2019

bergquist left a comment

aocenas commented Apr 18, 2019

aocenas commented Apr 24, 2019

bergquist left a comment

		@@ -1,30 +1,61 @@
		package notifiers

		import "github.com/grafana/grafana/pkg/components/simplejson"

Provisioning: Interpolate env vars in provisioning files #16499

Provisioning: Interpolate env vars in provisioning files #16499

Conversation

aocenas commented Apr 10, 2019 • edited Loading

aocenas Apr 10, 2019

Choose a reason for hiding this comment

marefr commented Apr 11, 2019

aocenas Apr 11, 2019

Choose a reason for hiding this comment

aocenas commented Apr 11, 2019

bergquist left a comment

Choose a reason for hiding this comment

aocenas Apr 15, 2019

Choose a reason for hiding this comment

bergquist left a comment • edited Loading

Choose a reason for hiding this comment

aocenas commented Apr 17, 2019

bergquist left a comment

Choose a reason for hiding this comment

aocenas commented Apr 18, 2019

aocenas commented Apr 24, 2019

bergquist left a comment

Choose a reason for hiding this comment

aocenas commented Apr 10, 2019 •

edited

Loading

bergquist left a comment •

edited

Loading