Switch to runtime configuration YAML

[tomkerkhove]

As of today, all the runtime configuration is done via environment variables which is fine if you have a few of them but it feels like the list is becoming too long.

We should take a step back and see if we should go to a configuration model where we define them in YAML as well and load them at runtime, similar to how Prometheus and other tools approach this.

Benefits:

• Everything in one file which can be loaded via a volume (Docker) or config map (Kubernetes)
• Configuration can be added to source control
• Easier to read and get an idea what it is configured like
• Centralized configuration

Downsides:

• Harder to get started as you need to know how volumes/config maps work
  • I think we can provide the required guidance around this and/or provide default config, either in the container or docs, to use
• Existing customers need to migrate to the new model

The (de)serialization should use the out-of-the-box capability, and not built our own as we do for the metrics, given it's pretty straight forward.

Checklist

• Move all configuration to the structure below, except for auth information
• Update all documentation on this and use an example. Location should be /config/runtime.yaml
• Provide unit testing, where needed.
• Update Docker Hub bot.
• Ensure optional variables are working fine
• Update Helm Chart
• Change landing page to point to v1.0

Configuration Example

This configuration should include everything from our configuration docs except for authentication config which would still be via environment variables.

server:
  httpPort: 80
prometheus:
  scrapeEndpoint:
    baseUriPath: /prometheus/scrape
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml
logging:
  applicationInsights:
    instrumentationKey: ABC
    isEnabled: true
  containerLogs:
    isEnabled: true
    verbosity: warning
featureFlags:
  metrics:
    disableTimestamps: true # false by default

[tomkerkhove]

Adding @samneirinck as he might have some good feedback/insights as well

[tomkerkhove]

The above YAML is the way to go forward with v1.0.0.

[tomkerkhove]

@jsturtevant @brusMX This should be ready to be implemented and use the YAML mentioned above using the out-of-the-box deserialization.

Feel free to take this up this week, but it's not a must

[samneirinck]

What will be the plan to support deploying to multiple environments? Is the way forward to create 1 file per environment, or still have environment variables to be able to override certain parts? Or have some kind of placeholder support in this yaml file?

[tomkerkhove]

The environment variables will go away but if you're using the helm chart you should be able to let that create the file for you if you want and use different parameters based on the env. Makes sense?

[samneirinck]

Makes sense, but makes it a bit harder to use without helm.

[tomkerkhove]

I think it should be ok, which configurations do you think will always defer depending on the environment?

[samneirinck]

AI instrumentation key mainly

[tomkerkhove]

I get your point. Would it make sense if this is moved to environment variable but still ahs to be enabled here?

Plan is to add more sinks later on which are configured here but instrumentation key is a secret so should not be here indeed.

[tomkerkhove]

@datadot @ResDiaryLewis @adam-resdiary @marcinbudny @barakAtSoluto @jdevalk2 Feel free to have a look at this issue and provide feedback.

Personally, I think this is the best approach to go forward. If you are using Helm chart for deployment, this should be handled for you.

[ResDiaryLewis]

Hey Tom, this is definitely preferable for us 🙆‍♂. It's just a bit easier to view the config file than to print out each environment variable, it's also easier for us to just plug the YAML file in from our Helm values file directly.

[jdevalk2]

I like the idea of the config file but consider being able to override every setting with an environment variable as well. Example: How Grafana does things.

Ex.

[server]
name=bla

Can be overridden with environment variable GF_SERVER_NAME=bla2

Also consider not making the config file more than 2 levels deep, and prefer 1 level deep, keeping things at the root. I say this because tools like Helm mainly do not allow deep-merge. You can see this in prometheus operator chart where if you want to override a setting 3 levels deep you have to specify the entire subtree from the root downwards.

Also consideration is to be able to mount passwords on filesystem. If there are any passwords in the configuration then ideally those can be mounted through a mounted file secret (instead of over an environment variable). I.e. the secret is mounted on a folder and is contained in a file in that folder. The idea here is that an environment / environment variables are relatively easy to dump when software get compromised and if that dump contains your database credentials that could be used to escalate any attack that was going on.

[datadot]

@tomkerkhove sounds reasonable to me.
I'd agree with @jdevalk2 idea too with the environment variable override, however feel this could be a stretch goal for after v1.0.0.

[tomkerkhove]

I think we have a consensus that it's a good approach but overriding via environment variables are a good idea. Thanks for the input folks!

Overriding seems to be more of enhancement rather than a requirement for v1.0 so I tend to agree with @datadot. This improvement is tracked in #594.

Also consider not making the config file more than 2 levels deep, and prefer 1 level deep, keeping things at the root. I say this because tools like Helm mainly do not allow deep-merge. You can see this in prometheus operator chart where if you want to override a setting 3 levels deep you have to specify the entire subtree from the root downwards.

That's good feedback, thanks - I was not aware of that!

Also consideration is to be able to mount passwords on filesystem. If there are any passwords in the configuration then ideally those can be mounted through a mounted file secret (instead of over an environment variable). I.e. the secret is mounted on a folder and is contained in a file in that folder. The idea here is that an environment / environment variables are relatively easy to dump when software get compromised and if that dump contains your database credentials that could be used to escalate any attack that was going on.

Another good point, tracking this in #593

[tomkerkhove]

This should be done, only have to change docs to new version when shipping v1.0

[tomkerkhove]

PR for docs is waiting to be merged!

Closing this one already.