Environment variables in configs do not resolve anymore

[pichlermarc]

Relevant telegraf.conf

[global_tags]
[agent]
  interval = "${MY_TELEGRAF_INTERVAL}"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  hostname = ""
  omit_hostname = false
[[inputs.mem]]
[[outputs.file]]
  files = ["stdout"]
  data_format = "influx"

Logs from Telegraf

2023-06-16T09:02:51Z I! Loading config: telegraf.conf
2023-06-16T09:02:51Z E! error loading config file telegraf.conf: error parsing [agent]: line 3: (config.AgentConfig.Interval) time: invalid duration "${MY_TELEGRAF_INTERVAL}"

System info

Telegraf 1.28.0-1cfa48ee (git: master@1cfa48ee), Ubuntu 22.04

Docker

No response

Steps to reproduce

1. Use the config provided above as telegraf.conf
2. Use build from default branch 1.28.0-1cfa48ee (git: master@1cfa48ee)
3. Run telegraf MY_TELEGRAF_INTERVAL=10s ./telegraf --config telegraf.conf

Expected behavior

Telegraf starts up as it did with 1.27.0 and earlier

Actual behavior

Telegraf exits with 2023-06-16T09:02:51Z E! error loading config file telegraf.conf: error parsing [agent]: line 3: (config.AgentConfig.Interval) time: invalid duration "${MY_TELEGRAF_INTERVAL}"

Additional info

We initially discovered this in one of our tests, using an environment variable containing a URL for the Dynatrace output plugin.

[srebhan]

@pichlermarc sorry for this but we had many issues by replacing ${...} patterns as those are also used by different plugins for templating. Therefore, in #13451 we changed the shell-style environment variables to use a double-dollar sign to distinguish them from plugins' template-patterns.

Long story short, you should change interval = "${MY_TELEGRAF_INTERVAL}" to either interval = "$MY_TELEGRAF_INTERVAL" or interval = "$${MY_TELEGRAF_INTERVAL}".

[srebhan]

@pichlermarc can you please test the binary in #13457 once CI finished the tests successfully? Please let me know if this fixes your issue!

[pichlermarc]

@pichlermarc can you please test the binary in #13457 once CI finished the tests successfully? Please let me know if this fixes your issue!

That one works for our specific case; thank you @srebhan. 🙂

For us it's just a test that broke, but I was mostly worried that changes to the env var behavior could cause broken deployments for some people, which may have come unexpectedly in a semver minor bump. :)

[srebhan]

@pichlermarc yeah the behavior change was unintended and just slipped through... :-( Sorry for the inconvenience this may have caused!