feat(config): Support shell like syntax for environment variable substitution

[neelayu]

Required for all PRs

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

resolves #10557

Support for shell like environment variable expansion. This PR uses a library used by docker-compose to expand variables from the environment. It was a conscious choice to use a library and not build on existing regex patterns. The library gives more flexibility and is actively maintained.

It builds on #10726 and removes the comments before it is passed on to the toml parser. Most cases are handled in the code. Let me know if I missed any. The reason to remove comments before env substitution was due to the fact the default config generated includes the text

# Environment variables can be used anywhere in this config file, simply surround
# them with ${}. For strings the variable must be within quotes (ie, "${STR_VAR}"),
# for numbers and booleans they should be plain (ie, ${INT_VAR}, ${BOOL_VAR})

More specifically the syntax ${} throws an error. So for backwards compatibility, the choice was made to remove the comments. The code to remove comments handles cases where # can be part of string values.

Kindly review. Thanks

[powersj]

Hi,

Thanks for putting this up, this will be a nice addition. We will obviously require a bit of testing to ensure we really do not regress anyone's config as they are today.

I would like the --env-file removed from this PR. That should be a second PR where we can discuss the reasoning and justification for another config file.

I have some initial comments and observations.

Thanks!

[powersj]

I'm catching up on stuff post-vacation. I will take a look at this early next week after we do v1.26.3. I want to play with it a bit more.

[powersj]

Love it! When using env variables with bool and numeric you get TOML validation errors, which is unfortunate, but understandable and that happens today anyway as well.

Went through and played with boolean for debug value and for hostname via string. Tried a few places to toss variables in as comments with no negative effect.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[neelayu]

Thanks for reviewing.
Can you share your config containing bool and numeric that you were testing? Just want to make sure, I didn't miss any case!

[srebhan]

Nice. Thanks for the cool feature @neelayu!

[powersj]

Can you share your config containing bool and numeric that you were testing? Just want to make sure, I didn't miss any case!

First, I should clarify. Telegraf does not complain about any validation errors, but my IDE will complain about invalid toml when you say debug = ${TELEGRAF_DEBUG}. I was setting the agent-level debug and metric_batch_size values.