[regression] v1.27 shell-like expansion replaces non-environment variables #13432

peerfiet · 2023-06-13T09:34:17Z

Relevant telegraf.conf

[[processors.regex]]
order = 50
namepass = ["wallbox_status"]

[[processors.regex.fields]]
    key = "0_chargingRate"
    pattern = "Charging rate: (.*)kw.*\\[(.*)A (.*)pf \\]"
    replacement = "${1}"
    result_key = "charge_kW"

Logs from Telegraf

---

System info

Telegraf 1.27.0

Docker

No response

Steps to reproduce

This is the string used as input to regex plugin: "Charging rate: 0kw 3 phase [0A pf ] "

Expected behavior

field "charge_kW" should contain "0"

Actual behavior

field "charge_kW" contains "1}"

Additional info

changing
replacement = "${1}"
to
replacement = "$1"

makes it work as expected again in Telegraf 1.27. However, the documentation of the regex plugin is suggesting to use "${1}", which was working up to 1.26.x

The text was updated successfully, but these errors were encountered:

neelayu · 2023-06-14T10:27:14Z

It might be due to #13229
Let me take a look

neelayu · 2023-06-14T10:53:06Z

@powersj the docker-compose library doesn't handle the case of ${1}, where the first character after { is a digit.
We have two options here-

Escape the regex with $ making it $${1}. This is a workaround.
Change the documentation for regex processor to use the $1 syntax.

Both of these solutions are somewhat breaking the existing configurations unfortunately. 😞

I checked the codebase and I believe only regex processor has this issue.

If none of them are acceptable, we may need to revert the PR.

powersj · 2023-06-14T13:03:23Z

Thanks for looking into this. I think you are right as we now have 2 more issue reports about this in #13442 and #13438

If none of them are acceptable, we may need to revert the PR.

@srebhan and I will chat today and come up with next steps.

powersj · 2023-06-14T19:23:25Z

@neelayu,

When I tested this I was more concerned about not breaking the TOML parser and less about how we maintain backwards compatibility. I clearly missed the various regex like usage throughout the TOML config with Telegraf.

We do not want to break configurations between versions so we want to ensure the previous behavior is maintained.

In order to add your new feature, which I really do want to keep, @srebhan and I have two possible ideas:

What if we use the double dollar sign (e.g. $$), to indicate options with the new behavior. And leave all single dollar signs values alone? We would like to try this first.
If that first option does not work, what about adding a CLI flag that enables the new behavior and we can document for users that this could impact certain configuration options and that those options need to be escaped properly.

What do you think?

neelayu · 2023-06-15T10:40:34Z

@powersj
So I thought about this a lot today. It seems the first option would be difficult to implement. It would involve lot of string parsing and regex(probably) and even after that it might not guarantee correctness.

The second option seems feasible and basically enforces users to check their existing configurations if they are using the feature flag. docker-compose has this feature to interpolate env variables if the flag is set.

Let me know your thoughts.

srebhan · 2023-06-15T13:33:46Z

@powersj and @neelayu I think it is not so difficult you just have to do some preprocessing... i.e.

replace all $${.+} patterns with \ufffd{...}
replace all ${.+} patterns with $${...}
replace all \uffd{.+ patterns with ${...}

Alternatively, you can even use the SubstituteWith function to directly specify the double-dollar-pattern...

powersj · 2023-06-15T13:42:50Z

@srebhan,

$${.+}

Is there a case where this is a valid regex or usage in the existing processors? We are letting this feature take over the usage of the double dollar sign and I want to make sure we are not still stepping on someone's current and valid usage.

neelayu · 2023-06-15T14:32:18Z

Yes I have considered the possibility of using SubstituteWith func.

Just to clarify-
We want to support
$${VAR} syntax for env substitution? and their variations.
For older telegraf versions this would break because we only had ${VAR} and $VAR syntax. Is that correct?

srebhan · 2023-06-15T15:05:07Z

@peerfiet can you please PR #13451 and let me know if this fixes your issue!?

srebhan · 2023-06-15T15:06:20Z

@neelayu OF COURSE I took care of this. ;-) Now patterns $${VAR} and $VAR are substituted while ${VAR} is not.

peerfiet · 2023-06-15T15:32:08Z

@peerfiet can you please PR #13451 and let me know if this fixes your issue!?

@srebhan Sorry, I don't have a test system here and never built Telegraf from scratch yet. I'm using Telegraf as provided at Debian repositories from influxdata.com.

peerfiet · 2023-06-15T15:47:20Z

I just found the right package in the PR build artifacts and installed it :-) And: yes, this would fix the issue. Both is working then: $1 and ${1}.

srebhan · 2023-06-16T17:29:28Z

@peerfiet can you please test the binary in #13457 once CI finished the tests successfully? We needed to change the behavior again as we found other cases where the implementation broke existing configs... So now your original config )(the one with replacement = "${1}" should just work (tm)... ;-) Please let me know if you find any issues!

peerfiet · 2023-06-16T21:13:04Z

Sure. I will let you know tomorrow asap.

peerfiet · 2023-06-17T08:58:57Z

@srebhan I get a lengthy error message:

Jun 17 10:46:40 pi telegraf[21628]: 2023-06-17T08:46:40Z I! Loading config: /etc/telegraf/telegraf.d/inputs.conf
Jun 17 10:46:40 pi telegraf[21628]: 2023-06-17T08:46:40Z E! error loading config file /etc/telegraf/telegraf.d/inputs.conf: error parsing data: Invalid template: "\n\n[[inputs.redis]]\n  \n  \n  servers = [\"tcp://localhost:6379\"]\n\n\n[[inputs.sysl
og]]\n\n  server = \"udp://localhost:6514\"\n\n\n\n[[inputs.openweathermap]]\n  \n  app_id = \"xxxxxxx\"\n\n  \n  city_id = [\"xxxxxxx\"]\n\n  \n  \n  \n  \n  \n\n  \n  fetch = [\"weather\", \"forecast\"]\n\n  \n  \n\n  \n  \
n\n  \n  \n  \n\n  \n  \n  interval = \"10m\"\n\n\n\n[[inputs.disk]]\n  \n  \n  \n\n  \n  ignore_fs = [\"tmpfs\", \"devtmpfs\", \"devfs\", \"iso9660\", \"overlay\", \"aufs\", \"squashfs\"]\n\n  \n  \n  \n  \n\n\n\n[[inputs.diskio]]\n  \n  \n  \n  \n
 \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n  \n\n\n\n[[inputs.kernel]]\n  \n\n\n\n[[inputs.mem]]\n  \n\n\n\n[[inputs.processes]]\n  \n\n\n\n[[inputs.swap]]\n  \n\n\n\n[[inputs.system]]\n  \n\n\n\n[[inputs.cpu]]\n
\n  percpu = true\n  \n  totalcpu = true\n  \n  collect_cpu_time = false\n  \n  report_active = false\n  \n  core_tags = false\n  \n  \n\n\n\n\n\n\n\n\n\n\n  \n\n\n\n \n\n\n\n\n\n\n[[inputs.exec]]\n\talias = \"processor temp\"\n\tinterval = \"30s\"\n
\tname_override = \"processor_temp\"\n\n  \n commands = [\n   \"/home/iobroker/scripts/processor_temp.sh\",\n ]\n\n \n \n \n data_format = \"value\"\n data_type = \"float\"\n \n\n\n\n\n\n[[inputs.http]]\n  collection_jitter = \"30s\"\n  alias = \"pvF
orecast - Ost\"\n  urls = [\n    \"https://api.forecast.solar/estimate/watts/xxxxx/xxxxx/20/-60/2.26?time=utc\"\n  ]\n\n  headers = {\"accept\" = \"text/csv\", \"X-Delimiter\" = \"|\", \"X-Separator\" = \";\"}\n\n  data_format = \"csv\"\n  name_
override = \"pvForecast\"\n  interval = \"3600s\"\n  csv_header_row_count = 0\n  csv_column_names = [\"time\",\"power\"]\n  csv_delimiter = \";\"\n  csv_timestamp_column = \"time\"\n  csv_timestamp_format = \"2006-01-02T15:04:05-07:00\"\n  timeout =
\"30s\"\n  \n  [inputs.http.tags]\n\t\n\tname = \"Ost\"\n\n[[inputs.http]]\n  collection_jitter = \"30s\"\n  alias = \"pvForecast - West\"\n  urls = [\n    \"https://api.forecast.solar/estimate/watts/xxxxx/xxxxx/20/120/2.26?time=utc\"\n  ]\n\n
headers = {\"accept\" = \"text/csv\", \"X-Delimiter\" = \"|\", \"X-Separator\" = \";\"}\n\n  data_format = \"csv\"\n  name_override = \"pvForecast\"\n  interval = \"3600s\"\n  csv_header_row_count = 0\n  csv_column_names = [\"time\",\"power\"]\n  csv
_delimiter = \";\"\n  csv_timestamp_column = \"time\"\n  csv_timestamp_format = \"2006-01-02T15:04:05-07:00\"\n  timeout = \"30s\"\n  \n  [inputs.http.tags]\n\t\n\tname = \"West\"\n\n[[inputs.execd]]\n  alias = \"smartmeter\"\n  \n  \n  \n  \n  comma
nd = [\"/home/iobroker/scripts/smartmeter.py\"]\n\n  name_override = \"smartmeter\"\n\n  \n  \n  \n  \n  \n\n  \n  \n  \n  \n  \n  \n  \n  \n  signal = \"none\"\n\n  \n  restart_delay = \"10s\"\n\n  \n  \n  \n  \n  data_format = \"json\"\n  \n  tag_k
eys = [\n    \"Hersteller\",\n    \"Seriennummer\"\n  ]\n  \n \n \n\n\n\n\n\n\n\n\n\n\n  \n\n\n[[inputs.socket_listener]]\n  alias = \"UPD Listener (Water/Gas/shelly_button)\"\n  service_address = \"udp://:8094\"\n  data_format = \"json\"\n  json_nam
e_key = \"metric_name\"\n  tag_keys = [\"name\"]\n\n\n\n\n  \n[[inputs.exec]]\n\talias = \"seat\"\n\tinterval = \"600s\"\n\tname_override = \"seat\"\n\n  \n commands = [\n   \"/home/iobroker/scripts/seat_monitor.py\",\n ]\n  \n  timeout = \"60s\"\n\n
 data_format = \"json\"\n \n tag_keys = [\"vin\"]\n\n\n\n\n  \n[[inputs.exec]]\n\talias = \"speedtest\"\n\tinterval = \"1800s\"\n\tname_override = \"speedtest\"\n\n  \n commands = [\n   \"/home/iobroker/scripts/speedtest.py\",\n ]\n  \n  timeout = \"
30s\"\n\n data_format = \"json\"\n \n tag_keys = [\"server_name\", \"server_id\", \"server_location\"]\n\n\n\n\n[[inputs.http]]\n  alias = \"MoWaS - Messages\"\n  urls = [\"https://nina.api.proxy.bund.dev/api31/mowas/mapData.json\"]\n\n  data_format
= \"json_v2\"\n  name_override = \"mowas_messages\"\n  interval = \"60s\"\n  timeout = \"10s\"\n\n  [[inputs.http.json_v2]]\n\t [[inputs.http.json_v2.object]]\n\t\tpath = \"@this\"\n\t\ttags = [\"severity\", \"type\", \"id\"]  \n  \n  [inputs.http.ta
gs]\n\tinflux_output_database = \"telegraf_test\"\n\n[[inputs.http]]\n  alias = \"Wallbox Transactions\"\n  urls = [\"http://wallbox.fritz.box:12800/user/transactions\"]\n\n  data_format = \"json_v2\"\n  name_override = \"wallbox_transactions\"\n  in
terval = \"600s\"\n  timeout = \"10s\"\n\n  [[inputs.http.json_v2]]\n\t\n  \n\t [[inputs.http.json_v2.object]]\n\t\ttimestamp_format = \"unix_ms\"\n\t\tpath = \"value.#.@this\"\n\t\ttags = [\"chargepointIdentity\"]\n\t\ttimestamp_key = \"stopDate\"\n
  \n\n\n\n\n[[processors.starlark]]\nalias = \"Wallbox Transactions - No meterStop Remover\"\norder = 100\nnamepass = [\"wallbox_transactions\"]\nsource = '''\ndef apply(metric):\n  v = metric.fields.get('meterStop')\n  if v == None:\n\tmetric.fields
.clear()\n  return metric\n'''\n\n[[inputs.http]]\n  alias = \"Wallbox Status\"\n  urls = [\"http://wallbox.fritz.box:12800/user/status\"]\n\n  data_format = \"json_v2\"\n  name_override = \"wallbox_status\"\n  interval = \"10s\"\n  timeout = \"10s\"
\n\n  [[inputs.http.json_v2]]\n\n\t [[inputs.http.json_v2.object]]\n\t\tpath = \"{connectors.0,cpid,serial,model}\"\n\t\ttags = [\"cpid\", \"serial\", \"model\"]\n  \n  \n\t\n\n[[processors.regex]]\norder = 50\nnamepass = [\"wallbox_status\"]\n\n[[pr
ocessors.regex.fields]]\n    \n    key = \"0_chargingRate\"\n    \n    \n    pattern = \"Charging rate: (.*)kw.*\\\\[(.*)A (.*)pf \\\\]\"\n    replacement = \"${1}\"\n    \n    \n    result_key = \"charge_kW\"\n\n[[processors.regex.fields]]\n    \n
  key = \"0_chargingRate\"\n    \n    \n    pattern = \"Charging rate: (.*)kw.*\\\\[(.*)A (.*)pf \\\\]\"\n    replacement = \"${2}\"\n    \n    \n    result_key = \"charge_A\"\n\n[[processors.regex.fields]]\n    \n    key = \"0_chargingRate\"\n    \n
    \n    pattern = \"Charging rate: (.*)kw.*\\\\[(.*)A (.*)pf \\\\]\"\n    replacement = \"${3}\"\n    \n    \n    result_key = \"charge_pf\"\n\n\n\n[[processors.starlark]]\nalias = \"Wallbox Status - Converter\"\norder = 100\nnamepass = [\"wallbox_
status\"]\nsource = '''\nload(\"logging.star\", \"log\")\n\ndef convertFloat(m, name):\n    v = m.fields.get(name)\n    #log.info(m.fields.get(\"0_chargingRate\"))\n    if v == None:\n       m.fields.pop(name)\n       return\n    v = str(v.strip())\n
    #log.info(v)\n\n    if v == \"\":\n       m.fields.pop(name)\n       return\n    f = float(v)\n    m.fields[name] = f\n    if name == \"charge_kW\":\n      f *= 1000.0\n      m.fields[\"charge_W\"] = f\n\ndef apply(metric):\n    convertFloat(metr
ic, 'charge_kW')\n    convertFloat(metric, 'charge_A')\n    convertFloat(metric, 'charge_pf')\n    return metric\n'''\n\n[[processors.override]]\norder = 110\nnamepass = [\"wallbox_status\"]\nfieldpass = [\"charge_W\", \"charge_A\", \"charge_pf\"]\n\
n\n\n[[inputs.ping]]\n  \n  urls = [\"router.fritz.box\",\n\t  \"repeater-eg.fritz.box\",\n\t  \"repeater-og.fritz.box\",\n\t  \"repeater-garage.fritz.box\",\n\t  \"wallbox.fritz.box\",\n\t  \"PS4.fritz.box\",\n\t  \"PS4-LAN.fritz.box\",\n\t  \"PS5-L
AN.fritz.box\",\n\t  \"PS5.fritz.box\",\n\t  \"printer-ink.fritz.box\",\n\t  \"esp32-arduino-2-og.fritz.box\", \n\t  \"esp32-arduino-1.fritz.box\", \n\t  \"shelly-pv.fritz.box\",\n\t  \"shelly-temp-fbh-eg.fritz.box\",\n\t  \"shelly-temp-fbh-og.fritz.
box\",\n\t  \"shelly-temp-heizung.fritz.box\",\n\t  \"shelly-licht-aussen.fritz.box\",\n\t  \"shelly-17.fritz.box\", \n\t  \"shelly-18.fritz.box\", \n\t  \"shelly-4.fritz.box\", \n\t  \"shelly-11.fritz.box\", \n\t  \"shelly-3.fritz.box\", \n\t  \"she
lly-15.fritz.box\", \n\t  \"nas.fritz.box\",\n\t  \"shelly-14.fritz.box\", \n\t\t  \n  ]\n\n  \n  \n  \n  \n  \n  \n  \n  method = \"native\""
Jun 17 10:46:40 pi systemd[1]: telegraf.service: Main process exited, code=exited, status=1/FAILURE
Jun 17 10:46:40 pi systemd[1]: telegraf.service: Failed with result 'exit-code'.
Jun 17 10:46:40 pi systemd[1]: Failed to start Telegraf.

srebhan · 2023-06-20T06:53:26Z

@peerfiet can you please show me the inputs.conf you are using above? Are you sure you reverted back to your initial config, i.e.

[[processors.regex]]
order = 50
namepass = ["wallbox_status"]

[[processors.regex.fields]]
    key = "0_chargingRate"
    pattern = "Charging rate: (.*)kw.*\\[(.*)A (.*)pf \\]"
    replacement = "${1}"
    result_key = "charge_kW"

that should work again now without further changes?

peerfiet · 2023-06-20T08:43:32Z

@srebhan Here is my conf, that is giving above error (error loading config file /etc/telegraf/telegraf.d/inputs.conf: error parsing data: Invalid template:)

# Read metrics from one or many redis servers
[[inputs.redis]]
  ## specify servers via a url matching:
  ##  [protocol://][:password]@address[:port]
  servers = ["tcp://localhost:6379"]


[[inputs.syslog]]
# ## Protocol, address and port to host the syslog receiver.
  server = "udp://localhost:6514"


# Read current weather and forecasts data from openweathermap.org
[[inputs.openweathermap]]
  ## OpenWeatherMap API key.
  app_id = "xxxxxxxxxxxxxxxxxx"

  ## City ID's to collect weather data from.
  city_id = ["xxxxxxxxx"]

  ## Language of the description field. Can be one of "ar", "bg",
  ## "ca", "cz", "de", "el", "en", "fa", "fi", "fr", "gl", "hr", "hu",
  ## "it", "ja", "kr", "la", "lt", "mk", "nl", "pl", "pt", "ro", "ru",
  ## "se", "sk", "sl", "es", "tr", "ua", "vi", "zh_cn", "zh_tw"
  # lang = "en"

  ## APIs to fetch; can contain "weather" or "forecast".
  fetch = ["weather", "forecast"]

  ## OpenWeatherMap base URL
  # base_url = "https://api.openweathermap.org/"

  ## Timeout for HTTP response.
  # response_timeout = "5s"

  ## Preferred unit system for temperature and wind speed. Can be one of
  ## "metric", "imperial", or "standard".
  # units = "metric"

  ## Query interval; OpenWeatherMap weather data is updated every 10
  ## minutes.
  interval = "10m"


# Read metrics about disk usage by mount point
[[inputs.disk]]
  ## By default stats will be gathered for all mount points.
  ## Set mount_points will restrict the stats to only the specified mount points.
  # mount_points = ["/"]

  ## Ignore mount points by filesystem type.
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

  ## Ignore mount points by mount options.
  ## The 'mount' command reports options of all mounts in parathesis.
  ## Bind mounts can be ignored with the special 'bind' option.
  # ignore_mount_opts = []


# Read metrics about disk IO by device
[[inputs.diskio]]
  ## By default, telegraf will gather stats for all devices including
  ## disk partitions.
  ## Setting devices will restrict the stats to the specified devices.
  # devices = ["sda", "sdb", "vd*"]
  ## Uncomment the following line if you need disk serial numbers.
  # skip_serial_number = false
  #
  ## On systems which support it, device metadata can be added in the form of
  ## tags.
  ## Currently only Linux is supported via udev properties. You can view
  ## available properties for a device by running:
  ## 'udevadm info -q property -n /dev/sda'
  ## Note: Most, but not all, udev properties can be accessed this way. Properties
  ## that are currently inaccessible include DEVTYPE, DEVNAME, and DEVPATH.
  # device_tags = ["ID_FS_TYPE", "ID_FS_USAGE"]
  #
  ## Using the same metadata source as device_tags, you can also customize the
  ## name of the device via templates.
  ## The 'name_templates' parameter is a list of templates to try and apply to
  ## the device. The template may contain variables in the form of '$PROPERTY' or
  ## '${PROPERTY}'. The first template which does not contain any variables not
  ## present for the device is used as the device name tag.
  ## The typical use case is for LVM volumes, to get the VG/LV name instead of
  ## the near-meaningless DM-0 name.
  # name_templates = ["$ID_FS_LABEL","$DM_VG_NAME/$DM_LV_NAME"]


# Get kernel statistics from /proc/stat
[[inputs.kernel]]
  # no configuration


# Read metrics about memory usage
[[inputs.mem]]
  # no configuration


# Get the number of processes and group them by status
[[inputs.processes]]
  # no configuration


# Read metrics about swap memory usage
[[inputs.swap]]
  # no configuration


# Read metrics about system load & uptime
[[inputs.system]]
  # no configuration


# Read metrics about cpu usage
[[inputs.cpu]]
  ## Whether to report per-cpu stats or not
  percpu = true
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics
  collect_cpu_time = false
  ## If true, compute and report the sum of all non-idle CPU states
  report_active = false
  ## If true and the info is available then add core_id and physical_id tags
  core_tags = false
  
  
#growatt
#[[inputs.exec]]
#	alias = "growatt"
#	interval = "600s"
#	name_override = "growatt"

#  ## Commands array
# commands = [
#   "/home/iobroker/scripts/growatt.py",
# ]
  
#  timeout = "30s"

# data_format = "json"
 #json_string_fields = []

#[inputs.exec.tags]
#	influx_output_database = "telegraf_test"


# processor temp
[[inputs.exec]]
	alias = "processor temp"
	interval = "30s"
	name_override = "processor_temp"

  ## Commands array
 commands = [
   "/home/iobroker/scripts/processor_temp.sh",
 ]

 
 ## Timeout for each command to complete.
 #timeout = "2s"
 data_format = "value"
 data_type = "float"
 #json_string_fields = []

#[inputs.exec.tags]
#	influx_output_database = "telegraf_test"


[[inputs.http]]
  collection_jitter = "30s"
  alias = "pvForecast - Ost"
  urls = [
    "https://api.forecast.solar/estimate/watts/xxxxxxxx/xxxxxxxxx/20/-60/2.26?time=utc"
  ]

  headers = {"accept" = "text/csv", "X-Delimiter" = "|", "X-Separator" = ";"}

  data_format = "csv"
  name_override = "pvForecast"
  interval = "3600s"
  csv_header_row_count = 0
  csv_column_names = ["time","power"]
  csv_delimiter = ";"
  csv_timestamp_column = "time"
  csv_timestamp_format = "2006-01-02T15:04:05-07:00"
  timeout = "30s"
  
  [inputs.http.tags]
	#influx_output_database = "telegraf_test"
	name = "Ost"

[[inputs.http]]
  collection_jitter = "30s"
  alias = "pvForecast - West"
  urls = [
    "https://api.forecast.solar/estimate/watts/xxxxxxxxx/xxxxxxxx/20/120/2.26?time=utc"
  ]

  headers = {"accept" = "text/csv", "X-Delimiter" = "|", "X-Separator" = ";"}

  data_format = "csv"
  name_override = "pvForecast"
  interval = "3600s"
  csv_header_row_count = 0
  csv_column_names = ["time","power"]
  csv_delimiter = ";"
  csv_timestamp_column = "time"
  csv_timestamp_format = "2006-01-02T15:04:05-07:00"
  timeout = "30s"
  
  [inputs.http.tags]
	#influx_output_database = "telegraf_test"
	name = "West"

[[inputs.execd]]
  alias = "smartmeter"
  #interval = "1s"
  #metric_batch_size = 1
  ## One program to run as daemon.
  ## NOTE: process and each argument should each be their own string
  command = ["/home/iobroker/scripts/smartmeter.py"]

  name_override = "smartmeter"

  ## Environment variables
  ## Array of "key=value" pairs to pass as environment variables
  ## e.g. "KEY=value", "USERNAME=John Doe",
  ## "LD_LIBRARY_PATH=/opt/custom/lib64:/usr/local/libs"
  # environment = []

  ## Define how the process is signaled on each collection interval.
  ## Valid values are:
  ##   "none"    : Do not signal anything. (Recommended for service inputs)
  ##               The process must output metrics by itself.
  ##   "STDIN"   : Send a newline on STDIN. (Recommended for gather inputs)
  ##   "SIGHUP"  : Send a HUP signal. Not available on Windows. (not recommended)
  ##   "SIGUSR1" : Send a USR1 signal. Not available on Windows.
  ##   "SIGUSR2" : Send a USR2 signal. Not available on Windows.
  signal = "none"

  ## Delay before the process is restarted after an unexpected termination
  restart_delay = "10s"

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "json"
  
  tag_keys = [
    "Hersteller",
    "Seriennummer"
  ]
  
 #[inputs.execd.tags]
 #   influx_output_database = "telegraf_test"

#[[inputs.http]]
#  interval = "30s"
#  alias = "telegraf_diff"
#  urls = [
#    "http://localhost:8086/query?db=telegraf&q=SELECT%20non_negative_difference%28energy%29%20AS%20energy_diff%20INTO%20telegraf..shelly_relay%20FROM%20shelly_relay%20WHERE%20time%20%3E%20now%28%29-2m%20GROUP%20BY%20%2A"
#  ]
#  data_format = "value"
#  data_type = "string"
#  timeout = "30s"
  

# Water / Gas
[[inputs.socket_listener]]
  alias = "UPD Listener (Water/Gas/shelly_button)"
  service_address = "udp://:8094"
  data_format = "json"
  json_name_key = "metric_name"
  tag_keys = ["name"]

#   [inputs.socket_listener.tags]
#     influx_output_database = "telegraf_test"

  #seat
[[inputs.exec]]
	alias = "seat"
	interval = "600s"
	name_override = "seat"

  ## Commands array
 commands = [
   "/home/iobroker/scripts/seat_monitor.py",
 ]
  
  timeout = "60s"

 data_format = "json"
 #json_string_fields = ["position"]
 tag_keys = ["vin"]

#[inputs.exec.tags]
#	influx_output_database = "telegraf_test"

  #seat
[[inputs.exec]]
	alias = "speedtest"
	interval = "1800s"
	name_override = "speedtest"

  ## Commands array
 commands = [
   "/home/iobroker/scripts/speedtest.py",
 ]
  
  timeout = "30s"

 data_format = "json"
 #json_string_fields = []
 tag_keys = ["server_name", "server_id", "server_location"]

#[inputs.exec.tags]
#	influx_output_database = "telegraf_test"

[[inputs.http]]
  alias = "MoWaS - Messages"
  urls = ["https://nina.api.proxy.bund.dev/api31/mowas/mapData.json"]

  data_format = "json_v2"
  name_override = "mowas_messages"
  interval = "60s"
  timeout = "10s"

  [[inputs.http.json_v2]]
	 [[inputs.http.json_v2.object]]
		path = "@this"
		tags = ["severity", "type", "id"]  
  
  [inputs.http.tags]
	influx_output_database = "telegraf_test"

[[inputs.http]]
  alias = "Wallbox Transactions"
  urls = ["http://wallbox.fritz.box:12800/user/transactions"]

  data_format = "json_v2"
  name_override = "wallbox_transactions"
  interval = "600s"
  timeout = "10s"

  [[inputs.http.json_v2]]
	
  
	 [[inputs.http.json_v2.object]]
		timestamp_format = "unix_ms"
		path = "value.#.@this"
		tags = ["chargepointIdentity"]
		timestamp_key = "stopDate"
  
#  [inputs.http.tags]
#	influx_output_database = "telegraf_test"


[[processors.starlark]]
alias = "Wallbox Transactions - No meterStop Remover"
order = 100
namepass = ["wallbox_transactions"]
source = '''
def apply(metric):
  v = metric.fields.get('meterStop')
  if v == None:
	metric.fields.clear()
  return metric
'''

[[inputs.http]]
  alias = "Wallbox Status"
  urls = ["http://wallbox.fritz.box:12800/user/status"]

  data_format = "json_v2"
  name_override = "wallbox_status"
  interval = "10s"
  timeout = "10s"

  [[inputs.http.json_v2]]

	 [[inputs.http.json_v2.object]]
		path = "{connectors.0,cpid,serial,model}"
		tags = ["cpid", "serial", "model"]
  
  #[inputs.http.tags]
	#influx_output_database = "telegraf_test"

[[processors.regex]]
order = 50
namepass = ["wallbox_status"]

[[processors.regex.fields]]
    ## Field to change
    key = "0_chargingRate"
    ## All the power of the Go regular expressions available here
    ## For example, named subgroups
    pattern = "Charging rate: (.*)kw.*\\[(.*)A (.*)pf \\]"
    replacement = "${1}"
    ## If result_key is present, a new field will be created
    ## instead of changing existing field
    result_key = "charge_kW"

[[processors.regex.fields]]
    ## Field to change
    key = "0_chargingRate"
    ## All the power of the Go regular expressions available here
    ## For example, named subgroups
    pattern = "Charging rate: (.*)kw.*\\[(.*)A (.*)pf \\]"
    replacement = "${2}"
    ## If result_key is present, a new field will be created
    ## instead of changing existing field
    result_key = "charge_A"

[[processors.regex.fields]]
    ## Field to change
    key = "0_chargingRate"
    ## All the power of the Go regular expressions available here
    ## For example, named subgroups
    pattern = "Charging rate: (.*)kw.*\\[(.*)A (.*)pf \\]"
    replacement = "${3}"
    ## If result_key is present, a new field will be created
    ## instead of changing existing field
    result_key = "charge_pf"



[[processors.starlark]]
alias = "Wallbox Status - Converter"
order = 100
namepass = ["wallbox_status"]
source = '''
load("logging.star", "log")

def convertFloat(m, name):
    v = m.fields.get(name)
    #log.info(m.fields.get("0_chargingRate"))
    if v == None:
       m.fields.pop(name)
       return
    v = str(v.strip())
    #log.info(v)

    if v == "":
       m.fields.pop(name)
       return
    f = float(v)
    m.fields[name] = f
    if name == "charge_kW":
      f *= 1000.0
      m.fields["charge_W"] = f

def apply(metric):
    convertFloat(metric, 'charge_kW')
    convertFloat(metric, 'charge_A')
    convertFloat(metric, 'charge_pf')
    return metric
'''

[[processors.override]]
order = 110
namepass = ["wallbox_status"]
fieldpass = ["charge_W", "charge_A", "charge_pf"]


# Ping given url(s) and return statistics
[[inputs.ping]]
  ## Hosts to send ping packets to.
  urls = ["router.fritz.box",
	  "repeater-eg.fritz.box",
	  "repeater-og.fritz.box",
	  "repeater-garage.fritz.box",
	  "wallbox.fritz.box",
	  "PS4.fritz.box",
	  "PS4-LAN.fritz.box",
	  "PS5-LAN.fritz.box",
	  "PS5.fritz.box",
	  "printer-ink.fritz.box",
	  "esp32-arduino-2-og.fritz.box", # BLE2MQTT OG
	  "esp32-arduino-1.fritz.box", # BLE2MQTT EG
	  "shelly-pv.fritz.box",
	  "shelly-temp-fbh-eg.fritz.box",
	  "shelly-temp-fbh-og.fritz.box",
	  "shelly-temp-heizung.fritz.box",
	  "shelly-licht-aussen.fritz.box",
	  "shelly-17.fritz.box", # (Garage Links)
	  "shelly-18.fritz.box", # (Garage Rechts)
	  "shelly-4.fritz.box", # GSM
	  "shelly-11.fritz.box", # WM
	  "shelly-3.fritz.box", # Heizung
	  "shelly-15.fritz.box", # Trockner
	  "nas.fritz.box",
	  "shelly-14.fritz.box", #TVAV
		  
  ]

  ## Method used for sending pings, can be either "exec" or "native".  When set
  ## to "exec" the systems ping command will be executed.  When set to "native"
  ## the plugin will send pings directly.
  ##
  ## While the default is "exec" for backwards compatibility, new deployments
  ## are encouraged to use the "native" method for improved compatibility and
  ## performance.
  method = "native"

srebhan · 2023-06-20T10:51:47Z

@peerfiet can you please try again once the updated version is built by our CI!?

peerfiet · 2023-06-20T13:30:34Z

@srebhan Using the binary from #13457 (comment) I can confirm above config is working as expected. Furthermore, using $1 is working as well.

peerfiet added the bug unexpected problem or unintended behavior label Jun 13, 2023

peerfiet changed the title ~~regex plugin is no longer working with "${1}" in "replacement", need to use "$1" instead, documentation is suggesting "${1}"~~ [regex] plugin is no longer working with "${1}" in "replacement", need to use "$1" instead, documentation is suggesting "${1}" Jun 13, 2023

This was referenced Jun 14, 2023

name_templates of diskio plugin is invalid #13438

Closed

lost functionality with jsonAta #13442

Closed

powersj mentioned this issue Jun 15, 2023

jsonnet $lookup function breaks the config in 1.27 #13449

Closed

srebhan mentioned this issue Jun 15, 2023

fix(config): Move bracketed environment variable substitution to double-dollar #13451

Merged

3 tasks

neelayu mentioned this issue Jun 15, 2023

Outputs.influxdb authorization failed messages after upgrade to v1.27 #13455

Closed

powersj changed the title ~~[regex] plugin is no longer working with "${1}" in "replacement", need to use "$1" instead, documentation is suggesting "${1}"~~ [regression] v1.27 shell-like expansion replaces non-environment variables Jun 15, 2023

srebhan closed this as completed in #13451 Jun 15, 2023

srebhan mentioned this issue Jun 16, 2023

fix(config): Restore old environment var behavior with option #13457

Merged

3 tasks

srebhan reopened this Jun 16, 2023

neelayu mentioned this issue Jun 18, 2023

regex behavior changed since 1.26 #13460

Closed

srebhan added a commit to srebhan/telegraf that referenced this issue Jun 20, 2023

Fix regexp patterns as shown in influxdata#13432

398d7d2

srebhan self-assigned this Jun 20, 2023

powersj closed this as completed in #13457 Jun 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[regression] v1.27 shell-like expansion replaces non-environment variables #13432

[regression] v1.27 shell-like expansion replaces non-environment variables #13432

peerfiet commented Jun 13, 2023

neelayu commented Jun 14, 2023

neelayu commented Jun 14, 2023

powersj commented Jun 14, 2023

powersj commented Jun 14, 2023

neelayu commented Jun 15, 2023

srebhan commented Jun 15, 2023 •

edited

Loading

powersj commented Jun 15, 2023 •

edited

Loading

neelayu commented Jun 15, 2023

srebhan commented Jun 15, 2023

srebhan commented Jun 15, 2023

peerfiet commented Jun 15, 2023

peerfiet commented Jun 15, 2023 •

edited

Loading

srebhan commented Jun 16, 2023

peerfiet commented Jun 16, 2023 via email •

edited

Loading

peerfiet commented Jun 17, 2023 •

edited

Loading

srebhan commented Jun 20, 2023

peerfiet commented Jun 20, 2023

srebhan commented Jun 20, 2023

peerfiet commented Jun 20, 2023

[regression] v1.27 shell-like expansion replaces non-environment variables #13432

[regression] v1.27 shell-like expansion replaces non-environment variables #13432

Comments

peerfiet commented Jun 13, 2023

Relevant telegraf.conf

Logs from Telegraf

System info

Docker

Steps to reproduce

Expected behavior

Actual behavior

Additional info

neelayu commented Jun 14, 2023

neelayu commented Jun 14, 2023

powersj commented Jun 14, 2023

powersj commented Jun 14, 2023

neelayu commented Jun 15, 2023

srebhan commented Jun 15, 2023 • edited Loading

powersj commented Jun 15, 2023 • edited Loading

neelayu commented Jun 15, 2023

srebhan commented Jun 15, 2023

srebhan commented Jun 15, 2023

peerfiet commented Jun 15, 2023

peerfiet commented Jun 15, 2023 • edited Loading

srebhan commented Jun 16, 2023

peerfiet commented Jun 16, 2023 via email • edited Loading

peerfiet commented Jun 17, 2023 • edited Loading

srebhan commented Jun 20, 2023

peerfiet commented Jun 20, 2023

srebhan commented Jun 20, 2023

peerfiet commented Jun 20, 2023

srebhan commented Jun 15, 2023 •

edited

Loading

powersj commented Jun 15, 2023 •

edited

Loading

peerfiet commented Jun 15, 2023 •

edited

Loading

peerfiet commented Jun 16, 2023 via email •

edited

Loading

peerfiet commented Jun 17, 2023 •

edited

Loading