Add support for datadog query function wrappers - WIP

[dalgibbard]

Allow the Datadog Scaler to parse/validate queries which contain a wrapping function; eg:

per_second(sum:system.cpu.user{*}.rollup(avg, 30))

Previously, the code would see these as missing an aggregator; and prefix them with avg: resulting in an invalid query of:

avg:per_second(sum:system.cpu.user{*}.rollup(avg, 30))

Checklist

• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (repo) (if applicable)
• Changelog has been updated and is aligned with our changelog requirements

Fixes #2761

[zroubalik]

Pardon my ignorance, just looking at all these regexes, isn't there any validator in any Datadog library? That could be used here. What is being used in Datadog to validate this?

[dalgibbard]

An excellent question; will check!

[dalgibbard]

Had a quick look around, plenty for submitting metrics etc, but not a lot in the way of validating if a metrics query is valid.

The Datadog UI posts to https://app.datadoghq.com/api/v2/query/timeseries when changing queries interactively, and returns 400 when it's invalid - but seeing as this requires auth anyway, if going down that route, you may as well skip validation in the Scaler, and just handle errors when they come back from the upstream.

[dalgibbard]

Closing this, as it makes sense to not validate in the code, and instead use the upstream API.