[ML] Improve messaging and support for datafeed using aggregated and scripted fields

[qn895]

Summary

This PR adds better messaging for unsupported configurations and improved support for aggregated and scripted fields. Changes include:

1. Add better messaging on why some jobs cannot be viewed using the Single Metric Viewer
   
   

2. Add ability to create and view job when datafeed uses something other than buckets in the datafeed's aggregation (e.g. an arbitrary name like 'my_buckets'). Previously, this is only supported via the API. This PR makes it so that the fields are listed correctly in the job creation wizard and makes the charts plot correctly.
   
   

3. Ability to plot Anomaly Explorer charts for jobs with aggregated or scripted fields with model plot data in cases if source chart is not chartable.

• Before
  
• After
  

4. Add fix to we can no longer click on disabled buttons

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[lcawl]

I think this could be clarified a bit. For example:

           'the datafeed has aggregation fields and the aggregation interval is not the same as the bucket span',

or:

           'bucket span and aggregation interval are not the same for a datafeed with aggregation fields', 

I prefer the first suggestion.

[peteharverson]

My vote goes to the first suggestion from @lcawl

[qn895]

Updated here 02a579b

[peteharverson]

My vote goes to the first suggestion from @lcawl

[peteharverson]

For this use case, I think we should enable access to the Single Metric Viewer, and show the charts in the Anomaly Explorer, but display a Toast / Callout which explains that the numbers on the chart may differ from the values reported for the anomaly. For example, this chart provides a lot of value:

[peteharverson]

Update - as discussed, we should just show the charts in this case, with no Toast / Callout about the differing bucket span and aggregation interval, as otherwise they would show up for jobs created in the Single Metric wizard, where we set the agg interval to 10% of the bucket span. We can address that edge case in a future PR, to show a toast warning only if the function used in the detector is different to that used in the datafeed aggregation.

[peteharverson]

I noticed that some of my jobs are greyed out in the job selector in the Single Metric Viewer when they shouldn't be. Is this because some of the logic in here is not quite correct? Jobs which don't use partitioning fields are being disabled here:

[qn895]

Removed here 15cc5ed

[peteharverson]

I think this needs are inserted, source data and model plot are not chartable

[qn895]

Updated here ce964f5

[peteharverson]

Swap the word records for charts.

[qn895]

Updated here ce964f5

[qn895]

As discussed I have updated a logic to be less strict so that it will not necessarily disable when there's nested term aggregations or when the bucket span != aggregation interval. This is so examples like below will still show up, without callouts, despite the anomaly markers not plotting correctly:

{
  "datafeed_id": "",
  "job_id": "",
  "indices": [
    "cloudwatch-*"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        }
      ]
    }
  },
  "aggregations": {
    "buckets": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "15m",
        "time_zone": "UTC"
      },
      "aggregations": {
        "@timestamp": {
          "max": {
            "field": "@timestamp"
          }
        },
        "instance": {
          "terms": {
            "field": "instance"
          },
          "aggs": {
            "DiskReadBytesSum": {
              "sum": {
                "field": "DiskReadBytes"
              }
            }
          }
        }
      }
    }
  }
}

  "analysis_config": {
    "bucket_span": "30m",
    "summary_count_field_name": "doc_count",
    "detectors": [
      {
        "detector_description": "mean DiskReadBytesSum partition instance",
        "function": "mean",
        "field_name": "DiskReadBytesSum",
        "partition_field_name": "instance",
        "detector_index": 0
      }
    ],
    "influencers": [
      "instance"
    ]
  },

[walterra]

Latest changes LGTM, great you added some functional tests too! 👍

[peteharverson]

Tested latest edits and LGTM!

[qn895]

Started flaky test suite runner...

[qn895]

Started flaky test suite runner...

[pheyos]

Functional tests LGTM, thanks for adding them as part of this PR! 🎉
Just one small nit after the latest change, but feel free to merge as is if you want.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: eb36a3a

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
ml	1592	1593	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
ml	5.2MB	5.3MB	+2.8KB

Distributable file count

id	before	after	diff
default	47010	47770	+760

History

• 💚 Build #93257 succeeded c95a44a
• 💚 Build #93161 succeeded 53d06b6
• 💔 Build #92951 failed 10ea7e4
• 💚 Build #92801 succeeded 3835583
• 💚 Build #92408 succeeded 18df04e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream