[ML] Support plot of metric data when datafeed uses scripted field

[elasticmachine]

Original comment by @peteharverson:

The Anomaly Explorer and Single Metric Viewer charts (if model_plot_config is disabled) do not currently support plotting metric data when the datafeed uses a derivative aggregation on a scripted field.

With no metric data available for plotting, the anomalies are plotted at a y-axis value of 0.

Note should also confirm whether charts are plotted correctly for non-derivative aggregations on a scripted field.

Extract of a sample job configuration using a derivative aggregation on a scripted field:

{
   "analysis_config":{
      "bucket_span":"1m",
      "summary_count_field_name":"doc_count",
      "detectors":[
         {
            "detector_description":"indexing_rate",
            "function":"mean",
            "field_name":"indexing_rate",
            "detector_rules":[

            ]
         }
      ],
      "influencers":[

      ]
   },
   "data_description":{
      "time_field":"timestamp",
      "time_format":"epoch_ms"
   },
   "datafeed_config":{
      "datafeed_id":"datafeed-logstash",
      "job_id":"indexing_rate",
      "query_delay":"60s",
      "frequency":"150s",
      "indexes":[
         ".monitoring-*"
      ],
      "types":[
         "index_stats"
      ],
      "query":{
         "bool":{
            "filter":[
               {
                  "term":{
                     "cluster_uuid":{
                        "value":"abcde",
                        "boost":1
                     }
                  }
               },
               {
                  "prefix":{
                     "index_stats.index":{
                        "value":"logstash",
                        "boost":1
                     }
                  }
               }
            ],
            "disable_coord":false,
            "adjust_pure_negative":true,
            "boost":1
         }
      },
      "aggregations":{
         "by_minute":{
            "date_histogram":{
               "field":"timestamp",
               "time_zone":"UTC",
               "interval":"1m",
               "offset":0,
               "order":{
                  "_key":"asc"
               },
               "keyed":false,
               "min_doc_count":0
            },
            "aggregations":{
               "timestamp":{
                  "max":{
                     "field":"timestamp"
                  }
               },
               "primaries_indexing_total":{
                  "max":{
                     "field":"index_stats.primaries.indexing.index_total"
                  }
               },
               "change":{
                  "derivative":{
                     "buckets_path":[
                        "primaries_indexing_total"
                     ],
                     "gap_policy":"skip"
                  }
               },
               "indexing_rate":{
                  "bucket_script":{
                     "buckets_path":{
                        "d":"change"
                     },
                     "script":{
                        "inline":"Math.round((params.d / 60))",
                        "lang":"painless"
                     },
                     "gap_policy":"skip"
                  }
               }
            }
         }
      },
      "scroll_size":1000,
      "chunking_config":{
         "mode":"manual",
         "time_span":"180000000ms"
      },
      "state":"stopped"
   },
   "state":"closed"
}

[elasticmachine]

Original comment by @peteharverson:

One possible solution for plotting the metric values in the Anomaly Explorer charts when the datafeed uses a derivative aggregation on a scripted field would be to allow the user to configure extra plot_function and plot_field_name fields which would be used in place of the aggregation configured in the detector.

So for example, for a detector which was using sum of a scripted field reciprocal function (1 / count + 1) , the charts could be configured to plot in the anomaly charts as count. The anomalies would then be overlaid on top of the line, plotted at the plot_function values rather than the actual values of the anomaly.

cc @stevedodson

[peteharverson]

Note that if the detector does a non-derivative aggregation on a script field defined in the datafeed configuration charts are not displayed in the Anomaly Explorer and the detector cannot be viewed in the Single Metric Viewer - see #19205. Enhancements to support the plot of metric data should ideally cover both the non-derivative and derivative aggregation use cases.

[kcm]

Any progress here? Have a customer that would like to visualize some jobs with splits/influencers from script_fields.

[peteharverson]

@kcm I'm afraid we haven't been able to look into this issue yet. Will look to see what we can do in the short-term to improve our support of datafeeds using scripted fields in the UI. Is your customer's datafeed configuration a similar setup to the one I posted in the description above?

[kcm]

It's a non-derivative feed, using a simple scripted composite field of two text fields - he wants to track filesystem metrics per disk, so he is joining the metricbeat data of system.filesystem.used.pct and system.filesystem.mount_point, then splitting on hostname. For now, we've created an actual field for this and backfilled it to historical data.

[richcollier]

a +1 from this discuss forum entry: https://discuss.elastic.co/t/x-pack-single-metric-job/166669/13

[richcollier]

Another +1 from https://discuss.elastic.co/t/anomaly-detection-using-derivative-aggregation/249207

[qn895]

Closing via #81923 and #84594 which improved support for scripted fields in Anomaly Detection jobs. Since there's a few edge cases we are not supporting yet in 7.11, I'll create follow up issues for each specific issue.