[ML] DFA exploration: ROC chart not showing for Classification jobs with non default results field

[alvarezmelissa87]

Found in latest kibana.

Describe the bug:
When viewing the exploration page of a DFA classification job with results_field set to something other than the default ml , the ROC chart fails to load and shows an error callout.

Steps to reproduce:

1. Use the mushroom dataset to create a Classification job via the DFA wizard
2. Set dependent variable to edibility
3. Set results_field to something other than the default value (in my example I set it to "bob")
4. Open the results view and see that the ROC chart doesn't load

Expected behavior:
ROC chart should load correctly

Errors in browser console (if relevant):
Request sent to _evaluate endpoint:

{
   "index":"mushroom-class-01",
   "query":{
      "bool":{
         "must":[
            
         ]
      }
   },
   "evaluation":{
      "classification":{
         "actual_field":"edibility",
         "predicted_field":"bob.edibility_prediction",
         "metrics":{
            "accuracy":{
               
            },
            "recall":{
               
            },
            "auc_roc":{
               "include_curve":true,
               "class_name":"e"
            }
         }
      }
   }
}

Error message returned from _evaluate endpoint:

{
   "statusCode":400,
   "error":"Bad Request",
   "message":"[status_exception]: No documents found containing all the required fields [edibility, bob.edibility_prediction, ml.top_classes.class_name, ml.top_classes.class_probability]",
   "attributes":{
      "body":{
         "error":{
            "root_cause":[
               {
                  "type":"status_exception",
                  "reason":"No documents found containing all the required fields [edibility, bob.edibility_prediction, ml.top_classes.class_name, ml.top_classes.class_probability]"
               }
            ],
            "type":"status_exception",
            "reason":"No documents found containing all the required fields [edibility, bob.edibility_prediction, ml.top_classes.class_name, ml.top_classes.class_probability]"
         },
         "status":400
      }
   }
}

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[alvarezmelissa87]

@dimitris-athanasiou - I took a look at the evaluate docs and I don't see any specific examples for when results_field is not the default 'ml' value.

Also looks like the request sent is the same for Classification jobs with results_field set to the default and with a non default value. Curious if you have some insight into what we might be missing in sending to _evaluate?

[dimitris-athanasiou]

For classification, when the results field is different, you also need to provide the path to the top_classes field. As the _evaluate API is decoupled from the job config (in order to allow usage with indices not created with a DFA job), the API doesn't know of the job's results field. Basically, the request should be:

{
   "index":"mushroom-class-01",
   "query":{
      "bool":{
         "must":[
            
         ]
      }
   },
   "evaluation":{
      "classification":{
         "actual_field":"edibility",
         "predicted_field":"bob.edibility_prediction",
         "top_classes_field": "bob.top_classes",
         "metrics":{
            "accuracy":{
               
            },
            "recall":{
               
            },
            "auc_roc":{
               "include_curve":true,
               "class_name":"e"
            }
         }
      }
   }
}

Note that the UI app could always set that to {results_field}.top_classes where results_field is ml when the default results_field is used in order to avoid if-logic.