[ML] Datafeed script fields cannot be selected in the edit detector modal

[peteharverson]

When creating an Advanced ML job with a datafeed that uses a script field, the script field cannot be selected in the edit detector modal.

For example, this datafeed config creates a lat-lon field, but it does not appear in the field_name dropdown:

The only way to use this field in the detector is to select another field, and then edit it to the script field in the JSON tab.

Example job confiugration for testing:

{
  "job_id": "apache_lat_lon",
  "job_type": "anomaly_detector",
  "description": "Apache2 lat-long",
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "detector_description": "lat-long(location)",
        "function": "lat_long",
        "field_name": "lat-lon",
        "over_field_name": "apache2.access.remote_ip"
      }
    ],
    "influencers": [
      "apache2.access.remote_ip"
    ]
  },
  "analysis_limits": {
    "model_memory_limit": "1024mb",
    "categorization_examples_limit": 4
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_snapshot_retention_days": 1,
  "model_snapshot_min_version": "6.3.0",
  "datafeed_config": {
    "indices": [
      "filebeat-6.0.0-beta1-2017-apache2-*"
    ],
    "types": [],
    "query": {
      "exists": {
        "field": "apache2.access.geoip.location",
        "boost": 1
      }
    },
    "script_fields": {
      "lat-lon": {
        "script": {
          "source": "doc['apache2.access.geoip.location'].lat + ',' + doc['apache2.access.geoip.location'].lon",
          "lang": "painless"
        },
        "ignore_failure": false
      }
    },
    "chunking_config": {
      "mode": "auto"
    }
  },
  "groups": []
}

[elasticmachine]

Pinging @elastic/ml-ui

[peteharverson]

It looks like the fix applied in #21205 only works if the script field created in the datafeed-config exists when the Advanced Job wizard is opened, with the fields in the script_fields section only parsed when the wizard is opened (or when the datafeed index is edited in the wizard and the 'Reload index' button is pressed).

For example, if a new job is created, and a script_fields section is added to the datafeed config in the 'Edit JSON' tab, this field does not appear in the edit detector modal.

Reopening issue.

[peteharverson]

Issue raised in #18689 (comment) with script_fields added when job is being edited has been fixed by #23475. Closing issue.