[ML] Change way the job config is passed to autodetect

[droberts195]

For historical reasons the job config that autodetect uses is passed to it in a mixture of 2 Windows .ini file style config files and a few command line options.

It would make life much easier and would make the code clearer if it was passed as a single config file in JSON format containing the exact job config document from the .ml-config index.

Benefits would be:

1. Much less code on the Java side - currently it has to translate the job config into a completely different format to pass to autodetect
2. Less of a learning curve for new joiners - when looking at the config handling code in autodetect they will be able to look at job config examples from our public documentation
3. Easier reproduction of problems directly using the autodetect process from the command line - currently when we get a job config from a support diag we have to manually translate it to the .ini formats and command line arguments in order to try to reproduce problems

[droberts195]

One tricky thing about this change is how to keep the way Java passes the inputs to C++ working while the changes are made.

I think the way to do it is as follows:

1. Add an option to the C++ code to accept a job config in JSON format. This could be a --config=/path/to/config.json argument. Initially nothing needs to be done with the supplied config, not even confirming that the file contains JSON.
2. Change the Java code to pass --config=/path/to/config.json as well as all the arguments it currently passes. The JSON file it should save to disk (in the same temporary directory where it currently writes the limits, model plot and field config files) should contain the job config JSON in UTF-8 format (as created by Job.toXContent()) with no adjustments. No adjustments is critical to making debugging easier in the future.
3. Change the C++ code to get its config information from the JSON config specified to --config instead of the mishmash of options it gets them from at the moment. This is obviously the main part of the work. Additionally the options that are no longer needed should be made optional (if they aren't already) but not completely removed.
4. Change the Java code not to pass the redundant options: --limitconfig, --fieldconfig, --bucketspan, --jobid, --modelplotconfig, --latency, --summarycountfield, --delimiter, --multivariateByFields, --stopCategorizationOnWarnStatus, --maxQuantileInterval, --persistInterval. This should allow quite a lot of Java code to be deleted.
5. Change the C++ code not to accept the redundant options: --limitconfig, --fieldconfig, --bucketspan, --jobid, --modelplotconfig, --latency, --summarycountfield, --delimiter, --multivariateByFields, --stopCategorizationOnWarnStatus, --maxQuantileInterval, --persistInterval.

Step 3 is by far the biggest of these steps. But the beauty of passing the config in both formats during the transition is that step 3 can be broken down into smaller steps, and these can even span multiple releases. (There is a slight cost if they span multiple releases in that anyone doing low level debugging of problems with that release will have to create both config formats. However, since the new config option takes the unmodified job config in JSON format the extra format is not hard to obtain.). We are only completely committed to this project at step 4.

One subtlety with the redundant options is that --timeformat option is still required. This is because the job config contains the time field format in Java notation, whereas the C++ processes require it in C++ format. So the C++ processes will have to ignore the time format in the JSON job config and use the override from the command in the unlikely event it's provided - it's not used in production, where Java passes the timestamps in the C++'s default format. So --timeformat is only for command line debugging using human readable CSV files.

[droberts195]

Once all this work is done the command line arguments of autodetect will be roughly the same as those of data_frame_analyzer - compare and contrast https://github.com/elastic/ml-cpp/blob/499338d5fcc16f5a5548e07311973e5381071bc4/bin/autodetect/CCmdLineParser.cc and https://github.com/elastic/ml-cpp/blob/499338d5fcc16f5a5548e07311973e5381071bc4/bin/data_frame_analyzer/CCmdLineParser.cc to see how much cleaner it will be.

[edsavage]

• Add option to C++ to accept job config in JSON format. Completed in [ML] Add argument for config file to autodetect #1540
• Java code to pass path to anomaly job config JSON file to C++. Completed in [ML] Pass job config in JSON file to autodetect elasticsearch#63865
• Parse anomaly job config JSON passed to autodetect.
  • Use jobid extracted from top level of JSON config - [ML] Parse anomaly job JSON config file #1552, [ML] Initialize detectors parser with filters map #1562
  • Extract memory limit and max categorisation examples values from JSON config, pass to CLimits. - [ML] Initialise CLimits from JSON config #1597
  • Extract bucketSpan, summaryMode, summaryCountFieldName, latency and multivariateByFields values from JSON config, pass to CAnomalyDetectorModelConfig. - [ML] Init AD model config from JSON config #1602
  • Encapsulate existing logic in autodetect's Main that creates CAnomalyDetectorModelConfig instance as a CAnomalyJobConfig::makeModelConfig method - [ML] Init AD job config with scheduled events #1607
  • Initialise CAnomalyJobConfig with scheduled events obtained from CFieldConfig (as for filters). Handle any updates to scheduled events. - [ML] Init AD job config with scheduled events #1607
  • Replicate CFieldConfig code that parses clauses and determines function types in CAnomalyJobConfig. - [ML] Parse field clauses using CAnomalyJobConfig #1619
  • Pass CAnomalyJobConfig instance to CFieldDataCategorizer and extract categorizer config from it. - [ML] Use JSON config for categorization #1635
  • Modify Java to write filter lists and scheduled events referenced by detector config to JSON formatted config file. - [ML] Pass extra JSON config files to autodetect elasticsearch#66736 & elastic/ml-json-schemas-private#18
  • Modify C++ code to parse and extract filter lists and scheduled events from JSON config file (Code path to remain disabled until [ML] Pass extra JSON config files to autodetect elasticsearch#66736 merged). - [ML] Parse rule filters and scheduled events from JSON config files #1640
  • Enable filter list and scheduled events parsing code disabled in [ML] Parse rule filters and scheduled events from JSON config files #1640. Remove now unneeded CFieldConfig class entirely (and supporting files) - [ML] Remove CFieldConfig and related files #1644
• Make no longer needed autodetect options optional - [ML] Prepare for removal of unneeded arguments #1648, [ML] Remove the 'no.model.state.persist' setting elasticsearch#67240
• Do not pass redundant command line arguments to autodetect from Java. Remove associated Java code. - [ML] Remove unneeded command line arguments elasticsearch#67344
• Modify autodetect to reject redundant options. - [ML] Remove redundant autodetect options #1663
• Modify Java code to send configuration updates as JSON. Temporarily mute failing tests. - [ML] Send autodetect config updates as JSON elasticsearch#67721
• Modify C++ code to parse autodetect configuration updates in new JSON format. When Java configuration updates are available from the CI artefacts the corresponding C++ code can be merged. - [ML] Parse JSON format config updates #1682
• Re-enable temporarily muted tests in [ML] Send autodetect config updates as JSON elasticsearch#67721 - elasticsearch#67818