[ML] _xpack/usage should return stats about forecasting

[hendrikmuhs]

_xpack/usage currently does not return any stats about forecasts of ml jobs. Information of interest:

• general usage (overall, per job)
• resource usage
  • memory
  • runtime
• counts
  • duration
  • data points

[elasticmachine]

Pinging @elastic/ml-core

[tsullivan]

Given that this is more info that what we're displaying in the Monitoring dashboards, it would be great if we can get this data is in to the monitoring indices for telemetry, but use it for charting in the Monitoring UI as well.

[hendrikmuhs]

After some investigation, I suggest to add forecast stats to _xpack/ml/anomaly_detectors/{jobid|_all}/_stats which could look like this:

Note: This is just an example for the job stats integration, categories of interest are to be decided. From there it is easy to expose it to _usage and monitoring.

An alternative solution would be forecast own _stats endpoint.

@sophiec20
CC @elastic/ml-core

[sophiec20]

Interesting, I like the idea of forecast_stats. Team discussion required, for API and UI (as stats are shown in the Job List).

[hendrikmuhs]

I checked:

The UI (I guess you mean the Counts tab) is not affected, it looks like it has a an overlay (CC @peteharverson) over the results from the job stats API. So fortunately there is no immediate action required for the UI. Longer term it would be nice to make use of these new stats, of course.

The main question, which API endpoint:

1. Proposal, reuse: _xpack/ml/anomaly_detectors/{jobid|_all}/_stats

2. New: _xpack/ml/forecast_stats/{jobid|_all}

3. New: _xpack/ml/forecast/{jobid|_all}/_stats

4. New: _xpack/ml/anomaly_detectors/{jobid|_all}/_forecast_stats

Note: The way forecast is triggered is _xpack/ml/anomaly_detectors/{jobid|_all}/_forecast, therefore option 2 and 3 are in my opinion not a good choice, except we want to change triggering as well.

Interesting, I like the idea of forecast_stats

Which of the above choices match that?

I collect a couple more material and then schedule a team discussion but happily receive some more feedback here, maybe there are more options than the above?

[stevedodson]

1. Proposal, reuse: _xpack/ml/anomaly_detectors/{jobid|_all}/_stats

+1 for reusing the endpoint

[davidkyle]

Can we add units to the forecast memory stats (min, max, avg, sum) I presume they are bytes

[droberts195]

Can we add units to the forecast memory stats (min, max, avg, sum) I presume they are bytes

There are two ways that suggestion can be interpreted: add units to the names or add units to the values. For consistency with model_size_stats and data_counts I think _bytes should be added to the field names rather than units added to the field values. If we were starting from scratch I think it would be better to add units to the field values, but doing that in this case would create an inconsistency with other fields returned by the same endpoint.

[hendrikmuhs]

@tsullivan Regarding your request:

Given that this is more info that what we're displaying in the Monitoring dashboards, it would be great if we can get this data is in to the monitoring indices for telemetry, but use it for charting in the Monitoring UI as well.

As far as I can see it, the change in #31647 automatically adds the new telemetry data into monitoring as well, because monitoring indexes the jobstats object as is. Querying the .monitoring-es... indexes contains the added data points. So this part is fine.

For follow up, I suggest a new issue. I think there are several follow up items. Short term (probably easy to implement) would be to add the number of forecasts to the already existing Jobs table. Long term I would like to see charts, but 1st for jobs, 2nd for forecasts.