[Enhancement] Automate updating async tasks status for BATCH_PREDICTION

[Zhangxunmt]

What is the bug?
Starting from OpenSearch 2.17, the async task type "BATCH_PREDICTION" relies on the "Get Task" API to update its status. This dependency means that the task status may become outdated if the Get Task API is not frequently invoked. As a result, tasks may remain incorrectly in the "RUNNING" status, which can cause issues with the task rate limiter. The rate limiter depends on an accurate count of tasks in the "RUNNING" state, and an excess of these tasks can lead to request throttling.

To prevent tasks from remaining in the wrong status, it’s essential to track the status of "BATCH_PREDICTION" tasks promptly within ML Commons. This will help avoid incorrect status counts and ensure that the rate limiter functions efficiently.

How can one reproduce the bug?
Create "BATCH_PREDICTION" task and it stays in "Created" or "Running" status forever without invoking "Get Task" API.

What is the expected behavior?
The "BATCH_PREDICTION" tasks should update status on its own.

[pyek-bot]

Is this related to this: #3161?

[Zhangxunmt]

Yes it's the same issue. I didn't notice this was created. Will combine it with the original one and let Bhavana to handle this issue.