[Logs UI Meta Issue] Create API routes to manage ML categorization jobs

[jasonrhodes]

⚠️ This requirement is obsolete

Summary

The infra app Kibana HTTP API needs to be able to send queries to the ML API in order to create the 3 ML jobs required, to query for results from those jobs when they exist, and to remove/disable the jobs.

The APIs to access the jobs' results are described in #41805.

Acceptance criteria

General properties

• The access of the Elasticsearch API is performed using the logged-in user's credentials to ensure they have the necessary permissions.
• The payload is validated on the server and client.
• Operations on multiple ML entities behave atomically, i.e. either fail or succeed as a whole.
• The payload schemata are consistent (the JSON:API spec could be a useful guide for that):
  • Query parameters are sent in a data field of the request.
  • Results are returned in a data field of the response.

Task breakdown

• Create API route to create ML categorization jobs ([Logs UI] Create API route to create ML categorization jobs  #42017)
• Create API route to check the status of ML jobs ([Logs UI] Create API route to check the status of ML jobs #42018)
• Create API route to remove ML categorization jobs ([Logs UI] Create API route to remove ML categorization jobs #42019)

Original ACs

• Documentation for the endpoints below exists first, here in this ticket
  • Doesn't have to be official, just a list of endpoints and what they require and return, roughly, so we can begin building the next parts of this work based on that contract
• An endpoint exists that will create the 3 required jobs using the information provided about field to categorize, an optional ECS event.dataset value to filter on, and a time range (and the index from the source configuration)
  • Note: I think if possible we should treat these 3 jobs as all-or-nothing for now, and not give users the ability to create or remove only 1 or 2 of the jobs.
• An endpoint exists that will query for the 3 existing jobs and return the 3 sets of data (this can be one endpoint or 3 separate ones, depending on what seems to make sense)
  • Note: we should handle the case where up to 2 of these jobs don't exist gracefully, in case a user has manually removed the jobs. If all 3 are removed, we should treat this case the same way as if the user had never enabled the ML analysis integration and show the onboarding screen again
• An endpoint exists to remove/disable any of the 3 jobs that still currently exist

Note:

• Possible: an endpoint that uses a try/catch method to determine if the current user has the correct permissions to use ML in this way, since the typical capabilities API may not give us that information?

[elasticmachine]

Pinging @elastic/infra-logs-ui

[weltenwort]

I think if possible we should treat these 3 jobs as all-or-nothing for now, and not give users the ability to create or remove only 1 or 2 of the jobs.

I agree, but the user might have removed one of them via the ML UI. The routes except for the removal route should therefore first check for the correct presence of the jobs and return a corresponding error if they don't match the expectations.

[weltenwort]

edit: I moved this comment's content into the main issue description

[jasonrhodes]

This is a good start to the documentation, we can break out the results endpoints to a new ticket.

[Zacqary]

Going to split this into 3 issues, one for each endpoint. I want to make one PR for each of them instead of doing them all at once, so that will make tracking easier

[weltenwort]

Given that the existing module API will be extended to support our requirements (#42409), there is no need for a custom API.