[ML] Set data recognizer to ignore frozen indices

[qn895]

Summary

Addresses #116696. Currently, it will take a long time to get results from /api/ml/modules/recognize especially if the index pattern includes indices that are frozen. This PR sets ignore_throttled to true as we would not need to search for frozen. This should speed up the request.

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[kibanamachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 89ea248
• Storybooks Preview

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895

[peteharverson]

Code LGTM

[jgowdyelastic]

LGTM

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

[droberts195]

It's just been pointed out to me that the default for ignore_throttled is true. Therefore we shouldn't need to explicitly set it to true.

pattern includes indices that are frozen

I suspect that "frozen" in the context that caused the problem to be observed meant frozen tier aka searchable snapshots, and not frozen indices (which is what ignore_throttled relates to).

ignore_throttled and frozen indices are deprecated in 8.0, so unless there is more to this than meets the eye I think this PR should be reverted from both main and 8.0. But hold off doing this until next week to give me time to look more closely at the code.

If the problem really was with frozen tier aka searchable snapshots then the solution is instead to change the data recognizer to only search data from the most recent 30 or 90 days. Ideally this would be the most recent N days defined by the ILM policy that moves data into the frozen tier after N days. However, until there is an API that can answer the question of what that N is we'll have to go with a hardcoded 30 or 90 based on common practice for when data is moved to frozen.

[droberts195]

Today I learnt about elastic/elasticsearch#69288.

Frozen tier (aka searchable snapshots) is the modern way to manage old data and is completely different to frozen indices which is old functionality that's being deprecated in 8.0.

I think instead of setting ignore_throttled you should add a terms query on the term _tier looking for the values content, hot or warm. This means the data recoginzer will ignore the cold and frozen tiers. We can justify this by saying that if you haven't seen a particular type of data for ages then there's no point setting up ML jobs for it even though you saw it once.