[Monitoring] Dot prefixed indices are ignored for Alerts

[simianhacker]

Kibana version:

master

Elasticsearch version:

master

Original install method (e.g. download page, yum, from source, etc.):

source

Describe the bug:

I was testing alerting with some data from Agent, specifically the "Shard size" alert. I set the alert to a ridiculously low value to ensure it would still fire. When it didn't fire I started to investigate and realized that we are ignoring dot prefixed indices. This seems like a huge hole in our alerting since dot prefixed indices can grow as well.

Steps to reproduce:

1. Index data from Elastic Agent
2. Edit the "Shard size" alert (via setup mode on the overview page) and set the size to 0.00000000000001
3. The alert will never fire

Expected behavior:

I would expect that alerts are run on all indices regardless of the name when the index pattern is set to *. I understand that dot prefix indices are technically "hidden" but they are also resource that can grow and have issues.

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[igoristic]

@simianhacker This was done intentionally: #89410 (comment), but you have a valid point.

Maybe a better UX would be to default to something like *,-.*, so there's still the option to include internal indices if needed.

cc: @ravikesarwani

[ravikesarwani]

In my mind alerts needs to be actionable by the users. Indices starting with "." are internal indices that we as Elastic own and not something that the solution users should really care about. If we are really growing our internal indices to more than the recommended size (50 GB, but we are using 55 GB default to account for 10% grace) then we should fix that rather than alert users about it.

[igoristic]

@ravikesarwani I see your point, but the user can still control that by proxy, eg: to control .monitoring-* index size they can change: xpack.monitoring.collection.interval and xpack.monitoring.history.duration. Other apps (that are managing internal indices), will probably have similar settings/options

[igoristic]

@ravikesarwani @simianhacker I'm going to add the *,-.* default. Let me know if there are any objections

[ravikesarwani]

I am good with that. It makes explicit that we are excluding dot prefixed indices by default.