[APM] Add range query to terms enum calls

[dgieselaar]

We don't add an index filter to all of our terms enum calls. This can lead to slowness if the request hits a node in the cold or frozen tier. We should always add an index filter, either with a range query with a reasonable range (e.g. the last 24 hours) or a must_not on _tier: data_cold, data_forozen.

Usages of terms enum without range query

kibana/x-pack/plugins/apm/server/routes/services/get_services/get_service_names_from_terms_enum.ts

Line 27 in d8628fb

'get_services_from_terms_enum',

kibana/x-pack/plugins/apm/server/routes/suggestions/get_suggestions_with_terms_enum.ts

Line 28 in 320abce

const response = await apmEventClient.termsEnum('get_suggestions', {

Related https://github.com/elastic/sdh-beats/issues/3467 (internal)

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[gbamparop]

Can't we use the same time range that is selected in the UI as get_suggestions does? This can still hit cold / frozen data but it'll be consistent with the filters

[sorenlouv]

It feels like a very easy fix to exclude the frozen/cold tier or add a time range filter. Prioritise for 8.9?

[dgieselaar]

@gbamparop @sqren in both cases, sgtm. Btw, the sdh that triggered the creation of the issue turned out to be a Fleet performance bug.

[gbamparop]

It feels like a very easy fix to exclude the frozen/cold tier or add a time range filter. Prioritise for 8.9?

I would suggest 8.10 by looking at our current workload and how much time is left for FF.

[achyutjhunjhunwala]

Storage Explorer was the only missing page where the range query has now been successfully added