Group field-caps node requests by index mapping hash

[dnhatn]

This optimization is for field-caps requests targeting many indices with an index pattern. Instead of reaching out to many data nodes to retrieve field-caps, we can group indices by their mapping hashes then send a single node request with representative indices instead. This optimization is significant in large clusters.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Hi @dnhatn, I've created a changelog YAML for you.

[javanna]

Do I understand correctly that before we would ask for field_caps for all indices, and eventually get back one set of fields per distinct mapping hash; with this change, we send one single request per mapping hash, and then apply the same response to all indices that have the same hash? This has the advantage of minimizing the amount of roundtrips when many indices have the same mappings: many indices all with the same mappings would likely send one request per data node, and get back the full set of fields once from each node. This is much better than what we did before as we would otherwise repeat the set of fields for every index, resulting in much bigger transport response. Though with this change, we would ask only one node, and get back the full set of fields once from it, so the amount of requests is no longer a function of the number of data nodes (again provided that mappings are the same for all indices involved).

This change is a great improvement, but it would be even better to be able to measure the improvement through benchmarks, relates to #84504 .

@original-brownbear would you mind having a look too given your involvement with the many shards scalability project?

[jtibshirani]

This looks like a great optimization! I left a few small comments but the logic + tests look good to me.

I wonder if Kibana is using index_filter or not, this will really affect when the optimization helps! Unfortunately I think it would require a big change to how we execute field caps in order to get it to work with index_filter...

[original-brownbear]

Sorry for the delay here Nhat! I'm looking at this today and will try to at least manually benchmark this a little.

[original-brownbear]

@dnhatn I benchmarked this against the many shards benchmark setup now. Interestingly it does not provide any (measurable) throughput improvement. My best guess as to why that is, is that we are bottle-necked on the REST side of the network layer in some form (this isn't something that can be fixed here so I wouldn't worry about this).
It does however look like this change measurably reduces the GC pressure (about half as much CPU goes into GC after these changes) and while I was not able to measure the effects of this on the amount of bytes+messages that the transport layer has to handle (not because they're not there, just a matter of the complexity of setting up the experiment), reducing them is always a win in terms of stability.

The before and after of innerMerge (which is effectively all the CPU load on the coordinating node during field_caps) also looks nicer now and is able to compile the capturing lambda key -> new FieldCapabilities.Builder(field, key) better which saves us some CPU on the coordinator.

before:

after:

-> LGTM from my end though I agree with Julie's points on documentation :)

[original-brownbear]

see above

[dnhatn]

@original-brownbear Thank you for running the benchmark. Did you run against all indices or only auditbeat* indices? I see a small improvement (7%) with auditbeat* indices only.

| 50th percentile latency |              field-caps |   4576.27        |   4254.51        |   -321.753   |     ms |    -7.03% |
| 90th percentile latency |              field-caps |   5482.8         |   4805.08        |   -677.729   |     ms |   -12.36% |
| 99th percentile latency |              field-caps |   6001.41        |   5248.21        |   -753.198   |     ms |   -12.55% |
|100th percentile latency |              field-caps |   6241.83        |   5353.3         |   -888.527   |     ms |   -14.24% |

[dnhatn]

@jtibshirani @original-brownbear Thank you for your reviews. I think I have addressed your comments.

I wonder if Kibana is using index_filter or not, this will really affect when the optimization helps! Unfortunately I think it would require a big change to how we execute field caps in order to get it to work with index_filter...

One of the options is to add the can_match phase for requests with index_filter to eliminate unmatched copies as we discussed. We should have a single execution mode in RequestDispatcher after that. I can do this in a follow-up.

[dnhatn]

The optimization introduced in this PR doesn't reduce memory usage or latency of field-caps requests. I will close this PR and try to get #86323 in instead. Thanks everyone for reviewing.