Investigate migrating from native ANN search at weight.scorer create level to query.rewrite leve

[jmazanec15]

Description

Currently, the ANN search is executed at KNNWeight.scorer() stage. Scorers are created at the per segment level. The scorer returned more or less just iterates over the matches that are returned.

In Lucene, they do the Ann search the query rewrite phase. This will build scorers for each segment, but limit the number of hits for the shard to k.

I want to investigate feasibility of migrating to ANN search execution at rewrite state in the plugin. Off the top of my head, it has the following benefits:

1. Number of results returned per shard are always k. For the plugin, we will do num_segments*k results for matching. Just returning k results will therefore reduce the search behavior dependency on the number of segments. This would let us also limit the number of re-score operations that would take place as well as part of [RFC] Optimized Disk-Based Vector Search #1779
2. Ability to share state between segments during search. Lucene introduced a really cool feature in https://github.com/apache/lucene/pull/12962/files that relies on collectors in order to share best hit information between segments. With this, segment count can be less of a factor for performance. The shared state across segments is implemented via the KnnCollector interface. This may be possible via creating something new in the KNNWeight class, but would require duplicate effort
3. On duplicate effort, moving to follow convention of Lucene to execute search at re-write would give us the ability to more easily migrate fully to the KnnVectorFormat provided by Lucene (see Investigate migrating custom codec from BinaryDocValuesFormat to KnnVectorsFormat #1087).

That being said, I still need to figure out what the downsides may be. At the moment, I cannot think of any, but will update here if I find any.

In order to do this migration, I think we would need to

1. Build PoC - see what tests break
2. A/B performance test the PoC
3. Feature compatibility deep dive (concurrent segment search, neural search, hybrid search, etc.)