[Reporting] use point-in-time for paging search results

[tsullivan]

Summary

Closes #143144

Based on de10eae

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

Reviewer guide

See documentation on paging search results:

The search response includes an array of sort values for each hit. ... To retrieve the next page of results, repeat the request, take the sort values from the last hit, and insert those into the search_after array:

The open point in time request and each subsequent search request can return different id; thus always use the most recently received id for the next search request.

The search response includes an array of sort values for each hit. If you used a PIT, a tiebreaker is included as the last sort values for each hit. This tiebreaker called _shard_doc is added automatically on every search requests that use a PIT. The _shard_doc value is the combination of the shard index within the PIT and the Lucene’s internal doc ID, it is unique per document and constant within a PIT.

Release Note

Fixed a bug with CSV export in Discover, where searching over hundreds of shards would result in an incomplete CSV file.

[majagrubic]

This should be enforced in search source, before setting the pit.

[tsullivan]

we chatted a bit to clarify this, and agreed it's better to just get rid of this comment.

[jloleysens]

Great job @tsullivan ! I'm glad to see we can employ the recommended way of deeply paging through data for CSV exports 👏🏻

---

Tesed locally on a small-ish sample - do you have any data on how this performs in Kibana with larger datasets?

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: b03f445
• Cloud Deployment

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	401.3KB	401.3KB	+43.0B

Unknown metric groups

API count

id	before	after	diff
data	3261	3263	+2

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	58	64	+6
osquery	108	113	+5
securitySolution	440	446	+6
total			+19

References to deprecated APIs

id	before	after	diff
discover	25	30	+5
reporting	5	0	-5
total			-0

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	66	72	+6
osquery	109	115	+6
securitySolution	517	523	+6
total			+20

History

• 💚 Build #85716 succeeded de821a8
• 💚 Build #85637 succeeded 1905042
• 💔 Build #85625 failed c5710b1
• 💚 Build #85563 succeeded a46385e
• 💛 Build #85318 was flaky dcccbb5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream