Support pit and search_after in server savedObjects.find

[lukeelmers]

Closes #86301

Summary

Adds support for both pit and search_after to Saved Objects on the server, enabling folks to reliably page through sets of results even if exceeding the index.max_result_window (or more accurately, if exceeding Kibana's maxImportExportSize config value).

Changes

• Adds searchAfter option to SO.find
• Adds pit option to SO.find
• Adds pit_id to SearchResponse
• Adds openPointInTimeForType to SO client, which allows you to pass a type (or list of types), and have a PIT opened against the index/indices backing the type(s)
• Also adds closePointInTime for cleaning up a PIT via SO client (without requiring you to use the Elasticsearch service)
• Updates security/spaces/audit logging to support openPointInTimeForType and closePointInTime
• Integrates pit/searchAfter with SO exports, so you can export objects that exceed the max_result_window default of 10000.
• Adds a PointInTimeFinder which returns a generator to make paging through results easier. Plan is to eventually expose this as a public helper.
• Update sort value with a tiebreaker field

Usage

With find

   const { id } = await savedObjectsClient.openPointInTimeForType(
     type: 'visualization',
     { keepAlive: '5m' },
   );

   const page1 = await savedObjectsClient.find({
     type: 'visualization',
     sortField: 'updated_at',
     sortOrder: 'asc',
     pit: { id },
   });
  
   const lastHit = page1.saved_objects[page1.saved_objects.length - 1];

   const page2 = await savedObjectsClient.find({
     type: 'visualization',
     sortField: 'updated_at',
     sortOrder: 'asc',
     pit: { id: page1.pit_id, keepAlive: '2m' },
     searchAfter: lastHit.sort,
   });
  
   await savedObjectsClient.closePointInTime(page2.pit_id);

Finder helper

 const findOptions: SavedObjectsFindOptions = {
   type: 'visualization',
   search: 'foo*',
   perPage: 100,
 };

 const finder = createPointInTimeFinder({
   logger,
   savedObjectsClient,
   findOptions,
 });

 const responses: SavedObjectFindResponse[] = [];
 for await (const response of finder.find()) {
   responses.push(...response);
   if (doneSearching) {
     await finder.close();
   }
 }

Follow-up tasks (will create a separate issue)

• Documentation for maxImportExportSize
• Update validation of maxImportExportSize to be number instead of bytes (which is incorrect)
• Mention the config option or link to the docs from the error message https://github.com/elastic/kibana/blob/master/src/core/server/saved_objects/export/errors.ts#L30
• Now that we support > 10k exports consider bumping default max to a larger size, as long as it doesn't consume so much memory that Kibana becomes unstable. It will depend a lot on the size of the specific saved objects, but maybe we can test if e.g. 20k objects would really be a problem.
• Expose find w/ PIT generator somewhere as a public helper (perhaps on SO client directly)

[pgayvallet]

Some comments after taking a first look

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[lukeelmers]

@elasticmachine merge upstream

(most of the functional test failures were due to #88737 not being merged yet)

[legrego]

First pass through, mostly questions and nits

[streamich]

AppServices change LGTM.

[lukeelmers]

SavedObject expert ™️

BRB, need to go get a tattoo. 😄

[lukeelmers]

Follow up issues:

• [core.savedObjects] Documentation & cleanup for maxImportExportSize config #91172
• [core.savedObjects] Expose PointInTimeFinder as a helper for paging through SOs #91175

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 6838ab3
• Documentation Changes

Metrics [docs]

✅ unchanged

History

• 💚 Build #106072 succeeded e5815d9
• 💚 Build #105844 succeeded a883a94
• 💚 Build #105845 succeeded bfcd99a
• 💚 Build #105751 succeeded 99b64df
• 💛 Build #105717 was flaky 669ab9e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

Backport result

{"level":"info","message":"POST https://api.github.com/graphql (status: 200)"}
{"level":"info","message":"POST https://api.github.com/graphql (status: 200)"}
{"meta":{"labels":["Feature:Saved Objects","Team:Core","auto-backport","release_note:enhancement","v7.12.0","v8.0.0"],"branchLabelMapping":{"^v8.0.0$":"master","^v7.12.0$":"7.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"},"existingTargetPullRequests":[]},"level":"info","message":"Inputs when calculating target branches:"}
{"meta":["7.x"],"level":"info","message":"Target branches inferred from labels:"}
{"meta":{"killed":false,"code":2,"signal":null,"cmd":"git remote rm kibanamachine","stdout":"","stderr":"error: No such remote: 'kibanamachine'\n"},"level":"info","message":"exec error 'git remote rm kibanamachine':"}
{"meta":{"killed":false,"code":2,"signal":null,"cmd":"git remote rm elastic","stdout":"","stderr":"error: No such remote: 'elastic'\n"},"level":"info","message":"exec error 'git remote rm elastic':"}
{"level":"info","message":"Backporting [{\"sourceBranch\":\"master\",\"targetBranchesFromLabels\":[\"7.x\"],\"sha\":\"7fc3d125bf569636cd4cc45ac9b73ffb8c8733e3\",\"formattedMessage\":\"Support `pit` and `search_after` in server `savedObjects.find` (#89915)\",\"originalMessage\":\"Support `pit` and `search_after` in server `savedObjects.find` (#89915)\",\"pullNumber\":89915,\"existingTargetPullRequests\":[]}] to 7.x"}

Backporting to 7.x:
{"level":"info","message":"Backporting via filesystem"}
{"meta":{"killed":false,"code":1,"signal":null,"cmd":"git cherry-pick 7fc3d125bf569636cd4cc45ac9b73ffb8c8733e3","stdout":"Auto-merging test/api_integration/apis/saved_objects/resolve_import_errors.ts\nAuto-merging test/api_integration/apis/saved_objects/export.ts\nAuto-merging src/plugins/data/server/server.api.md\nAuto-merging src/core/server/server.api.md\nAuto-merging src/core/server/saved_objects/service/lib/repository.ts\nAuto-merging src/core/server/saved_objects/service/lib/repository.test.js\nAuto-merging src/core/public/public.api.md\nAuto-merging docs/development/core/public/kibana-plugin-core-public.doclinksstart.md\nCONFLICT (content): Merge conflict in docs/development/core/public/kibana-plugin-core-public.doclinksstart.md\n","stderr":"error: could not apply 7fc3d125bf5... Support `pit` and `search_after` in server `savedObjects.find` (#89915)\nhint: after resolving the conflicts, mark the corrected paths\nhint: with 'git add <paths>' or 'git rm <paths>'\nhint: and commit the result with 'git commit'\n"},"level":"info","message":"exec error 'git cherry-pick 7fc3d125bf569636cd4cc45ac9b73ffb8c8733e3':"}
{"meta":{"killed":false,"code":2,"signal":null,"cmd":"git --no-pager diff --check","stdout":"docs/development/core/public/kibana-plugin-core-public.doclinksstart.md:20: leftover conflict marker\ndocs/development/core/public/kibana-plugin-core-public.doclinksstart.md:23: leftover conflict marker\ndocs/development/core/public/kibana-plugin-core-public.doclinksstart.md:25: leftover conflict marker\n","stderr":""},"level":"info","message":"exec error 'git --no-pager diff --check':"}
Commit could not be cherrypicked due to conflicts

[bhavyarm]

@lukeelmers how do I test this please? Thanks!

[lukeelmers]

@bhavyarm Ah, sorry - forgot to include testing notes:

1. From Stack Management > Saved Objects, try exporting saved objects & verify it still works (you can also export using the saved objects _export REST API directly)
2. In kibana.yml config, set savedObjects.maxImportExportSize to a number that's >10000
3. Create >10k saved objects (probably easiest to do this via the import api, I have a script that can help with it if you need it -- just let me know).
4. Try exporting again. With this PR you should be able to export >10k saved objects without issues.
5. Also verify that if you set savedObjects.maxImportExportSize to something like 10001, and attempt to export 10002 objects, you should get an error. The export size should never be allowed to exceed that setting.

Also worth mentioning that we do have some functional tests here that verify exporting >10k saved objects works.