Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core.savedObjects] Expose PointInTimeFinder as a helper for paging through SOs #91175

Closed
lukeelmers opened this issue Feb 11, 2021 · 2 comments · Fixed by #92981
Closed

[core.savedObjects] Expose PointInTimeFinder as a helper for paging through SOs #91175

lukeelmers opened this issue Feb 11, 2021 · 2 comments · Fixed by #92981
Assignees
Labels
chore Feature:Saved Objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@lukeelmers
Copy link
Member

In #89915 we introduced a PointInTimeFinder to make it easy to iterate over large numbers of saved objects using pit and search_after.

Currently this helper is only used internally by the SavedObjectsExporter, however we think it would be beneficial to expose this somewhere public as the logic is easy to mess up (for example, forgetting to close a PIT right after you are done with it could have detrimental effects on your cluster).

Depending on where it lives, the usage may need to be adjusted somewhat, but my immediate thought would be to put it in the saved objects client itself:

 const findOptions: SavedObjectsFindOptions = {
   type: 'visualization',
   search: 'foo*',
   perPage: 100,
 };

 // instead of requiring a logger we can probably inject one,
 // and we'll already have access to `find` from the client itself
 const finder = savedObjectsClient.createPointInTimeFinder({ findOptions });

 const responses: SavedObjectFindResponse[] = [];
 for await (const response of finder.find()) {
   responses.push(...response);
   if (doneSearching) {
     await finder.close();
   }
 }

An alternative would be exporting it as a static helper as it is currently used internally, which would require a logger and savedObjectsClient to be provided by the consumer. However, I think it might be nice to be able to keep the logs together under the savedobjects-service context, rather than having them attached to any arbitrary logger a consumer provides.

WDYT @pgayvallet @rudolf?

@lukeelmers lukeelmers added chore Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:Saved Objects labels Feb 11, 2021
@lukeelmers lukeelmers self-assigned this Feb 11, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@pgayvallet
Copy link
Contributor

pgayvallet commented Feb 12, 2021

An alternative would be exporting it as a static helper as it is currently used internally

I strongly think that exposing this an an API from the SO client is way better that a static helper.

The suggested

const finder = savedObjectsClient.createPointInTimeFinder({ findOptions });

looks great to me. We may just want to restrict some of the options of SavedObjectsFindOptions for this API, as manually providing a pitId wouldn't make any sense for example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore Feature:Saved Objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants