[CM] SavedObjectFinder server-side pagination

[Dosant]

Old description: Bring back partial response support

When migrating SavedObjectFinder from so.client to content management for backward compatibility reasons, we dropped includeFields functionality which allowed to return partial objects (for example, only title and description instead of full objects). #162904, #161545

This issue is to discuss and investigate if it is worth getting this functionality back in some form but in a backward-compatible way.

• we need to implement a way to decouple includeFields transferred over the network and saved object attributes, so that they can be migrated before performing search

Currently ,without this feature we over-fetch the data, but it is mitigated by the following:

• Only index-pattern, search, maps, lens, annotation-group are supported types. No larger canvas or dashboard
• There is a LISTING_LIMIT advanced setting (1000 is default), can be reduced if someone sees issues

Alternatively, we can reduce the limit, implement server-side search and pagination.

When migrating SavedObjectFinder from so.client to content management for backward compatibility reasons, we dropped includeFields functionality which allowed to return partial objects (for example, only title and description instead of full objects). #162904, #161545

Currently SavedObjectFinder uses a LISTING_LIMIT advanced setting (1000 is default) to fetch the items and then uses client side in-memory sorting and pagination.

This might become a performance problem in some cases, as now the api always returns full objects.
As discussed with @vadimkibana #163043 (comment), a good and the most straightforward solution would be to refactor the SavedObjectFinder to use server-side pagination.

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[vadimkibana]

There is a LISTING_LIMIT advanced setting (1000 is default), can be reduced if someone sees issues

It sounds like the problem is that existing listing pages don't paginate, but load all items? I think we might want to treat this as high priority tech debt and implement pagination, so 10 items (or so) are loaded at a time instead of 1,000.

[Dosant]

It sounds like the problem is that existing listing pages don't paginate, but load all items? I think we might want to treat this as high priority tech debt and implement pagination, so 10 items (or so) are loaded at a time instead of 1,000.

@vadimkibana, I fully agree. I think this makes the most sense and benefits everyone in the future.
We also have places which are intentionally asking for "*" all the fields and this fix will improve those.

I think we also have the same tech debt for the table list view

[kpatticha]

@Dosant if I understand correctly, the plan is to implement a pagination but return the full object.?

For our case in apm #167763 it is reasonable to have the ability to be able to narrow down the result. Especially when is supported by the Saved object API.

[Dosant]

@kpatticha, yes this issue is about the server-side pagination for our components (savedobjectfinder, tablelistview)
The partial objects support needs to be discussed/worked on separately

[kpatticha]

@Dosant thanks, is there any ticket/thread I can follow regarding the partial object support?

[Dosant]

@Dosant thanks, is there any ticket/thread I can follow regarding the partial object support?

No, afaik, we don't have it.

With content management we thought we won't have partial objects, because longer-term by having only full objects we wanted to make client side caching work (e.g. you show a list of dashboards, click on one, and immediately open a dashboard without waiting for another fetch). We hoped we can address slow list fetching concern with proper server-side pagination.

If this won't work for your use-case, then could you please create an issue, so we'd take a look at this again?

Alternatively, if the use-case is client-side / dashboard specific maybe this could be a separate dashboard specific server-side route that returns only ids/titles. (on the server it would still fetch the whole objects from es)