[Fleet] Improve pagination experience for >10k agents

[nchaulet]

Description

Pagination for agent is broken after 10k (or whatever the index.max_result_window setting is for the cluster) agents. Elasticsearch offers various ways to paginate search results, can we use one of them to improve this experience?: https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html

Tasks

• Improve UX for >10k agents; ideally there would be no error, and UI can display all agents beyond max result setting
  • This should be done using the point in time API provided by Elasticsearch. (note: below there was a discussion using saved objects' support for this, but these objects are no longer stored as saved objects so it's no longer relevant)
• Remove "First 10,000" limitation on UI
• Investigate if there is a similar limitation on backend when performing bulk actions (Jen doesn't think so, as we offer a "select all agents beyond 10k" option to perform actions against, but double check just in case)

How to reproduce

enroll 10k agents try to navigate to the last page of the agent listing

the details of the error

Result window is too large, from + size must be less than or equal to: [10000] but was [10260]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

Actually the error already exists with saved objects

[ph]

I think we have fixed this issue for something else @jen-huang ?

[jen-huang]

I think this is a known limitation? In the UI we display Showing 10,000 out of X agents if there are >10k agents. Maybe there is a small bug where the last page page does try to retrieve more than the first 10k, though.

[pzl]

Here is the SavedObject ticket reference: #77961

endpoint table follows the Agent pattern here of capping at 10k and showing that message ( PR: #82889 )

[ph]

The 10k limitation is nothing new, there is nothing easily done on our side.
I suggest we look if there is a small bug that cause the toast to appear.

I will put that issue on the back burner until we have a better proposal from kibana platform.

@mostlyjason Just to makes sure you know this.

[ruflin]

We had a discussion around tables and too many entries with @jasonrhodes and there are alternative options here. Maybe we need to change the design? What is the use case that a user would jump to page 502? Perhaps we only need to know the total number and allow the users to browse through the first 10 pages, have better query etc.?

[lukeelmers]

I will put that issue on the back burner until we have a better proposal from kibana platform.

Just thought I'd chime in with an update from the Core team:

We did recently merge #89915 which provides a way to page through >10k saved objects. The main difference is that using Point-In-Time with search_after means you don't really have a way to jump to a particular page as you need the last hit's sort value on the current page in order to retrieve the next page of results. This would require you to make it more of an infinite-scroll experience where you'd need to page using prev/next buttons, a bit like Github does when paging through commit history.

If you plan to go this route, you may want to wait until #91175 lands, as that will provide an even easier mechanism for paging (expecting a PR for that to be up in the coming days).

There are usage examples in the linked PR, but TLDR it will probably look something like this:

 const findOptions: SavedObjectsFindOptions = {
   type: 'visualization',
   search: 'foo*',
   perPage: 100,
 };

 const finder = savedObjectsClient.createPointInTimeFinder({ findOptions });

 const responses: SavedObjectFindResponse[] = [];
 for await (const response of finder.find()) {
   responses.push(...response);
   if (doneSearching) {
     await finder.close();
   }
 }

[jasonrhodes]

Relevant conversation here: elastic/eui#4506

@lukeelmers do you mind weighing in on that EUI issue too? I think many of us are coming across similar problems at the same time.

[joshdover]

@juliaElastic For pagination, I think we should use a fresh PIT when the user is on page 1 each time the page refreshes. If the user starts going through the pages, maybe we should stop refreshing the PIT and reuse the last one. So essentially new agents will only show up if the user is on page 1, all other pages will use the same PIT and get the same view.

[juliaElastic]

@joshdover Thanks, this is a good idea for making sure the user sees a consistent view when paginating. Though it might be problematic if the user leaves the view on page 2, 3, etc. and doesn't see the list refreshing again.

However, this doesn't solve the current issue of users not able to paginate to pages > 501.
I think it could work like this:

• display up to 500 pages in agents list at first
• when on page 500, and the user clicks on next page button (>), the query could use search_after by using the last value from page 500 to fetch the next page
• the UI would have to show page 501 as current page then
• this can continue to page 502, 503, etc.
• for back button (<), the UI has to cache the last values from page 500, 501, etc. to be able to query the right page data again with search_after.

I think this is quite complex to implement, especially changing the UI pagination to customize page display and behavior of next/back buttons.

Alternatively, we could limit the page display to 500 and suggest users to use filtering and sorting to reduce the dataset.
I think this could give more value than implementing the pagination. Realistically, I don't think many users would actually go through more than 500 pages manually.

What do you think? If we decide to implement the more complex option, I think we would have to re-estimate the effort as it might delay the other scalability work.

cc @jen-huang

[nimarezainia]

@juliaElastic i tend to agree with you. I would say even that it's a human limitation to flick through more than 20 pages. Limiting to 500 is ok IMO. Also if we can move forward with scale testing and this becomes an issue we can revisit.

At higher scales operations are more likely done via API. In the UI sorting and filtering to reduce the number of items to look at.

[juliaElastic]

@jen-huang @kpollich should we close this issue as we decided to limit agent list to 500 pages for now and defer the support for more than 500 pages?

[amolnater-qasource]

Hi @juliaElastic
We have revalidated this feature on latest 8.4 Snapshot and found it working fine.

• After 10k agents user is not able to navigate to next list using >.
• For 500 pages 20 agents are available per page.

Build details:
BUILD: 54194
COMMIT: f94d5ff

Screenshots:

cc: @joshdover
Thanks

[amolnater-qasource]

Hi Team
We have revalidated this feature on latest 8.4 BC1 Kibana cloud environment and found it working fine.

Observations:

• After 10k agents user is not able to navigate to next list using >.
• For 500 pages 20 agents are available per page.

Build details:
BUILD: 54999
COMMIT: 58f7eaf

Screenshot:

Hence, we are marking it as QA: Validated.
Thanks