[SOM] Fix UI and export when handling more than 10k objects #118335

pgayvallet · 2021-11-11T13:35:38Z

Summary

Fix #116776

Adapt the /api/kibana/management/saved_objects/scroll/counts endpoint to use PIT search
Adapt the SOM UI to improve experience when displaying more than 10k objects
- limit the results in the table to 10k
- when more than 10k results are returned, display a message stating that we're only displaying the first 10k results
Improve the toast error message displayed to the user when the export fail by displaying the error message as returned by the server
(cleanup) Remove the /api/kibana/management/saved_objects/scroll/export endpoint that was unused (was used by the legacy export that was removed long ago)

Screenshots

Displaying 12k vis

Toast notification in case of failure

Checklist

Documentation was added for features that require explanation or tutorials

Release Note

Fix a bug in the saved objects management section causing the export of more than 10k objects to fail.

pgayvallet

self-review

pgayvallet · 2021-11-11T16:34:03Z

test/api_integration/apis/saved_objects_management/scroll_count.ts

+        for (let i = startIdx; i <= endIdx; i++) {
+          const id = `test-vis-${i}`;
+          fileChunks.push(
+            JSON.stringify({
+              type: 'visualization',


I initially created a file with 10k entries with a script and used kibanaServer.importExport API, but the file is quite big and the API only accepts uncompressed files.

Also, this is how this is done for the export and import test suites too:

kibana/test/api_integration/apis/saved_objects/export.ts

Lines 277 to 281 in 7917e3c

fileChunks.push(

JSON.stringify({

type: 'visualization',

id: `${SPACE_ID}-${i}`,

attributes: {

pgayvallet · 2021-11-11T16:35:04Z

test/api_integration/apis/saved_objects_management/scroll_count.ts

+        await importVisualizations({ startIdx: 1, endIdx: 6000 });
+        await importVisualizations({ startIdx: 6001, endIdx: 12000 });


As the import limit is 10001, I'm imported the 12k in two batches. This is also an additional check that the scroll_count API works when there are more objects than the limit in the search results.

pgayvallet · 2021-11-11T16:36:08Z

test/api_integration/apis/saved_objects_management/scroll_count.ts

+    describe('scroll_count with more than 10k objects', () => {
+      const importVisualizations = async ({
+        startIdx = 1,
+        endIdx,


Only added FTR tests for this API because export and import were already properly working and are covered, e.g

kibana/test/api_integration/apis/saved_objects/export.ts

Lines 525 to 527 in 7917e3c

it('should allow exporting more than 10,000 objects if permitted by maxImportExportSize', async () => {

await supertest

.post(`/s/${SPACE_ID}/api/saved_objects/_export`)

pgayvallet · 2021-11-11T16:37:21Z

...lugins/saved_objects_management/public/management_section/objects_table/components/table.tsx

@@ -74,6 +74,8 @@ interface TableState {
  activeAction?: SavedObjectsManagementAction;
 }

+const MAX_PAGINATED_ITEM = 10000;


Not a big fan of this hardcoded limit, OTOH that's how it's done in other parts of Kibana too, and it's way easier than fetching the setting from all the SO indices targeted by the request.

Can an end-user face the problem when requesting /api/kibana/management/saved_objects/scroll/counts? If so, maybe this logic belongs to the server-side?

it can't when requesting /api/kibana/management/saved_objects/scroll/counts, or when performing the export, only when manually calling /api/kibana/management/saved_objects/_find if the requested page/perPage exceeds the threshold (but even if prefixed by /api, there shouldn't be a case where an integration uses this endpoint)

elasticmachine · 2021-11-11T16:37:41Z

Pinging @elastic/kibana-core (Team:Core)

pgayvallet · 2021-11-12T06:43:42Z

@elasticmachine merge upstream

…from-main-this-time

mshustov · 2021-11-15T16:12:56Z

...lugins/saved_objects_management/public/management_section/objects_table/components/table.tsx

@@ -74,6 +74,8 @@ interface TableState {
  activeAction?: SavedObjectsManagementAction;
 }

+const MAX_PAGINATED_ITEM = 10000;


Can an end-user face the problem when requesting /api/kibana/management/saved_objects/scroll/counts? If so, maybe this logic belongs to the server-side?

…from-main-this-time

kibana-ci · 2021-11-16T11:15:35Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`savedObjectsManagement`	82.0KB	82.6KB	+648.0B

History

💚 Build #6643 succeeded 9e0cc8b
💚 Build #6460 succeeded 9e0cc8b
💚 Build #6289 succeeded eec2486
💚 Build #6215 succeeded b573c69
💔 Build #6202 failed c8a4bb0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2021-11-16T11:20:11Z

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

…118335) * delete the unused `scroll/export` endpoint * adapt the scroll/counts endpoint to use PIT search * improve export error message * only display 10k first items * fix findAll unit tests * fix FTR test * remove unused comment Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2021-11-16T11:20:22Z

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

…#118669) * delete the unused `scroll/export` endpoint * adapt the scroll/counts endpoint to use PIT search * improve export error message * only display 10k first items * fix findAll unit tests * fix FTR test * remove unused comment Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Pierre Gayvallet <[email protected]>

…118335) * delete the unused `scroll/export` endpoint * adapt the scroll/counts endpoint to use PIT search * improve export error message * only display 10k first items * fix findAll unit tests * fix FTR test * remove unused comment Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # src/plugins/saved_objects_management/server/routes/scroll_export.ts

…#118718) * delete the unused `scroll/export` endpoint * adapt the scroll/counts endpoint to use PIT search * improve export error message * only display 10k first items * fix findAll unit tests * fix FTR test * remove unused comment Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # src/plugins/saved_objects_management/server/routes/scroll_export.ts

…118335) * delete the unused `scroll/export` endpoint * adapt the scroll/counts endpoint to use PIT search * improve export error message * only display 10k first items * fix findAll unit tests * fix FTR test * remove unused comment Co-authored-by: Kibana Machine <[email protected]>

* delete the unused `scroll/export` endpoint * adapt the scroll/counts endpoint to use PIT search * improve export error message * only display 10k first items * fix findAll unit tests * fix FTR test * remove unused comment Co-authored-by: Kibana Machine <[email protected]>

pgayvallet added 4 commits November 11, 2021 13:52

delete the unused scroll/export endpoint

885e255

adapt the scroll/counts endpoint to use PIT search

60ae369

improve export error message

7833846

only display 10k first items

c8a4bb0

pgayvallet added bug Fixes for quality problems that affect the customer experience Feature:Saved Objects Management Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.0.0 v8.1.0 release_note:fix labels Nov 11, 2021

pgayvallet added 3 commits November 11, 2021 15:02

fix findAll unit tests

e355d5a

fix FTR test

b573c69

remove unused comment

eec2486

pgayvallet commented Nov 11, 2021

View reviewed changes

pgayvallet marked this pull request as ready for review November 11, 2021 16:37

pgayvallet requested a review from a team as a code owner November 11, 2021 16:37

Merge branch 'main' into kbn-116776-SOM-10k-from-main-this-time

9e0cc8b

elastic deleted a comment from kibanamachine Nov 12, 2021

Merge remote-tracking branch 'upstream/main' into kbn-116776-SOM-10k-…

051a489

…from-main-this-time

mshustov approved these changes Nov 15, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/main' into kbn-116776-SOM-10k-…

b502e13

…from-main-this-time

pgayvallet added the auto-backport Deprecated - use backport:version if exact versions are needed label Nov 16, 2021

pgayvallet merged commit 42957d9 into elastic:main Nov 16, 2021

kibanamachine mentioned this pull request Nov 16, 2021

[8.0] [SOM] Fix UI and export when handling more than 10k objects (#118335) #118669

Merged

pgayvallet mentioned this pull request Nov 16, 2021

[7.16] [SOM] Fix UI and export when handling more than 10k objects (#118335) #118718

Merged

pgayvallet added the v7.16.0 label Nov 16, 2021

lukeelmers mentioned this pull request Nov 23, 2021

Saved objects management cannot export if more than 10k saved objects #117527

Closed

pgayvallet mentioned this pull request Dec 9, 2021

Kibana failed to export or filter saved objects if result window is too large #116106

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SOM] Fix UI and export when handling more than 10k objects #118335

[SOM] Fix UI and export when handling more than 10k objects #118335

pgayvallet commented Nov 11, 2021 •

edited

Loading

pgayvallet left a comment

pgayvallet Nov 11, 2021

pgayvallet Nov 11, 2021

pgayvallet Nov 11, 2021

pgayvallet Nov 11, 2021

mshustov Nov 15, 2021

pgayvallet Nov 16, 2021

elasticmachine commented Nov 11, 2021

pgayvallet commented Nov 12, 2021

mshustov Nov 15, 2021

kibana-ci commented Nov 16, 2021

kibanamachine commented Nov 16, 2021

kibanamachine commented Nov 16, 2021

	fileChunks.push(
	JSON.stringify({
	type: 'visualization',
	id: `${SPACE_ID}-${i}`,
	attributes: {

		await importVisualizations({ startIdx: 1, endIdx: 6000 });
		await importVisualizations({ startIdx: 6001, endIdx: 12000 });

	it('should allow exporting more than 10,000 objects if permitted by maxImportExportSize', async () => {
	await supertest
	.post(`/s/${SPACE_ID}/api/saved_objects/_export`)

[SOM] Fix UI and export when handling more than 10k objects #118335

[SOM] Fix UI and export when handling more than 10k objects #118335

Conversation

pgayvallet commented Nov 11, 2021 • edited Loading

Summary

Screenshots

Checklist

Release Note

pgayvallet left a comment

Choose a reason for hiding this comment

pgayvallet Nov 11, 2021

Choose a reason for hiding this comment

pgayvallet Nov 11, 2021

Choose a reason for hiding this comment

pgayvallet Nov 11, 2021

Choose a reason for hiding this comment

pgayvallet Nov 11, 2021

Choose a reason for hiding this comment

mshustov Nov 15, 2021

Choose a reason for hiding this comment

pgayvallet Nov 16, 2021

Choose a reason for hiding this comment

elasticmachine commented Nov 11, 2021

pgayvallet commented Nov 12, 2021

mshustov Nov 15, 2021

Choose a reason for hiding this comment

kibana-ci commented Nov 16, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

History

kibanamachine commented Nov 16, 2021

💚 Backport successful

kibanamachine commented Nov 16, 2021

💚 Backport successful

pgayvallet commented Nov 11, 2021 •

edited

Loading