Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core.savedObjects] Documentation & cleanup for maxImportExportSize config #91172

Closed
lukeelmers opened this issue Feb 11, 2021 · 3 comments · Fixed by #94019
Closed

[core.savedObjects] Documentation & cleanup for maxImportExportSize config #91172

lukeelmers opened this issue Feb 11, 2021 · 3 comments · Fixed by #94019
Assignees
Labels
chore docs Feature:Saved Objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@lukeelmers
Copy link
Member

In #89915 we changed the way the savedObjects.maxImportExportSize config is applied to SO exports.

Our default value for this is 10k because that's the default index.max_result_window in ES, which limits the number of hits you can page through.

In #89915 we started using pit with search_after, enabling us to page through any arbitrary number of hits. The immediate benefit this has is allowing users to import/export >10k saved objects if they set a higher number in maxImportExportSize.

Since how this config is used has been a source of confusion in the past, and to make life easier from a support perspective, we should work on some documentation & cleanup to make it more obvious how this works:

  • Add documentation for maxImportExportSize, and when it is applied.
  • If the limit is exceeded, mention the config option or link to the docs from the error message https://github.com/elastic/kibana/blob/master/src/core/server/saved_objects/export/errors.ts#L30
  • Now that we support > 10k exports consider bumping default max to a larger size, as long as it doesn't consume so much memory that Kibana becomes unstable. It will depend a lot on the size of the specific saved objects, but maybe we can test if e.g. 20k objects would really be a problem.
  • Update validation for maxImportExportSize: this is just a number, but is improperly validated as bytes in
    export const savedObjectsConfig = {
    path: 'savedObjects',
    schema: schema.object({
    maxImportPayloadBytes: schema.byteSize({ defaultValue: 26214400 }),
    maxImportExportSize: schema.byteSize({ defaultValue: 10000 }),
    }),
    };
  • Consider whether we could easily update core usage data, which confusingly labels this as maxImportExportSizeBytes even though it isn't bytes.
@lukeelmers lukeelmers added chore Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:Saved Objects docs labels Feb 11, 2021
@lukeelmers lukeelmers self-assigned this Feb 11, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@rudolf
Copy link
Contributor

rudolf commented Feb 12, 2021

I think a byte value might make some sense, so I can see value in either option:

  1. Saved Object count (number)
    This is simple to understand for users. "I have 20k saved objects, so I need to bump the limit to 20k to export them". It makes it easy to choose the smallest value you would need, but not the largest value that's safe to use. You could for instance choose 100k but there's no way to answer the question "will exporting 100k documents crash my server?"
  2. Total export size in bytes (or just collapse maxImportPayloadBytes and maxImportExportSize into maxImportExportPayloadBytes)
    Although we were previously limited by Elasticsearch's 10k hits limit, now that that problem has been solved the limit is only there to protect Kibana from running out of memory (I think?). Exporting 10MB will consume more than 10MB heap, but we could probably come up with a rule of thumb like "ensure that the node process has at least 3*maxImportExportPayloadBytes heap available"

@pgayvallet
Copy link
Contributor

Total export size in bytes (or just collapse maxImportPayloadBytes and maxImportExportSize into maxImportExportPayloadBytes

I would rather go with Saved Object count (number), as calculating the exported objects total payload may be slightly tricky

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore docs Feature:Saved Objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants