-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Copy settings and mappings from last index during rollover #8419
Comments
@dblock What are your thoughts on this ? |
I think your suggestion of adding an option to either copy the index mapping from the previous index during rollover, vs. today's default of applying a template, is fairly straightforward and backwards compatible. It will visibly improve performance for these use-cases with no downsides as far as I can see. |
Created an issue in ISM repo for tracking the ISM plugin changes: opensearch-project/index-management#849 |
Proposed API changes
Alternate Approach
|
This not just helps with performance but also prevent mapping conflicts where same field gets mapped to different field types during rollover if index template is not defined. |
cc: @rwali-aws to help with prioritizing the fix. |
Is your feature request related to a problem? Please describe.
When ISM policies are setup to rollover aliases and data streams to new write indices, the mappings are either blank or applied from the matching template. Rollover of aliases and data streams is mostly used for log analytics use cases where the indices are created on a daily basis most commonly. In such cases, customers create the template with static mappings once and then forget about them. Very often, new mappings are added during ingestion but not updated in the template. Due to this, when new index is rolled over using the stale template, new mappings start getting added immediately on ingestion. These dynamic mapping updates cause cluster state updates thereby increasing load on master.
Describe the solution you'd like
Instead of using the mappings from the template, we can copy over the mappings from the last index. This way the new index will not need dynamic mapping updates immediately after creation. Also, customers will not need to keep updating the templates everytime there is some field mapping in their application.
This behavior can be controlled through a setting. When this setting is enabled, the template would be used only for the first index creation. From the second index onwards, the mappings would be copied over from the previous index.
The text was updated successfully, but these errors were encountered: