Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Copy settings and mappings from last index during rollover #8419

Open
soosinha opened this issue Jul 4, 2023 · 6 comments
Open
Labels
Cluster Manager enhancement Enhancement or improvement to existing feature or request

Comments

@soosinha
Copy link
Member

soosinha commented Jul 4, 2023

Is your feature request related to a problem? Please describe.
When ISM policies are setup to rollover aliases and data streams to new write indices, the mappings are either blank or applied from the matching template. Rollover of aliases and data streams is mostly used for log analytics use cases where the indices are created on a daily basis most commonly. In such cases, customers create the template with static mappings once and then forget about them. Very often, new mappings are added during ingestion but not updated in the template. Due to this, when new index is rolled over using the stale template, new mappings start getting added immediately on ingestion. These dynamic mapping updates cause cluster state updates thereby increasing load on master.

Describe the solution you'd like
Instead of using the mappings from the template, we can copy over the mappings from the last index. This way the new index will not need dynamic mapping updates immediately after creation. Also, customers will not need to keep updating the templates everytime there is some field mapping in their application.
This behavior can be controlled through a setting. When this setting is enabled, the template would be used only for the first index creation. From the second index onwards, the mappings would be copied over from the previous index.

@soosinha soosinha added enhancement Enhancement or improvement to existing feature or request untriaged labels Jul 4, 2023
@soosinha
Copy link
Member Author

soosinha commented Jul 5, 2023

@dblock What are your thoughts on this ?
@getsaurabh02 Does this feature make sense from ISM plugin perspective ?

@dblock
Copy link
Member

dblock commented Jul 6, 2023

I think your suggestion of adding an option to either copy the index mapping from the previous index during rollover, vs. today's default of applying a template, is fairly straightforward and backwards compatible. It will visibly improve performance for these use-cases with no downsides as far as I can see.

@soosinha
Copy link
Member Author

Created an issue in ISM repo for tracking the ISM plugin changes: opensearch-project/index-management#849

@soosinha soosinha changed the title Copy mappings from last index during rollover [FEATURE] Copy settings and mappigns from last index during rollover Jul 11, 2023
@soosinha soosinha changed the title [FEATURE] Copy settings and mappigns from last index during rollover [FEATURE] Copy settings and mappings from last index during rollover Jul 11, 2023
@soosinha
Copy link
Member Author

soosinha commented Jul 12, 2023

Proposed API changes

  1. The ISM policy structure in the ISM plugin will be modified to add a parameter copy_metadata in the rollover operation. This parameter will be passed on to the rollover API changed above.
  2. _rollover API will be modified to have an additional boolean parameter copy_metadata in the request body. If the parameter copy_metadata is true, the settings and mappings from the last index will be copied over to the new index.

Alternate Approach

  1. The ISM policy structure in the ISM plugin will be modified to add a parameter copy_metadata in the rollover operation. If this parameter is true, the ISM rollover operation will pass the settings and mappings of the last index to the rollover API.
  2. There will be no external facing changes in the _rollover API. It already has settings and mappings parameters in the request body which will be used by the ISM rollover operation.

@shwetathareja
Copy link
Member

This not just helps with performance but also prevent mapping conflicts where same field gets mapped to different field types during rollover if index template is not defined.

@shwetathareja
Copy link
Member

cc: @rwali-aws to help with prioritizing the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Cluster Manager enhancement Enhancement or improvement to existing feature or request
Projects
Status: 🆕 New
Development

No branches or pull requests

3 participants