[BUG] index rollover - doesn't create all aliases in the newly created index

[oridool]

What is the bug?
When using ISM rollover and the index contains multiple aliases, only the alias configured for rollover is created in the new index.
Other aliases existing in original index (before rollover) are not copied.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. create ism policy and index template such as:

PUT  _index_template/rollovertests_template
{
  "index_patterns": [
    "rollovertests-*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "plugins.index_state_management.rollover_alias": "rolloveralias"
    },
    "mappings": {
      "_routing": {
      "required": true
      },
      "properties": {
        "timestamp": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        }
      }
    }
  }
}

PUT _plugins/_ism/policies/rollovertests_policy
{
  "policy": {
    "description": "test rollover policy.",
    "default_state": "rollover",
    "states": [
      {
        "name": "rollover",
        "actions": [
          {
            "rollover": {
              "min_index_age": "2m"
            }
          }
        ],
        "transitions": []
      }
    ] 
    ,
    "ism_template": {
      "index_patterns": ["rollovertests*"],
      "priority": 100
    }
  }
}

2. create index with more than one alias:

PUT rollovertests-000001
{
  "aliases": {
    "rolloveralias": {
      "is_write_index": true,
      "filter": {
        "term": {
          "timestamp": "1574641891142"
        }
      },
      "routing": "12345"
    },
    "anotheralias": {
      "filter": {
        "term": {
          "timestamp": "157464189133"
        }
      },
      "routing": "12345"
    }
  }
}

3. After rollover, check the newly created index:

GET rollovertests-000002

Result:

{
  "rollovertests-000002": {
    "aliases": {
      "rolloveralias": {
        "filter": {
          "term": {
            "timestamp": "1574641891142"
          }
        },
        "index_routing": "12345",
        "search_routing": "12345",
        "is_write_index": true
      }
    },...

As you can see, the alias 'anotheralias' is missing from the new index 'rollovertests-000002'.

What is the expected behavior?

The alias 'anotheralias' should be copied to the new index upon rollover.
In fact, all existing aliases with all of their properties (routing, filter,...) should be copied.
The _rollover API supports that by providing the "aliases" part in the body. But the ISM just doesn't call it properly.

As a proof, I can do the rollover manually, and then I get the expected results:

POST rolloveralias/_rollover
{
  "aliases": {
    "anotheralias": {
      "filter": {
        "term": {
          "timestamp": "157464189133"
        }
      },
      "routing": "12345"
    }
  }
}

GET rollovertests-000002

Result:

{
  "rollovertests-000002": {
    "aliases": {
      "anotheralias": {
        "filter": {
          "term": {
            "timestamp": "157464189133"
          }
        },
        "index_routing": "12345",
        "search_routing": "12345"
      },
      "rolloveralias": {
        "filter": {
          "term": {
            "timestamp": "1574641891142"
          }
        },
        "index_routing": "12345",
        "search_routing": "12345",
        "is_write_index": true
      }
    },...

What is your host/environment?

• OS: Linux
• Version 2.5 (AWS Opensearch)

[Angie-Zhang]

Thanks for reporting this bug!

[bowenlan-amzn]

This doesn't seem to be a bug. Like you said, even using rollover API, you need to explicitly define the aliases you want on the new index.

If the alias you want to add is a static one, I think maybe index template is what you want https://opensearch.org/docs/latest/im-plugin/index-templates/. You can define the template to add alias to the matched new index.

But if it's a dynamic one, like I can see timestamp field in the alias, if that's the time of when rollover API is called, then this could be a new feature to add.

[oridool]

@bowenlan-amzn ,
I'll try to better explain why it is a bug.
The timestamp is just an example for a field. As you already observed, this is dynamic alias that can contain any filter and any field.
In my index, I maintain several dynamic aliases - each with a different filter.
When ISM is calling rollover API automatically for me, then it makes no sense to rollover a single alias. Instead, it should rollover (or copy) all aliases of the index.
The ISM plugin should call the existing rollover API properly, with all available options.
If it doesn't do it, then I'll have to copy all aliases by myself, which makes the automatic rollover pretty much useless for the case of multiple aliases.
In case of a single alias, then it will behave as today.

Thanks.

[bowenlan-amzn]

I want to clarify our understanding of the term "dynamic" in this context. I use dynamic to mean that the alias changes during rollover. e.g., during index1 w/ alias -> rollover -> index2 w/ alias, some fields in the alias like timestamp needs to be updated at that time.
However, since you are saying copy the alias to the new index, then it means there are no change during rollover so it doesn't seem to be dynamic to me. Perhaps you could explain further.
If the alias is not dynamic, I recommend you look into using index template to add alias, and this is the standard way people add alias to a group of indices during index creation time.

[oridool]

@bowenlan-amzn ,
When I wrote "dynamic", I didn't mean that alias should be modified during rollover.
I meant that the alias is dynamic and changes over time, before rollover occurs.
I create index aliases and update their filters dynamically, and therefore index template is irrelevant in this case.

Example for index lifecycle:

1. create an index rollovertests-000001 with rollover alias rolloveralias.
2. After some time, add another alias anotheralias to the index.
3. After some more time, update the filter of anotheralias by API.
4. After some more time ISM performs a rollover according to predefined conditions.
5. At this point, I expect both aliases to be part of rollovertests-000002 with the exact properties as they were in rollovertests-000001.

I hope now it is more clear and that you see why it is a bug and not a new feature.
I expect ISM to provide all aliases (if exists) to the rollover API call, just as I demonstrated above. Effectively, it will copy all aliases to the new index.

Thank you.

[bowenlan-amzn]

To move forward, we need to enhance the rollover action https://opensearch.org/docs/latest/im-plugin/ism/policies/#rollover
Enable user to define another field, copy_alias, and default to false to keep existing user's experience un-changed

In the code, before executing rollover https://github.com/opensearch-project/index-management/blob/main/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/step/rollover/AttemptRolloverStep.kt#L116
Add logic to handle coping alias.

@oridool do you want to help implement this function?

[oridool]

@bowenlan-amzn ,
Are you sure we need an additional flag? I think that if someone has created additional aliases for the index then it is the natural thing to copy existing ones into the newly created index. Or at least the flag default should be true IMO.

Unfortunately I'm not familiar with Kotlin or the Opensearch project specifically, and therefore it won't be effective for me to implement it. Appreciate if you can take it or someone else who is more familiar with this project.

Thanks.

[bowenlan-amzn]

I do not believe that the default setting would be the user's preference. In my understanding, an alias serves as a namespace for a set of indices (index patterns). Therefore, if the user wishes to assign an alias to a newly created index within a specific index pattern, this can be achieved through an index template, as previously proposed. It should also be noted that Rollover is an operation performed on an alias in order to create a new index that also conforms to the specified index pattern (since only the suffix number increases).

So in your use case, apart from updating the alias of one index, you can also update the alias in the index template so it works for all the newly created index. Let me know if you see any issues with this approach.

[oridool]

@bowenlan-amzn ,
I thought we already agreed on the correct solution: upon rollover, the ISM job should provide all existing aliases to be copied to the new index.
What you are now suggesting won't work for the use case I wrote.

"an alias serves as a namespace" - not exactly, an alias is like a view on the index (if I try to compare OpenSearch to RDBMS).

"this can be achieved through an index template" - this is not correct. In index template you can only provide a static pre-configured alias, while some people create the aliases after the index is created or modifying an existing alias filter.

Please let's go ahead with what you already offered here: #734 (comment)