Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ILM history rollover missing alias #61863

Closed
albertzaharovits opened this issue Sep 2, 2020 · 1 comment · Fixed by #61880
Closed

ILM history rollover missing alias #61863

albertzaharovits opened this issue Sep 2, 2020 · 1 comment · Fixed by #61880
Labels
>bug :Data Management/ILM+SLM Index and Snapshot lifecycle management Team:Data Management Meta label for data/management team

Comments

@albertzaharovits
Copy link
Contributor

I've encountered a problem with the rollover of the ilm history index:

[2020-09-02T18:11:43,428][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [ilm-history-3-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] to [{"phase":"hot","action":"unfollow","name":"pause-follower-index"}] in policy [ilm-history-ilm-policy]
[2020-09-02T18:11:43,485][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [auditbeat-7.9.0-2020.09.02-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] to [{"phase":"hot","action":"unfollow","name":"pause-follower-index"}] in policy [auditbeat]
[2020-09-02T18:11:43,554][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [ilm-history-3-000001] from [{"phase":"hot","action":"unfollow","name":"pause-follower-index"}] to [{"phase":"hot","action":"unfollow","name":"close-follower-index"}] in policy [ilm-history-ilm-policy]
[2020-09-02T18:11:43,610][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [auditbeat-7.9.0-2020.09.02-000001] from [{"phase":"hot","action":"unfollow","name":"pause-follower-index"}] to [{"phase":"hot","action":"unfollow","name":"close-follower-index"}] in policy [auditbeat]
[2020-09-02T18:11:43,656][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [ilm-history-3-000001] from [{"phase":"hot","action":"unfollow","name":"close-follower-index"}] to [{"phase":"hot","action":"unfollow","name":"unfollow-follower-index"}] in policy [ilm-history-ilm-policy]
[2020-09-02T18:11:43,708][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [auditbeat-7.9.0-2020.09.02-000001] from [{"phase":"hot","action":"unfollow","name":"close-follower-index"}] to [{"phase":"hot","action":"unfollow","name":"unfollow-follower-index"}] in policy [auditbeat]
[2020-09-02T18:11:43,757][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [ilm-history-3-000001] from [{"phase":"hot","action":"unfollow","name":"unfollow-follower-index"}] to [{"phase":"hot","action":"unfollow","name":"open-follower-index"}] in policy [ilm-history-ilm-policy]
[2020-09-02T18:11:43,804][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [auditbeat-7.9.0-2020.09.02-000001] from [{"phase":"hot","action":"unfollow","name":"unfollow-follower-index"}] to [{"phase":"hot","action":"unfollow","name":"open-follower-index"}] in policy [auditbeat]
[2020-09-02T18:11:43,858][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [ilm-history-3-000001] from [{"phase":"hot","action":"unfollow","name":"open-follower-index"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-yellow-step"}] in policy [ilm-history-ilm-policy]
[2020-09-02T18:11:43,912][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [auditbeat-7.9.0-2020.09.02-000001] from [{"phase":"hot","action":"unfollow","name":"open-follower-index"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-yellow-step"}] in policy [auditbeat]
[2020-09-02T18:11:43,963][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [ilm-history-3-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-yellow-step"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [ilm-history-ilm-policy]
[2020-09-02T18:11:44,028][INFO ][o.e.x.i.IndexLifecycleTransition] [ElasticMBP.local] moving index [auditbeat-7.9.0-2020.09.02-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-yellow-step"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [auditbeat]
[2020-09-02T18:21:43,436][ERROR][o.e.x.i.IndexLifecycleRunner] [ElasticMBP.local] policy [ilm-history-ilm-policy] for index [ilm-history-3-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [ilm-history-3-000001] is empty or not defined
        at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:65) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:174) [x-pack-ilm-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:327) [x-pack-ilm-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:265) [x-pack-ilm-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

while simply using an audit beat which I believe just triggered its own rollover.

@andreidan suggested this was introduced in #61457

@albertzaharovits albertzaharovits added >bug :Data Management/ILM+SLM Index and Snapshot lifecycle management needs:triage Requires assignment of a team area label labels Sep 2, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Sep 2, 2020
@probakowski probakowski removed the needs:triage Requires assignment of a team area label label Sep 2, 2020
probakowski added a commit that referenced this issue Sep 3, 2020
This change fixed settings accidentally changed in #61457, this fixes ability of ILM history index to rollover.
It also adds test to make sure we can't hit this kind of bug in the future.

Closes #61863
probakowski added a commit to probakowski/elasticsearch that referenced this issue Sep 3, 2020
This change fixed settings accidentally changed in elastic#61457, this fixes ability of ILM history index to rollover.
It also adds test to make sure we can't hit this kind of bug in the future.

Closes elastic#61863
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Data Management/ILM+SLM Index and Snapshot lifecycle management Team:Data Management Meta label for data/management team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants