Fix ILM history index settings #61880
Conversation
probakowski
added
>bug
:Data Management/ILM+SLM
|
Pinging @elastic/es-core-features (:Core/Features/ILM+SLM) |
| //wait for all history items to index to avoid waiting for timeout in ILMHistoryStore beforeBulk | ||
| Thread.sleep(6000); |
There was a problem hiding this comment.
why would beforeBulk timeout? would removing the policy from the test index and/or stopping ILM solve the issue of outstanding ilm-history entries? (we're manually rolling over the ilm-history alias so ILM is only needed to trigger the creation of the first history index)
There was a problem hiding this comment.
I've changed it, I stop ILM to reduce amount of history items to index and I wait for the ones that are created. It should be better than explicit wait. WDYT?
There was a problem hiding this comment.
Thanks for fixing this @probakowski
I think it's nearly done, just one more note on the test as I think it might yield some flakiness
| //wait for all history items to index to avoid waiting for timeout in ILMHistoryStore beforeBulk | ||
| assertBusy(() -> { | ||
| SearchResponse search = client().prepareSearch(firstIndex).setQuery(matchQuery("index", firstIndex)).setSize(0).get(); | ||
| assertThat(search.getHits().getTotalHits().value, greaterThan(0L)); |
There was a problem hiding this comment.
should we avoid the randomisation of the ILM policy and use an empty hot phase only ILM policy? that way we can assert exactly how many items must be inserted before moving on with the test tear down. I think there might be another race condition possible otherwise as the stop ILM call is not a hard stop but has to be detected and depending on what step we're in we might execute more steps (see https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/IndexLifecycleService.java#L133 )
There was a problem hiding this comment.
You may be right there is still flakiness there. I've simplified policy, wait for exact number of messages to be indexed and wait for ILM to be stopped
|
@elasticmachine update branch |
This change fixed settings accidentally changed in elastic#61457, this fixes ability of ILM history index to rollover. It also adds test to make sure we can't hit this kind of bug in the future. Closes elastic#61863
This change fixed settings accidentally changed in #61457, this fixes ability of ILM history index to rollover.
It also adds test to make sure we can't hit this kind of bug in the future.
Closes #61863