[CI] DownsampleActionIT testRollupNonTSIndex failing #103981
Assignees
Labels
low-risk
An open issue or test failure that is a low risk to future releases
:StorageEngine/Downsampling
Downsampling (replacement for rollups) - Turn fine-grained time-based data into coarser-grained data
Team:StorageEngine
>test-failure
Triaged test failures from CI
Comments
kingherc
added
:StorageEngine/Rollup
elasticsearchmachine
added
blocker
needs:triage
wchaparro
added
Team:StorageEngine
and removed
needs:triage
martijnvg
added
low-risk
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this issue
Feb 5, 2024
and add more logging for when test fails next time. Relates to elastic#103981
martijnvg
added a commit
that referenced
this issue
Feb 5, 2024
and add more logging for when test fails next time. Relates to #103981
|
Failed again at https://gradle-enterprise.elastic.co/s/4hzqteph2qrpo, reinstating the mute. |
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this issue
Mar 5, 2024
increase ILM logging to TRACE. Relates to elastic#103981
martijnvg
added a commit
that referenced
this issue
Mar 5, 2024
increase ILM logging to TRACE. Relates to #103981
|
No reported failures since the last ~2 weeks, which was when #106563 was merged. I think that change stabilized this test as well. |
elasticsearchmachine
pushed a commit
that referenced
this issue
Apr 15, 2024
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
|
There was a single failure over the past week, over more than 2k runs. Re-enabling the test for coverage, keeping an eye for further failures. |
|
No more failures over the past week. Closing it for now. |
|
Seems like a race condition: Waiting for longer should allow for things to work. |
elasticsearchmachine
pushed a commit
that referenced
this issue
Apr 24, 2024
look_ahead_time is 1m, need to wait for longer than that. Related to #103981
|
Failed on 8.14 https://gradle-enterprise.elastic.co/s/ns3igizthf3eq @kkrik-es was the fix backported? |
|
Yeah I saw the failure.. I didn't backport the fix, will do so if I don't see another failure in main this week. |
|
No more failures, closing (and backporting the fix) for now. |
elasticsearchmachine
pushed a commit
that referenced
this issue
Jun 17, 2024
This PR fixes two test failures #103981 & #105437 and refactors the code a bit to make things more explicit. **What was the issue** These tests were creating an index with a policy before that policy was created. This could cause an issue if ILM would run after the index was created but before the policy was created. When ILM runs before the policy is added, the following happen: - the index encounters an error the ILM state sets that the current step is `null`, which makes sense since there is no policy to retrieve a step from. - A `null` step does not qualify to be executed periodically, which also makes sense because probably nothing changed, so chances are the index will remain in this state. - The test keeps waiting for something to happen, but this is not happening because no cluster state updates are coming like they would have if this was a "real" cluster. - Until the test tear down starts, then the index gets updates with the ILM policy but it's a bit too late. The previous scenario is confirmed by the logging too. ``` ----> The index gets created referring a policy that does not exist yet, ILM runs at least twice before the policy is there [2024-06-12T20:14:28,857][....] [index-sanohmhwxl] creating index, ...... [2024-06-12T20:14:28,870][....] [index-sanohmhwxl] retrieved current step key: null [2024-06-12T20:14:28,871][....] unable to retrieve policy [policy-tohpA] for index [index-sanohmhwxl], recording this in step_info for this index java.lang.IllegalArgumentException: policy [policy-tohpA] does not exist -----> Only now the policy is added [2024-06-12T20:14:29,024][....] adding index lifecycle policy [policy-tohpA] -----> ILM is running periodically but because the current step is null it ignores it [2024-06-12T20:15:23,791][....] job triggered: ilm, 1718223323790, 1718223323790 [2024-06-12T20:15:23,791][....] retrieved current step key: null [2024-06-12T20:15:23,791][....] maybe running periodic step (InitializePolicyContextStep) with current step {"phase":"new","action":"init","name":"init"} ``` This can also be locally reproduced by adding a 5s thread sleep before adding the policy. **The fix** Adding a non existing policy to an index is a not a supported path. For this reason, we refactored the test to reflect a more realistic scenario. - We add the policy as an argument in `private void createIndex(String index, String alias, String policy, boolean isTimeSeries)`. This way it's clear that a policy could be added. - We created the policy before adding the index, it does not appear that adding the policy later is crucial for the test, so simplifying it sounded like a good idea. - Simplified `testRollupIndexInTheHotPhaseWithoutRollover` that ensures that a downsampling action cannot be added in the hot phase without rollover. An index is not necessary for this test, so again simplifying it makes the purpose of the test more clear. Fixes: #103981 Fixes: #105437
gmarouli
added a commit
to gmarouli/elasticsearch
that referenced
this issue
Jun 17, 2024
…9787) This PR fixes two test failures elastic#103981 & elastic#105437 and refactors the code a bit to make things more explicit. **What was the issue** These tests were creating an index with a policy before that policy was created. This could cause an issue if ILM would run after the index was created but before the policy was created. When ILM runs before the policy is added, the following happen: - the index encounters an error the ILM state sets that the current step is `null`, which makes sense since there is no policy to retrieve a step from. - A `null` step does not qualify to be executed periodically, which also makes sense because probably nothing changed, so chances are the index will remain in this state. - The test keeps waiting for something to happen, but this is not happening because no cluster state updates are coming like they would have if this was a "real" cluster. - Until the test tear down starts, then the index gets updates with the ILM policy but it's a bit too late. The previous scenario is confirmed by the logging too. ``` ----> The index gets created referring a policy that does not exist yet, ILM runs at least twice before the policy is there [2024-06-12T20:14:28,857][....] [index-sanohmhwxl] creating index, ...... [2024-06-12T20:14:28,870][....] [index-sanohmhwxl] retrieved current step key: null [2024-06-12T20:14:28,871][....] unable to retrieve policy [policy-tohpA] for index [index-sanohmhwxl], recording this in step_info for this index java.lang.IllegalArgumentException: policy [policy-tohpA] does not exist -----> Only now the policy is added [2024-06-12T20:14:29,024][....] adding index lifecycle policy [policy-tohpA] -----> ILM is running periodically but because the current step is null it ignores it [2024-06-12T20:15:23,791][....] job triggered: ilm, 1718223323790, 1718223323790 [2024-06-12T20:15:23,791][....] retrieved current step key: null [2024-06-12T20:15:23,791][....] maybe running periodic step (InitializePolicyContextStep) with current step {"phase":"new","action":"init","name":"init"} ``` This can also be locally reproduced by adding a 5s thread sleep before adding the policy. **The fix** Adding a non existing policy to an index is a not a supported path. For this reason, we refactored the test to reflect a more realistic scenario. - We add the policy as an argument in `private void createIndex(String index, String alias, String policy, boolean isTimeSeries)`. This way it's clear that a policy could be added. - We created the policy before adding the index, it does not appear that adding the policy later is crucial for the test, so simplifying it sounded like a good idea. - Simplified `testRollupIndexInTheHotPhaseWithoutRollover` that ensures that a downsampling action cannot be added in the hot phase without rollover. An index is not necessary for this test, so again simplifying it makes the purpose of the test more clear. Fixes: elastic#103981 Fixes: elastic#105437
elasticsearchmachine
pushed a commit
that referenced
this issue
Jun 17, 2024
…109791) This PR fixes two test failures #103981 & #105437 and refactors the code a bit to make things more explicit. **What was the issue** These tests were creating an index with a policy before that policy was created. This could cause an issue if ILM would run after the index was created but before the policy was created. When ILM runs before the policy is added, the following happen: - the index encounters an error the ILM state sets that the current step is `null`, which makes sense since there is no policy to retrieve a step from. - A `null` step does not qualify to be executed periodically, which also makes sense because probably nothing changed, so chances are the index will remain in this state. - The test keeps waiting for something to happen, but this is not happening because no cluster state updates are coming like they would have if this was a "real" cluster. - Until the test tear down starts, then the index gets updates with the ILM policy but it's a bit too late. The previous scenario is confirmed by the logging too. ``` ----> The index gets created referring a policy that does not exist yet, ILM runs at least twice before the policy is there [2024-06-12T20:14:28,857][....] [index-sanohmhwxl] creating index, ...... [2024-06-12T20:14:28,870][....] [index-sanohmhwxl] retrieved current step key: null [2024-06-12T20:14:28,871][....] unable to retrieve policy [policy-tohpA] for index [index-sanohmhwxl], recording this in step_info for this index java.lang.IllegalArgumentException: policy [policy-tohpA] does not exist -----> Only now the policy is added [2024-06-12T20:14:29,024][....] adding index lifecycle policy [policy-tohpA] -----> ILM is running periodically but because the current step is null it ignores it [2024-06-12T20:15:23,791][....] job triggered: ilm, 1718223323790, 1718223323790 [2024-06-12T20:15:23,791][....] retrieved current step key: null [2024-06-12T20:15:23,791][....] maybe running periodic step (InitializePolicyContextStep) with current step {"phase":"new","action":"init","name":"init"} ``` This can also be locally reproduced by adding a 5s thread sleep before adding the policy. **The fix** Adding a non existing policy to an index is a not a supported path. For this reason, we refactored the test to reflect a more realistic scenario. - We add the policy as an argument in `private void createIndex(String index, String alias, String policy, boolean isTimeSeries)`. This way it's clear that a policy could be added. - We created the policy before adding the index, it does not appear that adding the policy later is crucial for the test, so simplifying it sounded like a good idea. - Simplified `testRollupIndexInTheHotPhaseWithoutRollover` that ensures that a downsampling action cannot be added in the hot phase without rollover. An index is not necessary for this test, so again simplifying it makes the purpose of the test more clear. Fixes: #103981 Fixes: #105437
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Build scan:
https://gradle-enterprise.elastic.co/s/ruzhsbyskkuk6/tests/:x-pack:plugin:ilm:qa:multi-node:javaRestTest/org.elasticsearch.xpack.ilm.actions.DownsampleActionIT/testRollupNonTSIndex
Reproduction line:
Applicable branches:
main
Reproduces locally?:
Didn't try
Failure history:
Failure dashboard for
org.elasticsearch.xpack.ilm.actions.DownsampleActionIT#testRollupNonTSIndexFailure excerpt:
The text was updated successfully, but these errors were encountered: