[CI] RollupActionIT testRollupIndexAndSetNewRollupPolicy failing #68609
Assignees
Labels
:Data Management/ILM+SLM
Index and Snapshot lifecycle management
:StorageEngine/Rollup
Turn fine-grained time-based data into coarser-grained data
Team:Analytics
Meta label for analytical engine team (ESQL/Aggs/Geo)
Team:Data Management
Meta label for data/management team
>test-failure
Triaged test failures from CI
Comments
mark-vieira
added
>test-failure
|
Pinging @elastic/es-core-features (Team:Core/Features) |
|
Since this is a newly added test I've muted it in |
|
@andreidan since you added this test do you mind taking a look? |
|
yes. I'll investigate. thanks for the ping and for muting the test! |
talevy
added
the
:StorageEngine/Rollup
elasticmachine
added
the
Team:Analytics
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
Ah, thanks for the clarification @dakrone. The "old" test has failed in a similar manner before as well: https://gradle-enterprise.elastic.co/s/5o6gpcxlp5uri/tests/:x-pack:plugin:ilm:qa:multi-node:javaRestTest/org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT/testRollupIndexAndSetNewRollupPolicy#1 |
|
FYI, I've seen other builds with slightly different test failures as well but the common thing I see is this in some of the node logs: I think these might all be related. |
|
Also @talevy we might want to improve some of the error messages in there as per what Lee and I talked about in slack. https://elastic.slack.com/archives/C0D8ST60Y/p1612563629493100 |
|
Yeah, pretty much every build failing on task The confusing bit is depending on when it happens and which node the tests happen to hit you'll get different tests failing with different errors. I know very little, but it seems like that is the root cause here. If we could prioritize a fix here that would be great since it's causing a whole host of build failures and it's not practical to mute them all unless we literally mute this entire project. |
talevy
added a commit
to talevy/elasticsearch
that referenced
this issue
Feb 11, 2021
This commit removes the assertion in RollupShardIndexer that verifies that temporary files are deleted. Since it is the responsibility of the indexer to instruct the OS to delete files, it may not do so in a timely manner. This results in a potentially flaky assertion. Instead, a new unit test is introduced that will introspect the indexer and assert that it had successfully called for the files to be deleted. Closes elastic#68609.
talevy
added a commit
that referenced
this issue
Feb 11, 2021
This commit removes the assertion in RollupShardIndexer that verifies that temporary files are deleted. Since it is the responsibility of the indexer to instruct the OS to delete files, it may not do so in a timely manner. This results in a potentially flaky assertion. Instead, a new unit test is introduced that will introspect the indexer and assert that it had successfully called for the files to be deleted. Closes #68609.
talevy
added a commit
to talevy/elasticsearch
that referenced
this issue
Feb 16, 2021
This commit removes the assertion in RollupShardIndexer that verifies that temporary files are deleted. Since it is the responsibility of the indexer to instruct the OS to delete files, it may not do so in a timely manner. This results in a potentially flaky assertion. Instead, a new unit test is introduced that will introspect the indexer and assert that it had successfully called for the files to be deleted. Closes elastic#68609.
|
This continues to fail on occasion, but now with a different error: @talevy mind taking a look? |
|
Drive-by comment. This seems to be a problem with the rollup operation itself? |
|
thank you. will take another look! |
|
We are still having "failed to delete policy" errors: https://gradle-enterprise.elastic.co/s/i2gfwo5lss5j4 This time |
|
Got a few more failures on this. Details in #70980 (which I have now closed as I realised it's a duplicate of this one) |
csoulios
added a commit
to csoulios/elasticsearch
that referenced
this issue
Mar 30, 2021
csoulios
added a commit
that referenced
this issue
Mar 30, 2021
csoulios
added a commit
that referenced
this issue
Mar 30, 2021
csoulios
added a commit
that referenced
this issue
Jul 26, 2022
This PR adds support for an ILM action that downsamples a time-series index by invoking the _rollup endpoint (#85708) A policy that includes the rollup action will look like the following PUT _ilm/policy/my_policy { "policy": { "phases": { "warm": { "actions": { "rollup": { "fixed_interval": "24h" } } } } } } Relates to #74660 Fixes #68609
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a new test case which has failed a few times now since being added.
Build scan:
https://gradle-enterprise.elastic.co/s/nckptifuyp4v2/tests/:x-pack:plugin:ilm:qa:multi-node:javaRestTest/org.elasticsearch.xpack.ilm.actions.RollupActionIT/testRollupIndexAndSetNewRollupPolicy#1
Repro line:
./gradlew ':x-pack:plugin:ilm:qa:multi-node:javaRestTest' --tests "org.elasticsearch.xpack.ilm.actions.RollupActionIT.testRollupIndexAndSetNewRollupPolicy" -Dtests.seed=F0A4BF22F8036B7E -Dtests.security.manager=true -Dtests.locale=sk-SK -Dtests.timezone=Pacific/Truk -Druntime.java=11Reproduces locally?:
Nope.
Applicable branches:
masterand7.x(error on the7.xbuild looked a bit different though)Failure history:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=America/Los_Angeles&tests.container=org.elasticsearch.xpack.ilm.actions.RollupActionIT&tests.sortField=FAILED&tests.unstableOnly=true
Failure excerpt:
The text was updated successfully, but these errors were encountered: