[CI] SearchableSnapshotsIntegTests testCreateAndRestoreSearchableSnapshot failing #77831
Assignees
Labels
:Distributed Coordination/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
Team:Distributed (Obsolete)
Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
>test-failure
Triaged test failures from CI
Comments
ywangd
added
:Distributed Coordination/Snapshot/Restore
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
tlrx
added a commit
that referenced
this issue
Nov 5, 2021
The tests testCreateAndRestoreSearchableSnapshot and testCreateAndRestorePartialSearchableSnapshot both failed once when asserting the shard folders using assertShardFolders(index, true). The failures occurred when the original index is first closed (not deleted) and mounted again under the same name (so it will be restored as a searchable snapshot index on top of the existing shard files). The SearchableSnapshotDirectory implementation takes care to clean up the shard files on disk using SearchableSnapshotDirectory.cleanExistingRegularShardFiles() and the tests verify that the shard index folder is indeed deleted from disk on all nodes but sometime fail because the folder is still present. I wasn't able to reproduce but I think that the closing of the original index + the creation of the .snapshot-blob-cache index trigger some shard relocations that are cancelled by the subsequent mount/restore, leaving some files on disk that should be cleaned up but maybe not immediately. This commit changes the tests to assertBusy() when verifying the shard folders and also adds more logging information in case waiting for the assertShardFolders(index, true) is not enough. Closes #77831
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this issue
Nov 5, 2021
The tests testCreateAndRestoreSearchableSnapshot and testCreateAndRestorePartialSearchableSnapshot both failed once when asserting the shard folders using assertShardFolders(index, true). The failures occurred when the original index is first closed (not deleted) and mounted again under the same name (so it will be restored as a searchable snapshot index on top of the existing shard files). The SearchableSnapshotDirectory implementation takes care to clean up the shard files on disk using SearchableSnapshotDirectory.cleanExistingRegularShardFiles() and the tests verify that the shard index folder is indeed deleted from disk on all nodes but sometime fail because the folder is still present. I wasn't able to reproduce but I think that the closing of the original index + the creation of the .snapshot-blob-cache index trigger some shard relocations that are cancelled by the subsequent mount/restore, leaving some files on disk that should be cleaned up but maybe not immediately. This commit changes the tests to assertBusy() when verifying the shard folders and also adds more logging information in case waiting for the assertShardFolders(index, true) is not enough. Closes elastic#77831
elasticsearchmachine
pushed a commit
that referenced
this issue
Nov 5, 2021
The tests testCreateAndRestoreSearchableSnapshot and testCreateAndRestorePartialSearchableSnapshot both failed once when asserting the shard folders using assertShardFolders(index, true). The failures occurred when the original index is first closed (not deleted) and mounted again under the same name (so it will be restored as a searchable snapshot index on top of the existing shard files). The SearchableSnapshotDirectory implementation takes care to clean up the shard files on disk using SearchableSnapshotDirectory.cleanExistingRegularShardFiles() and the tests verify that the shard index folder is indeed deleted from disk on all nodes but sometime fail because the folder is still present. I wasn't able to reproduce but I think that the closing of the original index + the creation of the .snapshot-blob-cache index trigger some shard relocations that are cancelled by the subsequent mount/restore, leaving some files on disk that should be cleaned up but maybe not immediately. This commit changes the tests to assertBusy() when verifying the shard folders and also adds more logging information in case waiting for the assertShardFolders(index, true) is not enough. Closes #77831
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this issue
Nov 5, 2021
The tests testCreateAndRestoreSearchableSnapshot and testCreateAndRestorePartialSearchableSnapshot both failed once when asserting the shard folders using assertShardFolders(index, true). The failures occurred when the original index is first closed (not deleted) and mounted again under the same name (so it will be restored as a searchable snapshot index on top of the existing shard files). The SearchableSnapshotDirectory implementation takes care to clean up the shard files on disk using SearchableSnapshotDirectory.cleanExistingRegularShardFiles() and the tests verify that the shard index folder is indeed deleted from disk on all nodes but sometime fail because the folder is still present. I wasn't able to reproduce but I think that the closing of the original index + the creation of the .snapshot-blob-cache index trigger some shard relocations that are cancelled by the subsequent mount/restore, leaving some files on disk that should be cleaned up but maybe not immediately. This commit changes the tests to assertBusy() when verifying the shard folders and also adds more logging information in case waiting for the assertShardFolders(index, true) is not enough. Closes elastic#77831
elasticsearchmachine
pushed a commit
that referenced
this issue
Nov 8, 2021
The tests testCreateAndRestoreSearchableSnapshot and testCreateAndRestorePartialSearchableSnapshot both failed once when asserting the shard folders using assertShardFolders(index, true). The failures occurred when the original index is first closed (not deleted) and mounted again under the same name (so it will be restored as a searchable snapshot index on top of the existing shard files). The SearchableSnapshotDirectory implementation takes care to clean up the shard files on disk using SearchableSnapshotDirectory.cleanExistingRegularShardFiles() and the tests verify that the shard index folder is indeed deleted from disk on all nodes but sometime fail because the folder is still present. I wasn't able to reproduce but I think that the closing of the original index + the creation of the .snapshot-blob-cache index trigger some shard relocations that are cancelled by the subsequent mount/restore, leaving some files on disk that should be cleaned up but maybe not immediately. This commit changes the tests to assertBusy() when verifying the shard folders and also adds more logging information in case waiting for the assertShardFolders(index, true) is not enough. Closes #77831
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Build scan:
https://gradle-enterprise.elastic.co/s/ni7wnr7bqmz3k/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests/testCreateAndRestoreSearchableSnapshot
Reproduction line:
./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot" -Dtests.seed=DF8F0E1B6CCE9D8B -Dtests.locale=ga -Dtests.timezone=Etc/GMT -Druntime.java=11Applicable branches:
7.x
Reproduces locally?:
No
Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests&tests.test=testCreateAndRestoreSearchableSnapshot
Failure excerpt:
The text was updated successfully, but these errors were encountered: