SnapshotIT testCreateSnapshot fails possibly due to simultaneous deletions #37581
Assignees
Labels
:Distributed Coordination/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
>test-failure
Triaged test failures from CI
Comments
jaymode
added
:Distributed Coordination/Snapshot/Restore
|
Pinging @elastic/es-distributed |
|
@jaymode this looks like some actual bug in snapshotting, there shouldn't be any concurrent actions on this file. nevermind figured out where this is coming from, no it :) |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Jan 18, 2019
* The repo id was determined wrong when the delete picked up on an in progress snapshot * NOTE: This solution is still a best-effort fix and there's a slight chance of running into concurrency issues here when multiple create and delete requests for the same snapshot name are happening concurrently, but these require a sequence of multiple cluster state updates between the changed method reading the genId and submitting its cluster state update task * Added test reproduced the issue reliably in about 50% of runs * Closes elastic#37581
|
Fix incoming in #37612 |
original-brownbear
added a commit
that referenced
this issue
Jan 21, 2019
* The repo id was determined wrong when the delete picked up on an in progress snapshot * NOTE: This solution is still a best-effort fix and there's a slight chance of running into concurrency issues here when multiple create and delete requests for the same snapshot name are happening concurrently, but these require a sequence of multiple cluster state updates between the changed method reading the genId and submitting its cluster state update task * Added test reproduced the issue reliably in about 50% of runs * Closes #37581
|
This failed again today in a PR build: https://gradle-enterprise.elastic.co/s/67kanac5qy4ok but this time with a slightly different error: I was unable to reproduce this with @original-brownbear I think this is something you've fixed before, does it just need retries? |
|
@dakrone huh, this funny enough is a completely unrelated bug to the one initially fixed here :) The problem is that we're looping over all the files in a directory and then reading their sizes one by one when doing -> I'll open a PR to fix that in a bit. |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Nov 15, 2019
If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes elastic#37581
|
#49142 Should do the trick here |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Nov 15, 2019
…lastic#49142) * Make FsBlobContainer Listing Resilient to Concurrent Modifications If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes elastic#37581
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Nov 15, 2019
…lastic#49142) * Make FsBlobContainer Listing Resilient to Concurrent Modifications If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes elastic#37581
original-brownbear
added a commit
that referenced
this issue
Nov 15, 2019
…49142) (#49176) * Make FsBlobContainer Listing Resilient to Concurrent Modifications If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes #37581
original-brownbear
added a commit
that referenced
this issue
Nov 15, 2019
…49142) (#49177) * Make FsBlobContainer Listing Resilient to Concurrent Modifications If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes #37581
This was referenced Feb 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I hit this failure locally multiple times but it does not reproduce.
Reproduce line:
Test runner output with test ordering:
Log from integ test cluster:
The text was updated successfully, but these errors were encountered: