Fix Broken Numeric Shard Generations in RepositoryData #57813
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix broken numeric shard generations when reading them from the wire or physically from the physical repository. This should be the cheapest way to clean up broken shard generations in a BwC and safe-to-backport manner for now. We can potentially further optimize this by also not doing the checks on the generations based on the versions we see in the `RepositoryData` but I don't think it matters much since we will read `RepositoryData` from cache in almost all cases. Closes elastic#57798
original-brownbear
added
>bug
:Distributed Coordination/Snapshot/Restore
|
Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore) |
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Nevermind, problematic corner case is actually good I think. |
ywelsch
suggested changes
Jun 8, 2020
| shardGenerations.put(indexId, i, gens.get(i)); | ||
| final String parsedGen = gens.get(i); | ||
| shardGenerations.put( | ||
| indexId, i, fixBrokenShardGens ? ShardGenerations.fixShardGeneration(parsedGen) : parsedGen); |
There was a problem hiding this comment.
I think we should avoid putting null values into the map (in particular as ShardGenerations.Builder.put does not seem to define @Nullable String generation, and the semantics of putting null into map can be problematic, as the Map interface says the behavior is underspecified whether a NPE is thrown). I would prefer a boolean method here with conditional logic
| @@ -129,7 +129,7 @@ public static void assertConsistency(BlobStoreRepository repository, Executor ex | |||
| try (InputStream blob = blobContainer.readBlob("index-" + latestGen); | |||
| XContentParser parser = XContentType.JSON.xContent().createParser(NamedXContentRegistry.EMPTY, | |||
| LoggingDeprecationHandler.INSTANCE, blob)) { | |||
| repositoryData = RepositoryData.snapshotsFromXContent(parser, latestGen); | |||
| repositoryData = RepositoryData.snapshotsFromXContent(parser, latestGen, true); | |||
There was a problem hiding this comment.
Right my bad no reason to be lenient here ... moved to false.
ywelsch
approved these changes
Jun 8, 2020
|
Thanks! @ywelsch about the backport, I was going to ask the same about the other PR as well ... I don't see why not? => I'll backport this and the other one to 7.7.2? |
|
👍 |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Jun 8, 2020
Fix broken numeric shard generations when reading them from the wire or physically from the physical repository. This should be the cheapest way to clean up broken shard generations in a BwC and safe-to-backport manner for now. We can potentially further optimize this by also not doing the checks on the generations based on the versions we see in the `RepositoryData` but I don't think it matters much since we will read `RepositoryData` from cache in almost all cases. Closes elastic#57798
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Jun 8, 2020
Fix broken numeric shard generations when reading them from the wire or physically from the physical repository. This should be the cheapest way to clean up broken shard generations in a BwC and safe-to-backport manner for now. We can potentially further optimize this by also not doing the checks on the generations based on the versions we see in the `RepositoryData` but I don't think it matters much since we will read `RepositoryData` from cache in almost all cases. Closes elastic#57798
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Jun 8, 2020
Fix broken numeric shard generations when reading them from the wire or physically from the physical repository. This should be the cheapest way to clean up broken shard generations in a BwC and safe-to-backport manner for now. We can potentially further optimize this by also not doing the checks on the generations based on the versions we see in the `RepositoryData` but I don't think it matters much since we will read `RepositoryData` from cache in almost all cases. Closes elastic#57798
original-brownbear
added a commit
that referenced
this pull request
Jun 8, 2020
Fix broken numeric shard generations when reading them from the wire or physically from the physical repository. This should be the cheapest way to clean up broken shard generations in a BwC and safe-to-backport manner for now. We can potentially further optimize this by also not doing the checks on the generations based on the versions we see in the `RepositoryData` but I don't think it matters much since we will read `RepositoryData` from cache in almost all cases. Closes #57798
original-brownbear
added a commit
that referenced
this pull request
Jun 8, 2020
Fix broken numeric shard generations when reading them from the wire or physically from the physical repository. This should be the cheapest way to clean up broken shard generations in a BwC and safe-to-backport manner for now. We can potentially further optimize this by also not doing the checks on the generations based on the versions we see in the `RepositoryData` but I don't think it matters much since we will read `RepositoryData` from cache in almost all cases. Closes #57798
original-brownbear
added a commit
that referenced
this pull request
Jun 8, 2020
Fix broken numeric shard generations when reading them from the wire or physically from the physical repository. This should be the cheapest way to clean up broken shard generations in a BwC and safe-to-backport manner for now. We can potentially further optimize this by also not doing the checks on the generations based on the versions we see in the `RepositoryData` but I don't think it matters much since we will read `RepositoryData` from cache in almost all cases. Closes #57798
|
Hey @original-brownbear it looks like we missed mentioning this PR in the release notes. |
Yea I guess technically that would be correct. Not sure how relevant it is at this point, but technically it's the first version released with this fix. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix broken numeric shard generations when reading them from the wire
or physically from the physical repository.
This seems to me to be the cheapest way to clean up broken shard generations
in a BwC and safe-to-backport manner for now. We can potentially
further optimize this by also not doing the checks on the generations
based on the versions we see in the
RepositoryDatabut I don't thinkit matters much since we will read
RepositoryDatafrom cache in almostall cases.
Closes #57798