Propagate last node to reinitialized routing tables #91549
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When closing or opening an index, or restoring a snapshot over a closed index, we reinitialize its routing table from scratch and expect the gateway allocators to select the appropriate node for each shard copy. With this commit we also keep track of the last-allocated node ID for each copy which makes it more likely that the desired balance of these shards remains unchanged too. Closes elastic#91472
DaveCTurner
added
>non-issue
:Distributed Coordination/Allocation
elasticsearchmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
…ount, expect all the node IDs to be filled in
idegtiarenko
approved these changes
Nov 14, 2022
| } | ||
| final var previousNodes = new ArrayList<String>(previousShardRoutingTable.size()); | ||
| previousNodes.add(primaryNode); | ||
| for (final var assignedShard : previousShardRoutingTable.assignedShards()) { |
There was a problem hiding this comment.
This also includes the target of relocations. I wonder if we should only look at active shards, since anything less will anyway not be considered good enough by the gateway allocator?
The problem I see with this is that if a relocation is ongoing, we risk a copy having a last allocated node id that is much worse than it could be (i.e., a node that only has just started the recovery)?
There was a problem hiding this comment.
Good point, thanks - see bd12ab9.
| assertThat(shard.unassignedInfo().getReason(), equalTo(expectedUnassignedReason)); | ||
| final var lastAllocatedNodeId = shard.unassignedInfo().getLastAllocatedNodeId(); | ||
| if (lastAllocatedNodeId == null) { | ||
| // restoring an index may change the number of shards/replicas so no guarantee that lastAllocatedNodeId is populated |
There was a problem hiding this comment.
I think only the number of replicas, not the number of shards can be changed? Probably what you meant with shards/replicas, but removing "shards/" would be better I think.
Suggested change
| // restoring an index may change the number of shards/replicas so no guarantee that lastAllocatedNodeId is populated | |
| // restoring an index may change the number of replicas so no guarantee that lastAllocatedNodeId is populated |
There was a problem hiding this comment.
On the contrary, I didn't think there's anything to require that the snapshot has the same number of shards as index on top of which it's being restored.
There was a problem hiding this comment.
Ahh, right, thanks.
Comment on lines
+271
to
+272
| // both original and restored index must have at least one shard tho | ||
| assertTrue(foundAnyNodeIds); |
There was a problem hiding this comment.
Can this not go one line up, i.e., we can check this for every shard id?
There was a problem hiding this comment.
Not if the shard count can change in a restore (which AFAIK it can)
DaveCTurner
added
the
auto-merge-without-approval
DaveCTurner
deleted the
2022-11-14-propagate-last-allocated-id-to-reinitialized-routing-tables
branch
weizijun
added a commit
to weizijun/elasticsearch
that referenced
this pull request
Nov 15, 2022
* main: (163 commits) [DOCS] Edits frequent items aggregation (elastic#91564) Handle providers of optional services in ubermodule classloader (elastic#91217) Add `exportDockerImages` lifecycle task for exporting docker tarballs (elastic#91571) Fix CSV dependency report output file location in DRA CI job Fix variable placeholder for Strings.format calls (elastic#91531) Fix output dir creation in ConcatFileTask (elastic#91568) Fix declaration of dependencies in DRA snapshots CI job (elastic#91569) Upgrade Gradle Enterprise plugin to 3.11.4 (elastic#91435) Ingest DateProcessor (small) speedup, optimize collections code in DateFormatter.forPattern (elastic#91521) Fix inter project handling of generateDependenciesReport (elastic#91555) [Synthetics] Add synthetics-* read to fleet-server (elastic#91391) [ML] Copy more settings when creating DF analytics destination index (elastic#91546) Reduce CartesianCentroidIT flakiness (elastic#91553) Propagate last node to reinitialized routing tables (elastic#91549) Forecast write load during rollovers (elastic#91425) [DOCS] Warn about potential overhead of named queries (elastic#91512) Datastream unavailable exception metadata (elastic#91461) Generate docker images and dependency report in DRA ci job (elastic#91545) Support cartesian_bounds aggregation on point and shape (elastic#91298) Add support for EQL samples queries (elastic#91312) ... # Conflicts: # x-pack/plugin/rollup/src/main/java/org/elasticsearch/xpack/downsample/RollupShardIndexer.java
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When closing or opening an index, or restoring a snapshot over a closed index, we reinitialize its routing table from scratch and expect the gateway allocators to select the appropriate node for each shard copy. With this commit we also keep track of the last-allocated node ID for each copy which makes it more likely that the desired balance of these shards remains unchanged too.
Closes #91472