ILM migrate data between tiers #61377
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License; | ||
| * you may not use this file except in compliance with the Elastic License. | ||
| */ | ||
| package org.elasticsearch.xpack.core.ilm; | ||
|
|
||
| import org.elasticsearch.client.Client; | ||
| import org.elasticsearch.common.ParseField; | ||
| import org.elasticsearch.common.Strings; | ||
| import org.elasticsearch.common.io.stream.StreamInput; | ||
| import org.elasticsearch.common.io.stream.StreamOutput; | ||
| import org.elasticsearch.common.xcontent.ConstructingObjectParser; | ||
| import org.elasticsearch.common.xcontent.XContentBuilder; | ||
| import org.elasticsearch.common.xcontent.XContentParser; | ||
| import org.elasticsearch.xpack.core.ilm.Step.StepKey; | ||
|
|
||
| import java.io.IOException; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
| import java.util.Objects; | ||
|
|
||
| /** | ||
| * A {@link LifecycleAction} which enables or disables the automatic migration of data between | ||
| * {@link org.elasticsearch.xpack.core.DataTier}s. | ||
| */ | ||
| public class MigrateAction implements LifecycleAction { | ||
| public static final String NAME = "migrate"; | ||
| public static final ParseField ENABLED_FIELD = new ParseField("enabled"); | ||
|
|
||
| private static final ConstructingObjectParser<MigrateAction, Void> PARSER = new ConstructingObjectParser<>(NAME, | ||
| a -> new MigrateAction((boolean) a[0])); | ||
|
|
||
| static { | ||
| PARSER.declareBoolean(ConstructingObjectParser.constructorArg(), ENABLED_FIELD); | ||
andreidan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
|
|
||
| private final boolean enabled; | ||
andreidan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| public static MigrateAction parse(XContentParser parser) { | ||
| return PARSER.apply(parser, null); | ||
| } | ||
|
|
||
| public MigrateAction(boolean enabled) { | ||
| this.enabled = enabled; | ||
| } | ||
|
|
||
| public MigrateAction(StreamInput in) throws IOException { | ||
| this(in.readBoolean()); | ||
| } | ||
|
|
||
| @Override | ||
| public void writeTo(StreamOutput out) throws IOException { | ||
| out.writeBoolean(enabled); | ||
| } | ||
|
|
||
| @Override | ||
| public String getWriteableName() { | ||
| return NAME; | ||
| } | ||
|
|
||
| public boolean isEnabled() { | ||
| return enabled; | ||
| } | ||
|
|
||
| @Override | ||
| public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException { | ||
| builder.startObject(); | ||
| builder.field(ENABLED_FIELD.getPreferredName(), enabled); | ||
| builder.endObject(); | ||
| return builder; | ||
| } | ||
|
|
||
| @Override | ||
| public boolean isSafeAction() { | ||
| return true; | ||
| } | ||
|
|
||
| @Override | ||
| public List<Step> toSteps(Client client, String phase, StepKey nextStepKey) { | ||
| if (enabled) { | ||
| Map<String, String> include = Map.of("_tier", "data_" + phase); | ||
andreidan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| AllocateAction migrateDataAction = new AllocateAction(null, include, null, null); | ||
|
There was a problem hiding this comment. should we remove the possible There was a problem hiding this comment. That's a good question, I'm not sure we'd want to piecemeal remove all of them, because they could be something for an additional attribute that we want to preserve between phases. I will have to think on it a bit, it also makes me wonder whether we should make it configurable whether all other index-level allocation filtering settings are removed/preserved when doing the migration. There was a problem hiding this comment. It's an interesting one - we do not allow to have the migrate action enabled in a phase that has the allocate action (configuring allocation), so maybe it makes sense for the migrate action to invalidate the possible allocation filterings the index might have (for eg. from a previous phase where the migrate action was disabled and the allocation action was configured) |
||
| return migrateDataAction.toSteps(client, phase, nextStepKey); | ||
| } else { | ||
| return List.of(); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public int hashCode() { | ||
| return Objects.hash(enabled); | ||
| } | ||
|
|
||
| @Override | ||
| public boolean equals(Object obj) { | ||
| if (obj == null) { | ||
| return false; | ||
| } | ||
| if (obj.getClass() != getClass()) { | ||
| return false; | ||
| } | ||
| MigrateAction other = (MigrateAction) obj; | ||
| return Objects.equals(enabled, other.enabled); | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| return Strings.toString(this); | ||
| } | ||
|
|
||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we make this configurable and serializable for the step? (ie. use just the one allocation decider depending on where it is used)
I'd say we want a fully allocated index so verifying both always is alright. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm... I actually think that we may want to split checking the allocation for the migrate action into a separate step. For example, the allocation routed step currently has a pretty generic message (
Waiting for [n] shards to be allocated to nodes matching the given filters). I think if we split this into a newMigrationRoutedstep we could give it a much better explanation, for example, something like:waiting [23m] for [3] shards to be allocated on nodes with the [data_warm] tieradditionally, I think we could also even throw an error to signal to the user when things were in a bad state, something like:
exception waiting for index to be moved to the [data_cold] tier, there are currently no [data_cold] nodes in the clusterThen the step could be retryable (so we check every 10 minutes) and it at least gives us a way of signaling to a user (alerting on the ilm-history index for example) when they are in an irreconcilable position and need to adjust their cluster.
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You make a great point on validating if the cluster has any node with a particular role available. I'll create another step for the migrate action (the nicer messages will be a great UX improvement as well)