Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Simplify ML upgrade step #40006

Merged
merged 6 commits into from
Mar 25, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 25 additions & 23 deletions docs/reference/upgrade/close-ml.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,33 +11,35 @@ POST _ml/set_upgrade_mode?enabled=false
// TEARDOWN
////////////

If your {ml} indices were created earlier than the previous major version, they
must be reindexed. In those circumstances, there must be no machine learning
jobs running during the upgrade.

In all other circumstances, there is no requirement to close your {ml} jobs.
There are, however, advantages to doing so. If you choose to leave your jobs
running during the upgrade, they are affected when you stop the {ml} nodes. The
jobs move to another {ml} node and restore the model states. This scenario has
the least disruption to the active {ml} jobs but incurs the highest load on the
cluster.

To close all {ml} jobs before you upgrade, see
{stack-ov}/stopping-ml.html[Stopping {ml}]. This method persists the model
state at the moment of closure, which means that when you open your jobs after
the upgrade, they use the exact same model. This scenario takes the most time,
however, especially if you have many jobs or jobs with large model states.

To temporarily halt the tasks associated with your {ml} jobs and {dfeeds} and
prevent new jobs from opening, use the <<ml-set-upgrade-mode,set upgrade mode API>>:
If your {ml} indices were created before {prev-major-version}, you must
<<reindex-upgrade,reindex the indices>>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative here is to temporarily halt the tasks associated with your {ml} jobs and {dfeeds} and prevent new jobs from opening by using the <<ml-set-upgrade-mode,set upgrade mode API>> during the reindex.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that step was added to the reindexing page in https://github.com/lcawl/elasticsearch/pull/1/files, so I've rebased and augmented the info there.


If your {ml} indices were created in {prev-major-version}, you can:

* Leave your {ml} jobs running during the upgrade. When you shut down a
{ml} node, its jobs automatically move to another node and restore the model
states. This option enables your jobs to continue running during the upgrade but
it puts increased load on the cluster.
* Temporarily halt the tasks associated with your {ml} jobs and {dfeeds} and
prevent new jobs from opening by using the
<<ml-set-upgrade-mode,set upgrade mode API>>:
+
--
[source,js]
--------------------------------------------------
POST _ml/set_upgrade_mode?enabled=true
--------------------------------------------------
// CONSOLE
This method does not persist the absolute latest model state, rather it uses the
last model state that was automatically saved. By halting the tasks, you avoid
incurring the cost of managing active jobs during the upgrade and it's quicker
than stopping {dfeeds} and closing jobs.
When you disable upgrade mode, the jobs resume using the last model
state that was automatically saved. This option avoids the overhead of managing
active jobs during the upgrade and is faster than explicitly stopping {dfeeds}
and closing jobs.
--
* {stack-ov}/stopping-ml.html[Stop all {dfeeds} and close all jobs]. This option
saves the model state at the time of closure. When you reopen the jobs after the
upgrade, they use the exact same model. However, saving the latest model state
takes longer than using upgrade mode, especially if you have a lot of jobs or
jobs with large model states.
2 changes: 1 addition & 1 deletion docs/reference/upgrade/cluster_restart.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ recovery.
include::synced-flush.asciidoc[]
--

. *Stop any machine learning jobs that are running.*
. *Temporarily stop the tasks associated with active {ml} jobs and {dfeeds}.* (Optional)
+
--
include::close-ml.asciidoc[]
Expand Down
20 changes: 16 additions & 4 deletions docs/reference/upgrade/reindex_upgrade.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,10 @@ ifdef::include-xpack[]
[TIP]
====
If you use {ml-features} and your {ml} indices were created before
{prev-major-version}, you must
{stack-ov}/stopping-ml.html[stop all {dfeeds} and close all {ml} jobs] before
you reindex the indices.
{prev-major-version}, you must temporarily halt the tasks associated with your
{ml} jobs and {dfeeds} and prevent new jobs from opening during the reindex. Use
the <<ml-set-upgrade-mode,set upgrade mode API>> or
{stack-ov}/stopping-ml.html[stop all {dfeeds} and close all {ml} jobs].
If you use {es} {security-features}, before you reindex `.security*` internal
indices it is a good idea to create a temporary superuser account in the `file`
Expand Down Expand Up @@ -112,6 +113,17 @@ indices from the previous major version to be upgraded to the
current major version. Skipping a major version means that you must
resolve any backward compatibility issues yourself.
ifdef::include-xpack[]
If you use {ml-features} and you're migrating indices from a 6.5 or earlier
cluster, the job and {dfeed} configuration information are not stored in an
index. You must recreate your {ml} jobs in the new cluster. If you are migrating
from a 6.6 or later cluster, it is a good idea to temporarily halt the tasks
associated with your {ml} jobs and {dfeeds} to prevent inconsistencies between
different {ml} indices that are reindexed at slightly different times. Use the
<<ml-set-upgrade-mode,set upgrade mode API>> or
{stack-ov}/stopping-ml.html[stop all {dfeeds} and close all {ml} jobs].
endif::include-xpack[]
=============================================

To migrate your indices:
Expand Down Expand Up @@ -184,4 +196,4 @@ monitor progress of the reindex job with the <<tasks,task API>>:
`30s` and `1`).

.. Once reindexing is complete and the status of the new index is `green`,
you can delete the old index.
you can delete the old index.
2 changes: 1 addition & 1 deletion docs/reference/upgrade/rolling_upgrade.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ include::synced-flush.asciidoc[]

--

. *Stop any machine learning jobs that are running.*
. *Temporarily stop the tasks associated with active {ml} jobs and {dfeeds}.* (Optional)
+
--
include::close-ml.asciidoc[]
Expand Down