Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

7.0 upgrade #242

Merged
merged 7 commits into from
Mar 26, 2019
Merged

7.0 upgrade #242

Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cf68e5c
[DOCS] First pass at 7.0 upgrade update.
debadair Mar 12, 2019
777198f
[DOCS] Fixed formatting issues & flagged the Upgrade Assistant as xpack.
debadair Mar 12, 2019
62530e4
Update docs/en/install-upgrade/upgrading-stack.asciidoc
lcawl Mar 13, 2019
e0d2869
[DOCS] Incorporated feedback from lcawl.
debadair Mar 13, 2019
7e28e05
[DOCS] Fixes broken link
lcawl Mar 14, 2019
4bc7f81
Merge branch '7.0' into 7.0-upgrade
lcawl Mar 15, 2019
d6c24fc
[DOCS] Removed tip about temporary superuser when upgrading the secur…
debadair Mar 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
319 changes: 83 additions & 236 deletions docs/en/install-upgrade/upgrading-stack.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,277 +1,124 @@
[[upgrading-elastic-stack]]
== Upgrading the Elastic Stack

When upgrading to a new version of Elasticsearch, you need to upgrade
each of the products in your Elastic Stack. Beats and Logstash 5.6 are
compatible with Elasticsearch {version} to give you flexibility in scheduling the
upgrade.

****
The steps you need to take to upgrade differ depending on which products you
are using. Want a list that's tailored to your stack? Try out our
{upgrade_guide}[Interactive Upgrade Guide].
****

If you are running a pre-6.0 version, we recommend upgrading to the most
recent 5.6 before upgrading to {version}. {xpack} 5.6
provides a free Upgrade Assistant that identifies issues you need to address
before upgrading and simplifies migrating indices that need to be reindexed
before you upgrade. The Upgrade Assistant is enabled with both Trial and
Basic licenses. You can install {xpack} solely for the purpose of upgrading.

Rolling upgrades are supported when upgrading from Elasticsearch 5.6 and
Elasticsearch 6.0-6.2 to {version}. Upgrading from any
version prior to 5.6 requires a full cluster restart.

IMPORTANT: 2.x indices are not compatible with {version}. You must
remove or reindex them on your 5.n cluster before upgrading to {version}. The internal
Kibana and {xpack} indices and the default Beats and Logstash mapping templates
also need to be updated to work with {version}.
When upgrading to a new version of {es}, you need to upgrade
each of the products in your Elastic Stack. Beats and Logstash 6.7 are
compatible with {es} {version} to give you flexibility in scheduling
the upgrade.

=== Preparing to upgrade

Before upgrading the Elastic Stack to {version}:
{es} supports rolling upgrades between minor versions, from {es} 5.6 to 6.7,
and from 6.7 to {version}.

. Back up your data. You **cannot roll back** to an earlier version unless
you have a backup of your data. For information about creating snapshots, see
{ref}/modules-snapshots.html[Snapshot and Restore].
IMPORTANT: 5.x indices are not compatible with {version}. You must
remove or reindex them to upgrade to {version}. The default Beats and
Logstash mapping templates also need to be updated to work with {version}.

. Check the Elasticsearch {ref}/settings.html[deprecation log] to see if
you're using any deprecated features and update your code accordingly.
By default, deprecation log messages are enabled at the `WARN` level.
=== Preparing to upgrade

. Review the breaking changes for each product you use
and make the necessary changes so your code is compatible with {version}. See
<<elastic-stack-breaking-changes>>.
Before upgrading the Elastic Stack to {version}:

. {ref}/docs-reindex.html[Reindex] or delete any indices created on 2.n. We recommend
upgrading to the most recent 5.6 and using the {xpack} Reindex Helper to reindex 2.n indices.
. Check the {es} {ref}/logging.html#deprecation-logging[deprecation log]
to see if you're using any deprecated features and update your code accordingly.
By default, deprecation warnings are logged when the log level is set to `WARN`.

. If Kibana and {xpack} are part of your stack, upgrade the internal Kibana
and {xpack} indices. We recommend using the {xpack} 5.6 Reindex Helper to
upgrade the internal indices. If you're performing a full cluster restart upgrade
from an earlier version, you can also
<<upgrade-internal-indices,use the `_migration/upgrade` API>> directly to
upgrade the internal indices after you install Elasticsearch {version}.
. Review the <<elastic-stack-breaking-changes>> and upgrade your code to work
with {version}.

. If you use {stack} {security-features} to secure your cluster:
.. Make sure TLS is enabled to encrypt communications between nodes. TLS must
be enabled to upgrade to {version}. For more information, see
{stack-ov}/encrypting-communications.html[Encrypting communications].
. Upgrade to 6.7 and use the {kibana-ref}/upgrade-assistant.html[{kib} Upgrade Assistant] to {ref}/docs-reindex.html[reindex]
any indices that are not compatible with {version}.
+
NOTE: Enabling TLS requires a full cluster restart. Nodes that have TLS
enabled cannot communicate with nodes that do not have TLS enabled. You must
restart all nodes to maintain communication across the cluster.

.. Make sure real passwords are configured for the built-in `elasticsearch`,
`kibana`, and `logstash_system` users. They cannot use the 5.n default
password (`changeme`). For more information, see
{stack-ov}/built-in-users.html[Built-in users].

. Consider closing {ml} jobs before you start the upgrade process. It is not
required, but there are pros and cons to leaving the jobs running. These
considerations are described in the steps related to
{ref}/setup-upgrade.html[upgrading {es}].
[role="xpack"]
.Upgrade Assistant
******
The Upgrade Assistant and migration APIs are enabled with both the Basic and
Trial licenses. You can install the default distribution of 6.7 to use the
Upgrade Assistant to prepare to upgrade even if you are upgrading to the OSS
distribution of {version}.
******

IMPORTANT: Test upgrades in a dev environment before upgrading your
production cluster.
. Use the Upgrade Assistant to identify any changes you need to make to your
cluster configuration.

[[upgrade-order-elastic-stack]]
=== Upgrade order

Upgrade the Elastic Stack products you use in the following order:

. Elasticsearch Hadoop: {hadoop-ref}/install.html[install instructions]
. Elasticsearch: {ref}/setup-upgrade.html[upgrade instructions]
. Kibana: {kibana-ref}/upgrade.html[upgrade instructions]
. Logstash: {logstash-ref}/upgrading-logstash.html[upgrade instructions]
. Beats: {beats-ref}/upgrading.html[upgrade instructions]
. APM Server: {apm-server-ref}/upgrading.html[upgrade instructions]
=== Upgrade process

NOTE: Logstash 5.6 and 6.n and Beats 5.6 and 6.n are compatible with all 6.n versions of
Elasticsearch. This provides flexibility in when you schedule the upgrades
for your Logstash instances and Beats agents. We recommend upgrading Logstash
and Beats as soon as possible to take advantage of performance improvements
and other enhancements.

=== Upgrading to 6.3
Starting in 6.3, the default distributions of {es}, {ls}, and {kib}
include {xpack} and a free Basic license that never expires.

You can perform rolling upgrades to 6.3 from OSS-only clusters running 5.6
or 6.0-6.2. Basic features are operational once the cluster is fully
upgraded. If you are already using {xpack}, your settings are preserved when
you upgrade.

IMPORTANT: If you use {xpack} you must explicitly remove the old {xpack} plugin
before restarting: `bin/elasticsearch-plugin remove x-pack`. The node will fail
to start if the {xpack} plugin is present.

You must explicitly enable data collection after the upgrade to use monitoring.
Set `xpack.monitoring.collection.enabled` to `true` with the `_cluster/settings`
API:

[source,json]
----------------------------------------------------------
PUT /_cluster/settings
{
"persistent" : {
"xpack.monitoring.collection.enabled" : "true"
}
}
----------------------------------------------------------
// CONSOLE

To take more of the {stack} features for a spin, you can start a 30-day trial
from Kibana or with the Start Trial API:

[source,json]
----------------------------------------------------------
POST _license/start_trial
----------------------------------------------------------
// CONSOLE

The 30-day trial enables you to try out the full set of platinum features,
including security, machine learning, alerting, graph capabilities, and more.

[role="xpack"]
[[xpack-stack-upgrade]]
=== Upgrading from 5.6
When you've made the necessary changes and are ready to upgrade from 6.7 to
{version}:

{xpack} 5.6 provides migration and upgrade APIs for Elasticsearch and a
Upgrade Assistant UI for Kibana. These tools are included with the trial
license and the free basic license.

To upgrade to {version} from 5.6:
. Test the upgrade in a dev environment *before* upgrading your
production cluster.

. {ref}/setup-upgrade.html[Upgrade Elasticsearch] to the most recent 5.6 and
install {xpack} on all nodes in your cluster. If you are upgrading from an
earlier 5.x release, you can perform a rolling upgrade. To upgrade from older
versions you must perform a full cluster restart.
+
If your trial license expires,
https://register.elastic.co/[register for a free Basic license]. To apply the
license, upload the license file with the `license` API:
+
[source,json]
----------------------------------------------------------
license -d @license.json
----------------------------------------------------------
. Back up your data. You **cannot roll back** to an earlier version unless
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran into a problem when I included the .kibana index in my snapshot. When I subsequently tried to restore this index it conflicted with the existing one. Can someone from @elastic/kibana-platform tell me if this was user error on my part or if it is expected?

If expected, then we should update the docs with guidance on avoiding this pitfall.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like general snapshot/restore behavior--you can't restore on top of an open index. As far as the upgrade process goes, we tell folks to back up their data, but don't really get into how to use the snapshot if the upgrade goes sideways. (I made the implicit assumption that they would understand how to use the backup if they needed to.) It might make sense to add a few words when we add the info about what to do if you have a "broken upgrade".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like general snapshot/restore behavior--you can't restore on top of an open index.

Thanks Deb! That makes sense.

It might make sense to add a few words when we add the info about what to do if you have a "broken upgrade".

Yes, that would be very helpful! Including a note about what this means for Kibana would also be helpful, I think. Something along the lines of, "Note for Kibana users: If you're restoring a snapshot that contains the .kibana index, you'll need to shut Kibana down, close the index, and restore using the command line".

you have a snapshot of your data. For information about creating snapshots, see
{ref}/modules-snapshots.html[Snapshot and Restore].

. If {xpack} **IS NOT** normally a part of your {stack}, disable the
{es} {security-features} in `elasticsearch.yml`:
+
[source,yaml]
----------------------------------------------------------
xpack.security.enabled: false
----------------------------------------------------------
. Consider closing {ml} jobs before you start the upgrade process. While {ml}
jobs can continue to run during a rolling upgrade, it increases the overhead
on the cluster during the upgrade process. For more information, see
{ref}/rolling-upgrades.html[Rolling upgrades].

. Upgrade {kib} to the most recent 5.6 and install {xpack}.
. Upgrade the components of your Elastic Stack in the following order:

. If you disabled the {es} {security-features}, also disable the {kib}
{security-features} in `kibana.yml`:
+
[source,yaml]
----------------------------------------------------------
xpack.security.enabled: false
----------------------------------------------------------

. Use the Upgrade Assistant in Kibana to
view incompatibilities that you need to fix, identify any 2.x indices that
need to be migrated or deleted, and upgrade the internal indices to the
{major-version} index format.
+
You can also call the Elasticsearch migration APIs directly:
+
`/_migration/assistance`:: Runs a series of checks on your cluster,
nodes, and indices and returns a list of issues that need to be
fixed before you can upgrade to {version}.
+
`/_migration/upgrade`:: Upgrades the indices for the {watcher} and
{security-features} to a single-type format compatible with Elasticsearch 6.x.
.. {es} Hadoop: {hadoop-ref}/install.html[install instructions]
.. {es}: {ref}/setup-upgrade.html[upgrade instructions]
.. Kibana: {kibana-ref}/upgrade.html[upgrade instructions]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per feedback from @tylersmalley, this is a little more complicated if you're using Kibana. Because Kibana comes with a new Upgrade Assistant which tells you if there are problems with your indices and cluster, the order should be:

  1. Once the user is on 6.7, go into Kibana and view the Upgrade Assistant. Address all critical items.
  2. Follow the ES upgrade steps.
  3. Follow the Kibana upgrade steps.

For context, here's what the Upgrade Assistant looks like (this screenshot may contain bugs so it's not appropriate for use in the docs):

image

.. Logstash: {logstash-ref}/upgrading-logstash.html[upgrade instructions]
.. Beats: {beats-ref}/upgrading.html[upgrade instructions]
.. APM Server: {apm-server-ref}/upgrading.html[upgrade instructions]

. Once you've resolved all of the migration issues, perform
a {ref}/rolling-upgrades.html[rolling upgrade] from Elasticsearch 5.6 to {version}.
NOTE: Logstash 6.7 and Beats 6.7 are compatible with all 7.x versions of
{es}. This provides flexibility in when you schedule the upgrades
for your Logstash instances and Beats agents, but we recommend upgrading as
soon as possible to take advantage of performance improvements
and other enhancements.

[[oss-stack-upgrade]]
=== Upgrading from a pre-5.6 installation

It is possible to upgrade directly to {major-version} from a pre-5.6 installation,
but it requires a {ref}/restart-upgrade.html[full cluster restart] and you must
manually reindex any 2.x indices you need to carry forward to {major-version}.

IMPORTANT: If you use Kibana or {xpack}, you also need to upgrade the
internal Kibana and {xpack} indices. For information about upgrading them
after you install Elasticsearch {version}, see
<<upgrade-internal-indices, Upgrading internal indices>>.

To manually reindex a 2.x index:

. Create an index with 6.x compatible mappings.
. Use the {ref}/docs-reindex.html[reindex API] to copy documents from the
2.x index into the new index. You can use a script to perform any necessary
modifications to the document data and metadata during reindexing.
. Use the {ref}/indices-aliases.html[_aliases] API to add the name of the 2.x
index as alias for the new index and delete the 2.x index.

[[upgrade-internal-indices]]
==== Upgrading internal indices for {major-version}

The format used for the internal indices used by Kibana and {xpack} has
changed in {major-version}. Before you can run Kibana and {xpack} in {version},
these indices must be upgraded to the new format. If you are upgrading from a
version prior to 5.6, you must upgrade them after after installing
Elasticsearch {version}.

To get a list of the indices that need to be upgraded, install {xpack} and use
the {ref}/migration-api-assistance.html[`_migration/assistance` API]:

[source,json]
----------------------------------------------------------
GET /_migration/assistance
----------------------------------------------------------
// CONSOLE

To upgrade the `.security` index:

. On a single node, add a temporary superuser account to the `file` realm.
. Use the {ref}/migration-api-upgrade.html[`_migration/upgrade`]
API to upgrade the security index, submitting the request with the credentials
for the temporary superuser:
+
--
[source,json]
----------------------------------------------------------
POST /_migration/upgrade/.security
----------------------------------------------------------
// CONSOLE
--

. Delete the temporary superuser account from the file realm.

You can use your regular administration credentials to upgrade the other
internal indices using the `_migration/upgrade` API.

TIP: Once you upgrade the `.kibana` index, you can run Kibana and use the
{xpack} Reindex Helper UI to upgrade the other indices.
=== Upgrading from 6.6 or earlier

To upgrade directly to {es} {version} from versions 6.0-6.6, you must
{ref}/reindex-upgrade.html[manually reindex] any 5.x indices you need to
carry forward, and perform a {ref}/restart-upgrade.html[full cluster restart].
This includes any internal indices created in 5.x, such as the `.kibana` and
`.security*` indices.

[TIP]
debadair marked this conversation as resolved.
Show resolved Hide resolved
====
Add a temporary superuser account in the `file` realm before reindexing the
internal `.security*` indices by running the
{ref}/users-command.html[elasticsearch-users useradd] command on a single node.
This provides a backup authentication method if there is a problem during
reindexing.
====

Make sure all 5.x indices have been deleted before upgrading to {version}. {es}
{version} will fail to start if any 5.x indices are present.

If you are running a version prior to 6.0,
https://www.elastic.co/guide/en/elastic-stack/6.7/upgrading-elastic-stack.html[upgrade to 6.7]
and reindex your old indices or bring up a new {version} cluster and
{ref}/reindex-upgrade-remote.html[reindex from remote].

debadair marked this conversation as resolved.
Show resolved Hide resolved
The recommended path is to upgrade to 6.7 before upgrading to {version}. This
makes it easier to identify the changes you need to make to upgrade and enables
you to perform a rolling upgrade with no downtime.

[[upgrade-elastic-stack-for-elastic-cloud]]
=== Upgrading on Elastic Cloud

A single click in the Elastic Cloud console can upgrade a cluster to a newer
version, add more processing capacity, change plugins, and enable or disable
high availability, all at the same time. During the upgrade process,
Elasticsearch, Kibana, {xpack} and the officially included plugins are
{es}, Kibana, {xpack} and the officially included plugins are
upgraded simultaneously.

Although upgrading your Elastic Cloud clusters is easy, you still need to
address breaking changes that affect your application. Minor version upgrades,
upgrades from 5.6 to {major-version}, and all other cluster configuration
upgrades from 6.7 to {version}, and all other cluster configuration
changes can be performed with no downtime.

To avoid downtime when a full cluster restart is required:

. Provision an additional cluster with the new Elasticsearch version, reindex
. Provision an additional cluster with the new {es} version, reindex
your data, and send index requests to both clusters temporarily.

. Verify that the new cluster performs as expected, fix any problems, and then
Expand Down