Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Clarifies how to upgrade internal indices #235

Closed
wants to merge 5 commits into from
Closed

[DOCS] Clarifies how to upgrade internal indices #235

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
129b031
[DOCS] Clarifies when to reindex internal indices
lcawl Mar 7, 2019
b5078ff
[DOCS] Reformats file realm tip
lcawl Mar 7, 2019
1fd7781
[DOCS] Changes section level for upgrading internal indices
lcawl Mar 8, 2019
19ad84f
[DOCS] Change upgrade API to reindex API
lcawl Mar 13, 2019
4d0cf92
[DOCS] Adds additional links
lcawl Mar 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/en/install-upgrade/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
:kib-repo-dir: {docdir}/../../../../kibana/docs

include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
include::{es-repo-dir}/Versions.asciidoc[]
include::{asciidoc-dir}/../../shared/versions.asciidoc[]

include::overview.asciidoc[]

Expand Down
70 changes: 41 additions & 29 deletions docs/en/install-upgrade/upgrading-stack.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,45 +215,57 @@ modifications to the document data and metadata during reindexing.
index as alias for the new index and delete the 2.x index.

[[upgrade-internal-indices]]
==== Upgrading internal indices for {major-version}

The format used for the internal indices used by Kibana and {xpack} has
changed in {major-version}. Before you can run Kibana and {xpack} in {version},
these indices must be upgraded to the new format. If you are upgrading from a
version prior to 5.6, you must upgrade them after after installing
Elasticsearch {version}.

To get a list of the indices that need to be upgraded, install {xpack} and use
the {ref}/migration-api-assistance.html[`_migration/assistance` API]:

[source,json]
=== Upgrading internal indices for {major-version}

{es} can read indices that were created in the *previous major version*. Older
indices must be reindexed or deleted. For example, {es} {major-version} can use
indices created in {es} {prev-major-version}, but not those created in earlier
versions. This restriction also applies to the internal indices that are used by
Kibana and the {xpack} features. Therefore, before you can use Kibana and
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that Kibana reindexes it's own index these days. @epixa can you confirm?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do, but if someone was attempting to upgrade from something like 6.2 to 7.x, then they wouldn't have run saved object migrations yet, so their index could have originated back in 5.6. So they'll still need to run something like 6.7 first to ensure the index is migrated.

We also don't delete the old indices automatically, we just create new ones for migrations. Does ES fail to start if it has old indices or does it just not recognize those indices?

Finally, this blurb of text would still mostly make sense even with Kibana doing migrations of its own index. We indirectly rely on ES things like the security index, so if that needs to be functional in order for Kibana (with security) to be accessible.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @epixa .

We also don't delete the old indices automatically, we just create new ones for migrations. Does ES fail to start if it has old indices or does it just not recognize those indices?

Yes. ES will refuse to start.

but if someone was attempting to upgrade from something like 6.2 to 7.x, then they wouldn't have run saved object migrations yet, so their index could have originated back in 5.6

This implies that they'll do a full cluster restart, which will also mean that their ES 7.0 will indeed fail to start and they will be forced to downgrade and fix it.

@lcawl part of the confusion is caused by the usage of the word "upgrade" - in 5.x we had a dedicated upgrade API that people needed to activate. With 6.x it's a vanilla reindex via the reindex API (or the migration assistant UI). I think we should change the doc a bit to reflect this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bleskes Do you mean that in these steps we should be advising users to call the reindex API (https://www.elastic.co/guide/en/elasticsearch/reference/7.0/docs-reindex.html) instead of the migration upgrade API
(https://www.elastic.co/guide/en/elasticsearch/reference/7.0/migration-api-upgrade.html)?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean that in these steps we should be advising users to call the reindex API instead of the migration upgrade API

We should tell people that the index needs to be reindexed. That can be done via the migration assistant UI or alternatively directly via the reindex API (In that order ;))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bleskes ! I've updated the text to reflect that. Should we also update the documentation for the migration upgrade API (https://www.elastic.co/guide/en/elasticsearch/reference/master/migration-api-upgrade.html) to recommend that they use the reindex API instead?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx @lcawl . Re the upgrade api - I had a quick chat with @gwbrown about what to do with them, I think he was taking it up to the team to decide. @gwbrown , do I recall correctly?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should also update the docs for the Migration Upgrade API to point users towards the Kibana Upgrade Assistant - we're also going to be deprecating the Migration Assistance and Migration Upgrade APIs in 6.7 and removing in 7.0, see elastic/elasticsearch#40014. I'll have PRs up for that later today.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also implemented those changes in #247

{xpack} features in {version}, you must ensure the internal indices have a
compatible index structure.

If you are upgrading from a version prior to {prev-major-version}, you must
upgrade to the most recent version of {prev-major-version} first. Then you can
use the {kibana-ref}/upgrade-assistant.html[{kib} Upgrade Assistant] or the
{ref}/docs-reindex.html[reindex API] to reindex the internal indices. For more
information, see {ref}/reindex-upgrade.html[Reindex before upgrading].

[TIP]
====
If you use {stack} {security-features}, before you reindex `.security*` internal
indices it is a good idea to create a temporary superuser account in the `file`
realm.

. On a single node, add a temporary superuser account to the `file` realm. For
example, run the {ref}/users-command.html[elasticsearch-users useradd] command:
+
--
[source,sh]
----------------------------------------------------------
GET /_migration/assistance
bin/elasticsearch-users useradd <user_name> \
-p <password> -r superuser
----------------------------------------------------------
// CONSOLE
--

To upgrade the `.security` index:
. Use these credentials when you reindex the `.security*` index. That is to say,
use them to log into {kib} and run the Upgrade Assistant or to call the
reindex API. You can use your regular administration credentials to
reindex the other internal indices.

. On a single node, add a temporary superuser account to the `file` realm.
. Use the {ref}/migration-api-upgrade.html[`_migration/upgrade`]
API to upgrade the security index, submitting the request with the credentials
for the temporary superuser:
. Delete the temporary superuser account from the file realm. For
example, run the {ref}/users-command.html[elasticsearch-users userdel] command:
+
--
[source,json]
[source,sh]
----------------------------------------------------------
POST /_migration/upgrade/.security
bin/elasticsearch-users userdel <user_name>
----------------------------------------------------------
// CONSOLE
--

. Delete the temporary superuser account from the file realm.

You can use your regular administration credentials to upgrade the other
internal indices using the `_migration/upgrade` API.

TIP: Once you upgrade the `.kibana` index, you can run Kibana and use the
{xpack} Reindex Helper UI to upgrade the other indices.
For more information, see
{ref}/configuring-file-realm.html[Configuring a file realm].
====

[[upgrade-elastic-stack-for-elastic-cloud]]
=== Upgrading on Elastic Cloud
Expand Down