Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade assistant|reindex: Prompt to remove include_in_all in addition to _all #41213

Closed
ppf2 opened this issue Jul 16, 2019 · 4 comments
Closed
Labels
bug Fixes for quality problems that affect the customer experience Feature:Upgrade Assistant Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more

Comments

@ppf2
Copy link
Member

ppf2 commented Jul 16, 2019

5.6 -> 6.8

It looks like there is a bug (miss) introduced as part of #27457.

In 27457, we added an automatic index changes feature as part of the reindexing functionality. For 6.7.0, these are limited to these 2 items below:

  • Removal of _all meta field from mappings
  • Conversion of deprecated boolean fields ("yes", "off", etc.). This is accomplished with a painless script added to the reindex command if any boolean fields are detected in the mapping.

However, item 1 above is not sufficient depending on the use case.

To reproduce:

  1. Create an index in 5.6 with a mapping that doesn't only have _all enabled, but mapped fields that have include_in_all:

image

  1. Upgrade to 6.7 (reproducible in both 6.7.0 and 6.8.1)

  2. Launch the upgrade assistant:

It will detect that this index was created in 5.x and requires reindexing. If you proceed to reindex, it will ask for the confirmation to perform the automatic index change as part of reindexing (i.e., removal of the _all field).

image

  1. The removal of the all field is successful. Unfortunately, reindexing fails because the target index it attempts to create has the include_in_all option still intact (so it is rejected by Elasticsearch 6.7):
{"reindexOp":{"indexName":"logstash-ams_metrics-2017.08.15_2","newIndexName":"reindexed-v6-logstash-ams_metrics-2017.08.15_2","status":2,"lastCompletedStep":20,"locked":"2019-07-15T18:39:40-07:00","reindexTaskId":null,"reindexTaskPercComplete":null,"errorMessage":"[mapper_parsing_exception] [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field. :: {\"path\":\"/reindexed-v6-logstash-ams_metrics-2017.08.15_2\",\"query\":{},\"body\":\"{\\\"settings\\\":{\\\"index.merge.scheduler.max_thread_count\\\":\\\"1\\\",\\\"index.number_of_replicas\\\":\\\"1\\\",\\\"index.number_of_shards\\\":\\\"1\\\",\\\"index.refresh_interval\\\":\\\"5s\\\",\\\"index.routing.allocation.total_shards_per_node\\\":\\\"3\\\"},\\\"mappings\\\":{\\\"logs\\\":{\\\"dynamic_templates\\\":[{\\\"message_field\\\":{\\\"path_match\\\":\\\"message\\\",\\\"match_mapping_type\\\":\\\"string\\\",\\\"mapping\\\":{\\\"norms\\\":false,\\\"type\\\":\\\"text\\\"}}},{\\\"string_fields\\\":{\\\"match\\\":\\\"*\\\",\\\"match_mapping_type\\\":\\\"string\\\",\\\"mapping\\\":{\\\"fields\\\":{\\\"keyword\\\":{\\\"type\\\":\\\"keyword\\\"}},\\\"norms\\\":false,\\\"type\\\":\\\"text\\\"}}}],\\\"properties\\\":{\\\"@timestamp\\\":{\\\"type\\\":\\\"date\\\",\\\"include_in_all\\\":false},\\\"@version\\\":{\\\"type\\\":\\\"keyword\\\",\\\"include_in_all\\\":false},\\\"DC\\\":{\\\"type\\\":\\\"text\\\",\\\"norms\\\":false,\\\"fields\\\":{\\\"keyword\\\":{\\\"type\\\":\\\"keyword\\\"}}},\\\"domainID\\\":{\\\"type\\\":\\\"text\\\",\\\"norms\\\":false,\\\"fields\\\":{\\\"keyword\\\":{\\\"type\\\":\\\"keyword\\\"}}},\\\"domainName\\\":{\\\"type\\\":\\\"text\\\",\\\"norms\\\":false,\\\"fields\\\":{\\\"keyword\\\":{\\\"type\\\":\\\"keyword\\\"}}},\\\"frmAgents\\\":{\\\"type\\\":\\\"long\\\"},\\\"geoip\\\":{\\\"dynamic\\\":\\\"true\\\",\\\"properties\\\":{\\\"ip\\\":{\\\"type\\\":\\\"ip\\\"},\\\"latitude\\\":{\\\"type\\\":\\\"half_float\\\"},\\\"location\\\":{\\\"type\\\":\\\"geo_point\\\"},\\\"longitude\\\":{\\\"type\\\":\\\"half_float\\\"}}},\\\"host\\\":{\\\"type\\\":\\\"text\\\",\\\"norms\\\":false,\\\"fields\\\":{\\\"keyword\\\":{\\\"type\\\":\\\"keyword\\\"}}},\\\"message\\\":{\\\"type\\\":\\\"text\\\",\\\"norms\\\":false},\\\"tags\\\":{\\\"type\\\":\\\"text\\\",\\\"norms\\\":false,\\\"fields\\\":{\\\"keyword\\\":{\\\"type\\\":\\\"keyword\\\"}}},\\\"totalAgents\\\":{\\\"type\\\":\\\"long\\\"},\\\"vccAdmin\\\":{\\\"type\\\":\\\"long\\\"},\\\"vccAgents\\\":{\\\"type\\\":\\\"long\\\"},\\\"vccSuper\\\":{\\\"type\\\":\\\"long\\\"},\\\"webAPIAdmin\\\":{\\\"type\\\":\\\"long\\\"},\\\"webSuper\\\":{\\\"type\\\":\\\"long\\\"}}}}}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"mapper_parsing_exception\\\",\\\"reason\\\":\\\"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.\\\"}],\\\"type\\\":\\\"mapper_parsing_exception\\\",\\\"reason\\\":\\\"Failed to parse mapping [logs]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"mapper_parsing_exception\\\",\\\"reason\\\":\\\"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.\\\"}},\\\"status\\\":400}\"}","runningReindexCount":null},"warnings":[0],"hasRequiredPrivileges":true}

If you look at the mapping included in Kibana's rest API exception above, you will see that the /reindexed-v6-logstash-ams_metrics-2017.08.15_2 index it tries to create is using a mapping that has the _all successfully removed, however, include_in_all is present so it is causing Elasticsearch to reject this new index.

Users with _all and include_in_all will likely have to workaround this by calling the ES reindex API directly outside of the upgrade assistant UI to perform the reindexing (after creating a target index mapping that is valid in 6.x first).

@ppf2 ppf2 added bug Fixes for quality problems that affect the customer experience Feature:Upgrade Assistant labels Jul 16, 2019
@tylersmalley
Copy link
Contributor

tylersmalley commented Jul 16, 2019

It appears this is in-fact a bug in the upgrade assistant as we do not account for also removing the include_in_all fields. Manually re-indexing would be the only workaround until we can get a fix in for this.

@tylersmalley tylersmalley added Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more Team:Operations Team label for Operations Team labels Jul 16, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/es-ui

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations

@tylersmalley tylersmalley removed the Team:Operations Team label for Operations Team label Oct 1, 2019
@alisonelizabeth
Copy link
Contributor

I'm going to close this issue since we are now focusing our efforts on the next major upgrade (7.x --> 8.0), where this will no longer be an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Feature:Upgrade Assistant Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more
Projects
None yet
Development

No branches or pull requests

4 participants