Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reindexing shrunken indices fails via the Upgrade Assistant #18469

Closed
elasticmachine opened this issue Mar 12, 2018 · 3 comments
Closed

Reindexing shrunken indices fails via the Upgrade Assistant #18469

elasticmachine opened this issue Mar 12, 2018 · 3 comments
Assignees
Labels
bug Fixes for quality problems that affect the customer experience Team:Operations Team label for Operations Team

Comments

@elasticmachine
Copy link
Contributor

Original comment by @talevy:

Kibana version: 5.6.x

Description of the problem including expected versus actual behavior:

Shrunken indices in Elasticsearch leverage internal indices for speedier allocation. The problem
with this is that there is no way to differentiate internal vs. public settings when copying over
settings to the upgraded index. For this reason, Reindexing a shrunken index fails within the Upgrade
Assistant.

It seems like the offending settings should be blacklisted from the upgraded index here:
LINK REDACTED

index.routing.allocation.initial_recovery._id should be added to this list.

Steps to reproduce:

  1. create index in 5.x
  2. shrink said index
  3. use the reindex helper to upgrade it

Provide logs and/or server output (if relevant):

{
    "statusCode": 500,
    "error": "Internal Server Error",
    "message": "An internal server error occurred",
    "code": "ERR_CREATE_INDEX_FAILED",
    "root_cause": [{
        "type": "remote_transport_exception",
        "reason": "[ifodjsijfdo][10.0.0.1:9300][indices:admin/create]"
    }],
    "type": "illegal_argument_exception",
    "reason": "unknown setting [index.routing.allocation.initial_recovery._id] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
    "reindexedIndexName": "index-shrink-reindexed-v5"
}
@elasticmachine elasticmachine added Team:Operations Team label for Operations Team bug Fixes for quality problems that affect the customer experience labels Apr 25, 2018
@joshdover
Copy link
Contributor

@talevy Sounds like in the new upgrade assistant we should blacklist this setting from being copied?

@joshdover joshdover self-assigned this Nov 29, 2018
@talevy
Copy link
Contributor

talevy commented Nov 30, 2018

@joshdover this may be worth discussing, but I do not see any reason why a reindexed copy of a shrunken index should retain this setting. It is an internal setting to the resize operations, and only concerns the internals of those operations. That is why it is not a registered setting that is meant to be set on an index during index creation. So, I would imagine that this is blacklisted for that reason, yeah

joshdover added a commit that referenced this issue Dec 18, 2018
* Get basic scaffolding working

* Wire up cluster checkup data

* Add types for elasticsearch plugin

* Implement basics of checkup tab

* Update style of deprecations

* Add copy and reload button to checkup

* Add filtering by warning level

* Add deprecation logging tab

* Copy updates and cleanup

* Type cleanup

* Move deprecation logging to overview tab

* Make filters togglable

* Move sections into tabs and add support for grouping

* Cleanup and add clearer labels

* Use tables for message grouping

* Cleanup and small fixes

* Allow console to load relative URLs

* Add reindex in console button to reindex tasks

* Merge documentation UI and uiButtons

* Fix tests

* Filter bar tweaks

* Filter out index settings that can't be set

* Fix types

* Add tests for deprecation_logging

* Add tests for reindex templates

* Make KibanaConfig generic

* Simplify integration test

* Finish backend unit tests

* Fixup types

* Fix uiButton updating for reindex items

* Fixed background color stretching

* Pulling tabs out and re-ordering filter buttons

* Making accordions more item-list like

* Turned Healths into Badges

- Couldn’t do the conversion within the cell because it only passed color

* Fix overflow issue

* Optional filter and expand/collapse controls

* Reorganizing

- Added placeholder for moving action button up into accordion header
- Removed repetitive message name outputs
- Slightly better listing of each message when sorting by index
- Only showing number of severity when sorting by index
  - Still need to allow showing all severity levels
- Added indice count when sorting by issue

* Putting `Deprecation logging` in a `EuiDescribedFormGroup`

* Added some stats, empty prompts, and all clear prompt

* Added docs link

* Cleaned up sass files

* Revert changes to fake_deprecations

* Update blacklisted settings

* wip

* Move data fetching and tab control

* Wire up overview summary

* Cleanup docs/uiButtons + move actions to index table

* Add expand/collapse all functionality

* Wire up search box

* Wire up severity indicators

* Fix types

* Round out functional tests

* Fix fake data

* Remove info deprecation level

* Fix extra space on cluster tab control bar

* Cleanup code and localize majority of UI controls

* Change overview tab to steps layout

* Update copy

* Localize overview tab

* Complete localization of checkup tabs

* Make ES version dynamic based on branch

* Add pagination to checkup tabs

* Rename checkup -> assistant

* Cleanup filter and group by bars

* WIP UI unit tests

* Copy tweaks

* Fix i18n formatting issues

* Update tests for copy

* Add tests for remaining UI

* Fix pagination w/ filter changes + table button color

* Small cleanup

* Add reindex button to old index deprecations

* Add shrunken indices setting to copy settings blacklist for #18469

* Add next steps to overview tab + update copy

* Remove usage of migration assistance API

* Use all/critical toggle for filter buttons

* Cloud upgrade copy

* Translate reindex button

* Remove hacked EUI type

* Show incomplete banner on all tabs

* Update copy for waiting for next version

* Review comments

* Update deprecation level type

* Update checkup tab snapshots

* Remove dependencies on types from #25168

* Use types from new global type defs

* Remove 'Reindex in Console' button

* Remove unused variable
joshdover added a commit to joshdover/kibana that referenced this issue Dec 18, 2018
* Get basic scaffolding working

* Wire up cluster checkup data

* Add types for elasticsearch plugin

* Implement basics of checkup tab

* Update style of deprecations

* Add copy and reload button to checkup

* Add filtering by warning level

* Add deprecation logging tab

* Copy updates and cleanup

* Type cleanup

* Move deprecation logging to overview tab

* Make filters togglable

* Move sections into tabs and add support for grouping

* Cleanup and add clearer labels

* Use tables for message grouping

* Cleanup and small fixes

* Allow console to load relative URLs

* Add reindex in console button to reindex tasks

* Merge documentation UI and uiButtons

* Fix tests

* Filter bar tweaks

* Filter out index settings that can't be set

* Fix types

* Add tests for deprecation_logging

* Add tests for reindex templates

* Make KibanaConfig generic

* Simplify integration test

* Finish backend unit tests

* Fixup types

* Fix uiButton updating for reindex items

* Fixed background color stretching

* Pulling tabs out and re-ordering filter buttons

* Making accordions more item-list like

* Turned Healths into Badges

- Couldn’t do the conversion within the cell because it only passed color

* Fix overflow issue

* Optional filter and expand/collapse controls

* Reorganizing

- Added placeholder for moving action button up into accordion header
- Removed repetitive message name outputs
- Slightly better listing of each message when sorting by index
- Only showing number of severity when sorting by index
  - Still need to allow showing all severity levels
- Added indice count when sorting by issue

* Putting `Deprecation logging` in a `EuiDescribedFormGroup`

* Added some stats, empty prompts, and all clear prompt

* Added docs link

* Cleaned up sass files

* Revert changes to fake_deprecations

* Update blacklisted settings

* wip

* Move data fetching and tab control

* Wire up overview summary

* Cleanup docs/uiButtons + move actions to index table

* Add expand/collapse all functionality

* Wire up search box

* Wire up severity indicators

* Fix types

* Round out functional tests

* Fix fake data

* Remove info deprecation level

* Fix extra space on cluster tab control bar

* Cleanup code and localize majority of UI controls

* Change overview tab to steps layout

* Update copy

* Localize overview tab

* Complete localization of checkup tabs

* Make ES version dynamic based on branch

* Add pagination to checkup tabs

* Rename checkup -> assistant

* Cleanup filter and group by bars

* WIP UI unit tests

* Copy tweaks

* Fix i18n formatting issues

* Update tests for copy

* Add tests for remaining UI

* Fix pagination w/ filter changes + table button color

* Small cleanup

* Add reindex button to old index deprecations

* Add shrunken indices setting to copy settings blacklist for elastic#18469

* Add next steps to overview tab + update copy

* Remove usage of migration assistance API

* Use all/critical toggle for filter buttons

* Cloud upgrade copy

* Translate reindex button

* Remove hacked EUI type

* Show incomplete banner on all tabs

* Update copy for waiting for next version

* Review comments

* Update deprecation level type

* Update checkup tab snapshots

* Remove dependencies on types from elastic#25168

* Use types from new global type defs

* Remove 'Reindex in Console' button

* Remove unused variable
@joshdover
Copy link
Contributor

We removed the reindexing feature in Phase 1. This will actually be closed in a future issue.

@joshdover joshdover reopened this Dec 18, 2018
joshdover added a commit that referenced this issue Dec 19, 2018
* Upgrade assistant - Phase 1 (#25168)

* Get basic scaffolding working

* Wire up cluster checkup data

* Add types for elasticsearch plugin

* Implement basics of checkup tab

* Update style of deprecations

* Add copy and reload button to checkup

* Add filtering by warning level

* Add deprecation logging tab

* Copy updates and cleanup

* Type cleanup

* Move deprecation logging to overview tab

* Make filters togglable

* Move sections into tabs and add support for grouping

* Cleanup and add clearer labels

* Use tables for message grouping

* Cleanup and small fixes

* Allow console to load relative URLs

* Add reindex in console button to reindex tasks

* Merge documentation UI and uiButtons

* Fix tests

* Filter bar tweaks

* Filter out index settings that can't be set

* Fix types

* Add tests for deprecation_logging

* Add tests for reindex templates

* Make KibanaConfig generic

* Simplify integration test

* Finish backend unit tests

* Fixup types

* Fix uiButton updating for reindex items

* Fixed background color stretching

* Pulling tabs out and re-ordering filter buttons

* Making accordions more item-list like

* Turned Healths into Badges

- Couldn’t do the conversion within the cell because it only passed color

* Fix overflow issue

* Optional filter and expand/collapse controls

* Reorganizing

- Added placeholder for moving action button up into accordion header
- Removed repetitive message name outputs
- Slightly better listing of each message when sorting by index
- Only showing number of severity when sorting by index
  - Still need to allow showing all severity levels
- Added indice count when sorting by issue

* Putting `Deprecation logging` in a `EuiDescribedFormGroup`

* Added some stats, empty prompts, and all clear prompt

* Added docs link

* Cleaned up sass files

* Revert changes to fake_deprecations

* Update blacklisted settings

* wip

* Move data fetching and tab control

* Wire up overview summary

* Cleanup docs/uiButtons + move actions to index table

* Add expand/collapse all functionality

* Wire up search box

* Wire up severity indicators

* Fix types

* Round out functional tests

* Fix fake data

* Remove info deprecation level

* Fix extra space on cluster tab control bar

* Cleanup code and localize majority of UI controls

* Change overview tab to steps layout

* Update copy

* Localize overview tab

* Complete localization of checkup tabs

* Make ES version dynamic based on branch

* Add pagination to checkup tabs

* Rename checkup -> assistant

* Cleanup filter and group by bars

* WIP UI unit tests

* Copy tweaks

* Fix i18n formatting issues

* Update tests for copy

* Add tests for remaining UI

* Fix pagination w/ filter changes + table button color

* Small cleanup

* Add reindex button to old index deprecations

* Add shrunken indices setting to copy settings blacklist for #18469

* Add next steps to overview tab + update copy

* Remove usage of migration assistance API

* Use all/critical toggle for filter buttons

* Cloud upgrade copy

* Translate reindex button

* Remove hacked EUI type

* Show incomplete banner on all tabs

* Update copy for waiting for next version

* Review comments

* Update deprecation level type

* Update checkup tab snapshots

* Remove dependencies on types from #25168

* Use types from new global type defs

* Remove 'Reindex in Console' button

* Remove unused variable

* Update snapshots
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Team:Operations Team label for Operations Team
Projects
None yet
Development

No branches or pull requests

3 participants