Remove fields from UA mappings which don't need to be searchable #64547
Assignees
Labels
chore
Feature:Upgrade Assistant
Team:Kibana Management
Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more
Comments
cjcenizal
added
chore
Team:Kibana Management
|
Pinging @elastic/es-ui (Team:Elasticsearch UI) |
|
This will address #43673. |
27 tasks
|
I wonder if we can prioritise this work, UA is currently using 18 fields for the |
This was referenced Feb 10, 2023
yuliacech
added a commit
that referenced
this issue
Feb 21, 2023
) ## Summary Partially addresses #64547 This PR removes unnecessary mappings from the saved objects type `upgrade-assistant-reindex-operation` that holds information about reindex processes. Currently, the reindexing is disabled via a config for Upgrade Assistant plugin `xpack.upgrade_assistant.featureSet.reindexCorrectiveActions`. ES also doesn't log any deprecations for indices created since 7.0 and indices created before 7.0 are incompatible with versions 8.0+. That means that when reindexing will be enabled in UA, it needs to be configured for the specific upgrade scenario, for example for 9.0. ### How to test Even though reindexing is currently disabled, I was able to do some testing for the changes. Here is how to test reindexing locally: 1. Add `xpack.upgrade_assistant.featureSet.reindexCorrectiveActions: true` to your `kibana.dev.yml` file 2. Start ES and Kibana with `yarn es snapshot --license=trial` and `yarn start` 3. Create a test user with the role `system_indices_superuser` (in Stack Management -> Security -> Users). Log in with that user for further steps. 4. Create an index via Dev Tools ``` POST /old_index/_doc { "test": "test" } ``` 5. replace `deprecations` on line 22 in the file [`/server/lib/es_deprecations_status.ts`](https://github.com/elastic/kibana/blob/main/x-pack/plugins/upgrade_assistant/server/lib/es_deprecations_status.ts#L22) with ``` const deprecations = { cluster_settings: [], node_settings: [], ml_settings: [], index_settings: { old_index: [ { level: 'critical', message: 'Index created before 7.0', url: 'https: //www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-8.0.html', details: 'This index was created using version: 6.8.13', resolve_during_rolling_upgrade: false, }, ], }, }; ``` 6. open UA and test the reindexing of `old_index` ### Follow up work A follow up PR if needed will be opened for saved objects types `upgrade-assistant-ml-upgrade-operation` and `upgrade-assistant-telemetry`
yuliacech
added a commit
that referenced
this issue
Feb 21, 2023
…1014) ## Summary Partially addresses #64547 Follow up to #150878 This PR removes unnecessary mappings from the saved objects type `upgrade-assistant-ml-upgrade-operation. This saved objects are used to store the resolution progress of ml snapshot deprecations. Currently, the ml deprecations are disabled via a config for Upgrade Assistant plugin `xpack.upgrade_assistant.featureSet.mlSnapshots`. And ES will probably not let us use any old ml snapshots in the 8.x version. ### How to test 1. Add `xpack.upgrade_assistant.featureSet.mlSnapshots: true` to your `kibana.dev.yml` file 2. Start ES and Kibana with `yarn es snapshot --license=trial` and `yarn start` 3. Add a sample data set, for example `kibana_sample_data_flights`. 4. Navigate to Analytics -> Machine Learning -> Anomaly Detection -> Jobs. 5. Create a job for a single metric, chose a time range with some data in it. Memorize the value you input for job ID. 6. Start and stop the job's datafeed a couple of times, so that you have more than 1 snapshot. Memorize the value of the snapshot ID. 7. replace `deprecations` on line 22 in the file [`/server/lib/es_deprecations_status.ts`](https://github.com/elastic/kibana/blob/main/x-pack/plugins/upgrade_assistant/server/lib/es_deprecations_status.ts#L22) with ``` const deprecations = { cluster_settings: [], node_settings: [], ml_settings: [ { level: 'critical', message: 'model snapshot [1676291073] for job [test_123] needs to be deleted or upgraded', url: '', details: 'details', _meta: { snapshot_id: '1676291073', job_id: 'test_123' }, resolve_during_rolling_upgrade: false, }, ], index_settings: {}, }; ``` where `job_id` and `snapshot_id` are values from step 5 and 6. 8. open UA and test the resolution of the ml snapshot deprecation.
yuliacech
added a commit
that referenced
this issue
Feb 27, 2023
## Summary Fixes #64547 Follow up to #150878 and #151014 This PR removes the saved object type `upgrade-assistant-telemetry` from the Upgrade Assistant plugin because it seems to not be used for telemetry or UA functionality. There is a telemetry object to track UA deprecation logging in this [file](https://github.com/elastic/kibana/blob/main/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json#L13628), I don't think that a saved object is needed for that. --------- Co-authored-by: kibanamachine <[email protected]>
bmorelli25
pushed a commit
to bmorelli25/kibana
that referenced
this issue
Mar 10, 2023
## Summary Fixes elastic#64547 Follow up to elastic#150878 and elastic#151014 This PR removes the saved object type `upgrade-assistant-telemetry` from the Upgrade Assistant plugin because it seems to not be used for telemetry or UA functionality. There is a telemetry object to track UA deprecation logging in this [file](https://github.com/elastic/kibana/blob/main/x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json#L13628), I don't think that a saved object is needed for that. --------- Co-authored-by: kibanamachine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Per convo with @joshdover, Upgrade Assistant's saved object mappings currently define (and therefore index) a number of fields which don't need to be mapped (see #64332).
We can set
dynamic: falseand then remove these fields from the mapping to avoid hitting the soft limit on the number of indexed fields in the.kibanaindex. These fields won't be indexed (searchable) but they'll still be persisted in_source.It looks like the only fields that need to be indexed are
indexNameandstatusbased on the fields originally defined in the mappings in the pre-NP code, though everything was being mapped due to the original use ofdynamic: true, so there's a possibility that code was depending on those dynamically mapped fields somewhere. When we make this change we should do some testing with Josh's help to make sure we haven't broken anything.The text was updated successfully, but these errors were encountered: