Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[UA] Fix reindex detection logic #202669

Open
jloleysens opened this issue Dec 3, 2024 · 2 comments
Open

[UA] Fix reindex detection logic #202669

jloleysens opened this issue Dec 3, 2024 · 2 comments
Labels
Feature:Upgrade Assistant Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.18.0 v9.0.0

Comments

@jloleysens
Copy link
Contributor

jloleysens commented Dec 3, 2024 •
edited
Loading

Kibana uses this code to detect if an index needs to be reindexed (data pulled from ES via _migration/deprecations):

https://github.com/elastic/kibana/blob/debe02bdb575be3ea53289aa723f43bc14ff36a2/x-pack/plugins/upgrade_assistant/server/lib/es_deprecations_status.ts#L306C9-L306C30

This depends on the contents of a message intended for human readability which is brittle. Today we are not surfacing deprecations like:

  "index_settings": {
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // This does not match our logic
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
  }

Resolution

  • ES will provide a machine-friendly way for us to detect indices/data streams that need to be reindexed by exposing a meta attribute (TBD)
  • We must use this instead of the regexp approach we are using today
@jloleysens jloleysens added Feature:Upgrade Assistant Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v9.0.0 labels Dec 3, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

This was referenced Dec 5, 2024
Open
Merged
jloleysens added a commit that referenced this issue Dec 5, 2024
@jloleysens
[UA] Ensure that old indices are detected for reindexing (#203082)
3579425 
## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">


## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* #202669
* elastic/elasticsearch#118062
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Dec 5, 2024
@jloleysens
[UA] Ensure that old indices are detected for reindexing (elastic#203082
dd708cb 
)

## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">

## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* elastic#202669
* elastic/elasticsearch#118062

(cherry picked from commit 3579425)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Dec 5, 2024
@jloleysens
[UA] Ensure that old indices are detected for reindexing (elastic#203082
6f97a43 
)

## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">

## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* elastic#202669
* elastic/elasticsearch#118062

(cherry picked from commit 3579425)
SoniaSanzV pushed a commit to SoniaSanzV/kibana that referenced this issue Dec 9, 2024
@jloleysens @SoniaSanzV
[UA] Ensure that old indices are detected for reindexing (elastic#203082
b5a8063 
)

## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">


## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* elastic#202669
* elastic/elasticsearch#118062
SoniaSanzV pushed a commit to SoniaSanzV/kibana that referenced this issue Dec 9, 2024
@jloleysens @SoniaSanzV
[UA] Ensure that old indices are detected for reindexing (elastic#203082
9e93047 
)

## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">


## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* elastic#202669
* elastic/elasticsearch#118062
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this issue Dec 9, 2024
@jloleysens @CAWilson94
[UA] Ensure that old indices are detected for reindexing (elastic#203082
4a61039 
)

## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">


## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* elastic#202669
* elastic/elasticsearch#118062
Samiul-TheSoccerFan pushed a commit to Samiul-TheSoccerFan/kibana that referenced this issue Dec 10, 2024
@jloleysens @Samiul-TheSoccerFan
[UA] Ensure that old indices are detected for reindexing (elastic#203082
c7f7073 
)

## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">


## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* elastic#202669
* elastic/elasticsearch#118062
@jloleysens
Copy link
Contributor Author

Note: to our knowledge this is currently not a blocker since we merged this workaround, but keeping this issue open due to not having a more formal solution.

mykolaharmash pushed a commit to mykolaharmash/kibana that referenced this issue Dec 11, 2024
@jloleysens @mykolaharmash
[UA] Ensure that old indices are detected for reindexing (elastic#203082
3018291 
)

## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">


## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* elastic#202669
* elastic/elasticsearch#118062
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this issue Dec 12, 2024
@jloleysens @CAWilson94
[UA] Ensure that old indices are detected for reindexing (elastic#203082
a9f0588 
)

## Summary

We need to handle the following response shape from `GET
_migration/deprecations`

```jsonc
    "myindex": [
      {
        "level": "critical",
        "message": "Old index with a compatibility version < 8.0", // specifically this message
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-9.0.html",
        "details": "This index has version: 7.17.25",
        "resolve_during_rolling_upgrade": false
      }
    ]
```

<img width="1142" alt="Screenshot 2024-12-05 at 12 44 59"
src="https://github.com/user-attachments/assets/723e19ab-dd9d-4b6a-bcda-26a5c8bffa0b">


## To reviewers

These changes were developed/tested on `8.x` and must be backported.

Related
* elastic#202669
* elastic/elasticsearch#118062
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature:Upgrade Assistant Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.18.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jloleysens @elasticmachine @rayafratkina