Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

8.8.0 test migrations success even on temporary ES failures #158818

Closed
Bamieh opened this issue Jun 1, 2023 · 3 comments
Closed

8.8.0 test migrations success even on temporary ES failures #158818

Bamieh opened this issue Jun 1, 2023 · 3 comments
Assignees
@gsoldevila
Labels
bug Fixes for quality problems that affect the customer experience Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.8.0

Comments

@Bamieh
Copy link
Member

Bamieh commented Jun 1, 2023 •
edited
Loading

Write test cases to check that when Kibana migrations fail due to a temporary problem in Elasticsearch, Kibana can automatically finish the migration when the failure condition is resolved.

2 Levels to testing migrator failures:

  1. Kibana instance failure
    a. Kill instance to mimic failure
  2. Inside a single instance, we can fail 1 migrator
    a. Introduce failures inside ES wrapper

These tests should focus on the second level.

We can introduce the failures at these migrator steps:

  1. fail es at alias change (final step)
  2. fail at the update target mappings (modifying operations, firs 2 ops)
  3. fail at any step in the clone target mappings
    a. update target mappings properties

Root cause Issue here: #158733
@Bamieh Bamieh added bug Fixes for quality problems that affect the customer experience Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects v8.8.0 labels Jun 1, 2023
@Bamieh Bamieh self-assigned this Jun 1, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Jun 1, 2023
@Bamieh Bamieh mentioned this issue Jun 4, 2023
Closed
4 tasks
@Bamieh Bamieh assigned gsoldevila and unassigned gsoldevila Jun 4, 2023
@davismcphee davismcphee added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc and removed needs-team Issues missing a team label labels Jun 14, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@Bamieh
Copy link
Member Author

Bamieh commented Aug 3, 2023 •
edited
Loading

The main testing for the temporary ES failure is already done and merged here #158940

Monkey testing (Draft PR here) needs more time investment since the results with the current implementation results in a flaky output.
Putting this under blocked until we address if we need to work more on it or if we are OK with the current testing situation.

@gsoldevila gsoldevila mentioned this issue Sep 21, 2023
Merged
gsoldevila added a commit that referenced this issue Sep 25, 2023
@gsoldevila
[Migrations] Ensure individual migrator failures do not break consist…
e4a088c 
…ency (#166924)

## Summary

Tackles #158818

The goal of the PR is to introduce failures in single migrators at some
of the crucial steps of the migration, testing that consistency is
maintained, and that subsequent migration attempts can successfully
complete the upgrade.

This is done by _proxying_ the `Client` class, which uses the
elasticsearch-js library underneath to perform all calls to ES.
Inspired on #158995.
@rudolf
Copy link
Contributor

rudolf commented Oct 12, 2023

Closed in #166924

@rudolf rudolf closed this as completed Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gsoldevila gsoldevila

Labels
bug Fixes for quality problems that affect the customer experience Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.8.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Test migrations success even on temporary ES failures Bamieh/kibana
5 participants
@rudolf @Bamieh @elasticmachine @gsoldevila @davismcphee