Migration v2 waits for yellow cluster #96788

mshustov · 2021-04-12T14:30:02Z

Summary

Closes #91107
This PR tries to stabilize the SO integration tests that fail since Kibana sent reindex requests to the red cluster.
Also, I added logs on how long every state transition lasts.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios
This was checked for breaking API changes and was labeled appropriately

otherwise the next request to Elasticsearch can fail

elasticmachine · 2021-04-12T14:30:05Z

Pinging @elastic/kibana-core (Team:Core)

mshustov · 2021-04-12T14:31:42Z

src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.ts

@@ -85,11 +87,12 @@ export async function migrationStateActionMachine({
  model: Model<State>;
 }) {
  const executionLog: ExecutionLog = [];
-  const starteTime = performance.now();


@rudolf I'm not sure we need such high precision. I rolled back to the standard Date instead. Let me know if I should revert the change.

yeah Date.now() is good enough.

mshustov · 2021-04-12T14:33:08Z

src/core/server/saved_objects/migrationsv2/model.test.ts

      });
-      test('INIT -> SET_SOURCE_WRITE_BLOCK when migrating from a v2 migrations index (>= 7.11.0)', () => {
+      test('INIT -> WAIT_FOR_YELLOW_SOURCE when migrating from a v2 migrations index (>= 7.11.0)', () => {


@rudolf
the master version of 0013_saved_object_migrations.md doesn't contain description for every implementation step. I could add WAIT_FOR_YELLOW_SOURCE to #93002

yes, it would be great to document it there.

rudolf

Wondered about adding an integration test, otherwise looks good.

rudolf · 2021-04-12T14:36:26Z

src/core/server/saved_objects/migrationsv2/integration_tests/migration.test.ts

+const asyncUnlink = Util.promisify(Fs.unlink);
+async function removeLogFile() {
+  // ignore errors if it doesn't exist
+  await asyncUnlink(logFilePath).catch(() => void 0);


rudolf · 2021-04-12T14:41:31Z

src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.ts

@@ -85,11 +87,12 @@ export async function migrationStateActionMachine({
  model: Model<State>;
 }) {
  const executionLog: ExecutionLog = [];
-  const starteTime = performance.now();


yeah Date.now() is good enough.

rudolf · 2021-04-12T14:42:47Z

src/core/server/saved_objects/migrationsv2/model.test.ts

      });
-      test('INIT -> SET_SOURCE_WRITE_BLOCK when migrating from a v2 migrations index (>= 7.11.0)', () => {
+      test('INIT -> WAIT_FOR_YELLOW_SOURCE when migrating from a v2 migrations index (>= 7.11.0)', () => {


yes, it would be great to document it there.

src/core/server/saved_objects/migrationsv2/actions/index.ts

src/core/server/saved_objects/migrationsv2/integration_tests/actions.test.ts

…ctions.test.ts Co-authored-by: Rudolf Meijering <[email protected]>

kibanamachine · 2021-04-12T18:38:45Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #118837 succeeded f45bb89
💚 Build #118793 succeeded 708d47d
💔 Build #118743 failed f4ff0ac

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2021-04-12T18:41:42Z

💔 Backport failed

Status	Branch	Result
❌	7.12	Commit could not be cherrypicked due to conflicts
❌	7.x	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 96788

mistic · 2021-04-12T19:55:38Z

@mshustov looks like that PR has made some jest tests to fail. I've skipped those at fb24006

* migrator waits for source index to be yellow otherwise the next request to Elasticsearch can fail * unskip integration tests that failed due to a red cluster * log how much the every step lasts * use Date.now instead of performance.now migration cannot finish in ms * update tests * clean log file before running tests * fix wrong type * add an integration test for waitForIndexStatusYellow # Conflicts: # src/core/server/saved_objects/migrationsv2/integration_tests/migration.test.ts

* Migration v2 waits for yellow cluster (#96788) * migrator waits for source index to be yellow otherwise the next request to Elasticsearch can fail * unskip integration tests that failed due to a red cluster * log how much the every step lasts * use Date.now instead of performance.now migration cannot finish in ms * update tests * clean log file before running tests * fix wrong type * add an integration test for waitForIndexStatusYellow # Conflicts: # src/core/server/saved_objects/migrationsv2/integration_tests/migration.test.ts * remove tests using so from v8 * skip as on master :(

* Migration v2 waits for yellow cluster (#96788) * migrator waits for source index to be yellow otherwise the next request to Elasticsearch can fail * unskip integration tests that failed due to a red cluster * log how much the every step lasts * use Date.now instead of performance.now migration cannot finish in ms * update tests * clean log file before running tests * fix wrong type * add an integration test for waitForIndexStatusYellow # Conflicts: # src/core/server/saved_objects/migrationsv2/integration_tests/migration.test.ts * remove test using so from v8 * skip as on master Co-authored-by: Kibana Machine <[email protected]>

mshustov added 6 commits April 12, 2021 15:51

migrator waits for source index to be yellow

8efae96

otherwise the next request to Elasticsearch can fail

unskip integration tests that failed due to a red cluster

ee72266

log how much the every step lasts

12f7305

use Date.now instead of performance.now migration cannot finish in ms

13510a2

update tests

33e7db4

clean log file before running tests

f4ff0ac

mshustov added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.0.0 project:ResilientSavedObjectMigrations Reduce Kibana upgrade failures by making saved object migrations more resilient v7.13.0 labels Apr 12, 2021

mshustov requested a review from a team as a code owner April 12, 2021 14:30

mshustov commented Apr 12, 2021

View reviewed changes

mshustov added the release_note:skip Skip the PR/issue when compiling release notes label Apr 12, 2021

mshustov enabled auto-merge (squash) April 12, 2021 14:36

rudolf reviewed Apr 12, 2021

View reviewed changes

mshustov marked this pull request as draft April 12, 2021 15:12

auto-merge was automatically disabled April 12, 2021 15:12
Pull request was converted to draft

mshustov added 2 commits April 12, 2021 17:43

fix wrong type

7a43549

add an integration test for waitForIndexStatusYellow

708d47d

mshustov marked this pull request as ready for review April 12, 2021 15:47

mshustov requested a review from rudolf April 12, 2021 15:47

mshustov enabled auto-merge (squash) April 12, 2021 15:59

rudolf approved these changes Apr 12, 2021

View reviewed changes

src/core/server/saved_objects/migrationsv2/integration_tests/actions.test.ts Outdated Show resolved Hide resolved

src/core/server/saved_objects/migrationsv2/integration_tests/actions.test.ts Outdated Show resolved Hide resolved

mshustov and others added 2 commits April 12, 2021 18:34

Update src/core/server/saved_objects/migrationsv2/integration_tests/a…

f45bb89

…ctions.test.ts Co-authored-by: Rudolf Meijering <[email protected]>

Update src/core/server/saved_objects/migrationsv2/integration_tests/a…

cf302c7

…ctions.test.ts Co-authored-by: Rudolf Meijering <[email protected]>

mshustov added v7.12.1 auto-backport Deprecated - use backport:version if exact versions are needed labels Apr 12, 2021

mshustov merged commit c2b1769 into elastic:master Apr 12, 2021

mistic added a commit that referenced this pull request Apr 12, 2021

skip flaky suite (#96788)

fb24006

mshustov deleted the migration-wait-for-yellow-cluster branch April 12, 2021 20:12

mshustov mentioned this pull request Apr 12, 2021

[7.x] Migration v2 waits for yellow cluster (#96788) #96887

Merged

mshustov mentioned this pull request Apr 12, 2021

[7.12] Migration v2 waits for yellow cluster (#96788) #96888

Merged

mshustov mentioned this pull request Apr 25, 2021

Rewriting SO id during migration #97222

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration v2 waits for yellow cluster #96788

Migration v2 waits for yellow cluster #96788

mshustov commented Apr 12, 2021

elasticmachine commented Apr 12, 2021

mshustov Apr 12, 2021

rudolf Apr 12, 2021

mshustov Apr 12, 2021

rudolf Apr 12, 2021

rudolf left a comment

rudolf Apr 12, 2021

rudolf Apr 12, 2021

rudolf Apr 12, 2021

kibanamachine commented Apr 12, 2021

kibanamachine commented Apr 12, 2021

mistic commented Apr 12, 2021

Migration v2 waits for yellow cluster #96788

Migration v2 waits for yellow cluster #96788

Conversation

mshustov commented Apr 12, 2021

Summary

Checklist

elasticmachine commented Apr 12, 2021

mshustov Apr 12, 2021

Choose a reason for hiding this comment

rudolf Apr 12, 2021

Choose a reason for hiding this comment

mshustov Apr 12, 2021

Choose a reason for hiding this comment

rudolf Apr 12, 2021

Choose a reason for hiding this comment

rudolf left a comment

Choose a reason for hiding this comment

rudolf Apr 12, 2021

Choose a reason for hiding this comment

rudolf Apr 12, 2021

Choose a reason for hiding this comment

rudolf Apr 12, 2021

Choose a reason for hiding this comment

kibanamachine commented Apr 12, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Apr 12, 2021

💔 Backport failed

mistic commented Apr 12, 2021