Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when migrating to a new version and the mappings' changes are still compatible the migrator skips reindexing #152448

Closed
kibanamachine opened this issue Mar 1, 2023 · 7 comments
Closed

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when migrating to a new version and the mappings' changes are still compatible the migrator skips reindexing #152448

kibanamachine opened this issue Mar 1, 2023 · 7 comments
Labels
Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Mar 1, 2023 •
edited
Loading

A test failed on a tracked branch

Error: expect(received).toMatch(expected)

Expected substring: "CHECK_VERSION_INDEX_READY_ACTIONS -> DONE."
Received string:    "{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.240-05:00\",\"message\":\"[.kibana_migrator_tests] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 6ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.244-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 4ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.251-05:00\",\"message\":\"[.kibana_migrator_tests] Starting to process 8 documents.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.251-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_TRANSFORM. took: 7ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.252-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_TRANSFORM -> TRANSFORMED_DOCUMENTS_BULK_INDEX. took: 1ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.264-05:00\",\"message\":\"[.kibana_migrator_tests] TRANSFORMED_DOCUMENTS_BULK_INDEX -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 12ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.271-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 7ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.274-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> OUTDATED_DOCUMENTS_REFRESH. took: 3ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T06:43:00.295-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_REFRESH -> CHECK_TARGET_MAPPINGS. took: 21ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2623}}
"
    at Object.toMatch (/var/lib/buildkite-agent/builds/kb-n2-4-spot-1d23a91fbe771531/elastic/kibana-on-merge/kibana/src/core/server/integration_tests/saved_objects/migrations/group3/skip_reindex.test.ts:125:18)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Mar 1, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Mar 1, 2023
@kibanamachine kibanamachine added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Mar 1, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Mar 1, 2023
@rudolf rudolf added the Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects label Mar 1, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

jbudz added a commit that referenced this issue Mar 2, 2023
@jbudz
skip flaky suite (#152448)
95dfb93
@jbudz
Copy link
Member

jbudz commented Mar 2, 2023

Skipped

main: 95dfb93

@gsoldevila gsoldevila mentioned this issue Mar 7, 2023
Merged
gsoldevila added a commit that referenced this issue Mar 7, 2023
@gsoldevila @kibanamachine
[Kibana migrator test kit] Flush logger after running migrations (#15…
163dae7 
…2818)

This PR addresses flakiness of:

- #152472
- #152448

These tests fail because we have a race condition. We are checking
certain conditions in the log files, and sometimes we do it before the
logs are actually written.

With the Kibana migrator test kit, we can actually flush + wait for all
the logging appenders. This way, we are sure that the logs are complete
when we inspect them in the tests.

---------

Co-authored-by: kibanamachine <[email protected]>
gsoldevila added a commit to gsoldevila/kibana that referenced this issue Mar 8, 2023
@gsoldevila
[Kibana migrator test kit] Flush logger after running migrations (ela…
bd17a5b 
…stic#152818)

This PR addresses flakiness of:

- elastic#152472
- elastic#152448

These tests fail because we have a race condition. We are checking
certain conditions in the log files, and sometimes we do it before the
logs are actually written.

With the Kibana migrator test kit, we can actually flush + wait for all
the logging appenders. This way, we are sure that the logs are complete
when we inspect them in the tests.

---------

Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit 163dae7)
gsoldevila referenced this issue Mar 8, 2023
@gsoldevila
[8.7] [Kibana migrator test kit] Flush logger after running migrations (
e70452f 
#152818) (#152890)

# Backport

This will backport the following commits from `main` to `8.7`:
- [[Kibana migrator test kit] Flush logger after running migrations
(#152818)](#152818)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-03-07T17:57:17Z","message":"[Kibana
migrator test kit] Flush logger after running migrations
(#152818)\n\nThis PR addresses flakiness of:\r\n\r\n-
https://github.com/elastic/kibana/issues/152472\r\n-
https://github.com/elastic/kibana/issues/152448\r\n\r\nThese tests fail
because we have a race condition. We are checking\r\ncertain conditions
in the log files, and sometimes we do it before the\r\nlogs are actually
written.\r\n\r\nWith the Kibana migrator test kit, we can actually flush
+ wait for all\r\nthe logging appenders. This way, we are sure that the
logs are complete\r\nwhen we inspect them in the
tests.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"163dae7154c11d92ff148bc6ac45cbea54680769","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","Team:Core","release_note:skip","test-failure-flaky","Feature:Migrations","backport:prev-minor","v8.7.0","v8.8.0"],"number":152818,"url":"https://github.com/elastic/kibana/pull/152818","mergeCommit":{"message":"[Kibana
migrator test kit] Flush logger after running migrations
(#152818)\n\nThis PR addresses flakiness of:\r\n\r\n-
https://github.com/elastic/kibana/issues/152472\r\n-
https://github.com/elastic/kibana/issues/152448\r\n\r\nThese tests fail
because we have a race condition. We are checking\r\ncertain conditions
in the log files, and sometimes we do it before the\r\nlogs are actually
written.\r\n\r\nWith the Kibana migrator test kit, we can actually flush
+ wait for all\r\nthe logging appenders. This way, we are sure that the
logs are complete\r\nwhen we inspect them in the
tests.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"163dae7154c11d92ff148bc6ac45cbea54680769"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/152818","number":152818,"mergeCommit":{"message":"[Kibana
migrator test kit] Flush logger after running migrations
(#152818)\n\nThis PR addresses flakiness of:\r\n\r\n-
https://github.com/elastic/kibana/issues/152472\r\n-
https://github.com/elastic/kibana/issues/152448\r\n\r\nThese tests fail
because we have a race condition. We are checking\r\ncertain conditions
in the log files, and sometimes we do it before the\r\nlogs are actually
written.\r\n\r\nWith the Kibana migrator test kit, we can actually flush
+ wait for all\r\nthe logging appenders. This way, we are sure that the
logs are complete\r\nwhen we inspect them in the
tests.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"163dae7154c11d92ff148bc6ac45cbea54680769"}}]}]
BACKPORT-->
bmorelli25 pushed a commit to bmorelli25/kibana that referenced this issue Mar 10, 2023
@jbudz @bmorelli25
skip flaky suite (elastic#152448)
3a9a9c9
bmorelli25 pushed a commit to bmorelli25/kibana that referenced this issue Mar 10, 2023
@gsoldevila @kibanamachine @bmorelli25
[Kibana migrator test kit] Flush logger after running migrations (ela…
17a4ede 
…stic#152818)

This PR addresses flakiness of:

- elastic#152472
- elastic#152448

These tests fail because we have a race condition. We are checking
certain conditions in the log files, and sometimes we do it before the
logs are actually written.

With the Kibana migrator test kit, we can actually flush + wait for all
the logging appenders. This way, we are sure that the logs are complete
when we inspect them in the tests.

---------

Co-authored-by: kibanamachine <[email protected]>
@kibanamachine kibanamachine reopened this Mar 20, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@gsoldevila gsoldevila mentioned this issue Mar 22, 2023
Merged
@mistic mistic closed this as completed Mar 22, 2023
mistic pushed a commit that referenced this issue Mar 22, 2023
@gsoldevila
Flush logger after running migrations, fix remaining test (#153450)
3f5961e 
Fixes flakiness on:
* #152448
* #152472

And also related PR builds failures, e.g.
#153332
([failure](https://s3.amazonaws.com/buildkiteartifacts.com/e0f3970e-3a75-4621-919f-e6c773e2bb12/0fda5127-f57f-42fb-8e5a-146b3d535916/018705a0-a7cc-422a-89a3-026a2637b9e4/018705a4-3025-4e3b-884c-ee805b529428/target/test_failures/018705a4-3025-4e3b-884c-ee805b529428_d47f90a974496415048dfbef2bc5134d.html?response-content-type=text%2Fhtml&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LYVRIJ2HP%2F20230322%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230322T131425Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEN3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDA9S4KjEvUes%2BBf46MNFrR91uVacUL4Fb6hFz2c%2BDUNAIhALC%2FK1HFIS%2FaFsiZeyn99na1OnoHESROe%2FPbjcYK%2F2RmKvoDCKX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQABoMMDMyMzc5NzA1MzAzIgyIyDrn3p%2Bf98NXvT4qzgM4eyTaC4EwuV5t7xCmqtUrfCy2lm9ftD3NmJplYEdeuQcOaKtd5Acz4hTEuReaDZYde1eoa9WgR%2BabHy5N3cWO1ndI%2FqfanO5CeUspza7kfaGg4jcDh1e68QAXRy2ciuK2hWWWw0OF8FErTYG3q5dmjOaHaLeUipG%2FdeWF4iIs4BLPVvJA3ZTNGTFPxwc6%2BtD1eJSobey%2Bi45tcTsrCkXm9UtuOSBtW1vYpFL2zkxLoDWV7m%2F%2F5MSNU53L4%2BWGp8bM4aFCo1fXa9IDYv5MQTn7z1CBi2qUn%2FJzAN0LlBwLaAYIDd7TdLN5c3FPzyoTGoknhQQh%2BB6G1qVmZtXwfSYHGYL7rOjv%2FmPHYHpwniyTKitGBHKGAvToHr%2B37wpFpBhpGzZoa8q4e6esRN%2F94wItt%2BoYTh%2FraQTxtqsG8PEZnUJ%2F2H%2FyeNck4rbePnirxVBTt1FAwABpM5TfJEFHd9EYESsSsJ6WIpv43Gc1XbOeiC6MUf502NfAlxhqayhJ1Rux6L9y%2FCI5wbVC9VwI9LrbsavgTt21re6TZgazqt6x9jTZYAqfAZXhIcoUXOreoEBsfqhSTf6RlDY%2FzJ31rDz0dAhXl0QiEZnp%2By%2F6S7cwyeHroAY6pAFByPZ8bWAwDobR8BwKUCZShocd7SssJQqhMyo5E%2FK5aYYPXo3uCijbUcKvsHnyqllCFpA1t5LtyPjoyxSRewnrEueEop%2FAAukBF7%2FrIVZEdwhgWzne5QqK2bV9SMP92RYpDQ8Pff9NrRsgk9qVTKfMdE%2BTKSDt0zBl5gguqn7DLKsJdjBfs4nnXrJlxAKh%2F%2BXpSvKSX2vNh%2F7KZaOEB5R1uuZCFw%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=0a5787d518338968b49ac9bc085ad1b7d019b47be1ad2cb860ecc8b92e11e4a0))


[#152818](#152818) was a first
attempt at fixing this, but I forgot to update one of the tests.
@kibanamachine kibanamachine reopened this May 23, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@dokmic
Copy link
Contributor

dokmic commented Jul 28, 2023

This was caused by #157802, which is unrelated to the failed test. The issue can be safely closed.

@dokmic dokmic closed this as completed Jul 28, 2023
@dokmic
Copy link
Contributor

dokmic commented Jul 28, 2023

This was reopened by #157802, which is unrelated to the failed test. The issue can be safely closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@rudolf @jbudz @mistic @dokmic @elasticmachine @gsoldevila @kibanamachine