[#192624] `multiple_es_nodes` #193899

afharo · 2024-09-24T16:38:48Z

Summary

Related #192624.

ES complained because the archive was created with 8.0.0, but ES 9.0.0 requires the datastore to be upgraded to 8.16.0.
Then, it complained because the index from the archives was created on 7.13.0, and required reindexing.

The following steps have been followed to update the archives:

Start both nodes of ES 8.16 with the affected data-archive on separate terminals:
1. Node 01: yarn es snapshot --version=8.16.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01
2. Node 02: yarn es snapshot --version=8.16.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02

After ES is ready (without starting Kibana), reindex the index .kibana_7.13.0_001

Retrieve the settings from the original index via curl -L 'http://localhost:9200/.kibana_7.13.0_001' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''
Create the target index with those settings:

curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
"mappings": {
    "properties": {
        "bar": {
            "properties": {
                "status": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                }
            }
        },
        "coreMigrationVersion": {
            "type": "keyword"
        },
        "foo": {
            "properties": {
                "status": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                }
            }
        },
        "migrationVersion": {
            "dynamic": "true",
            "properties": {
                "bar": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "foo": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                }
            }
        },
        "namespace": {
            "type": "keyword"
        },
        "namespaces": {
            "type": "keyword"
        },
        "originId": {
            "type": "keyword"
        },
        "references": {
            "type": "nested",
            "properties": {
                "id": {
                    "type": "keyword"
                },
                "name": {
                    "type": "keyword"
                },
                "type": {
                    "type": "keyword"
                }
            }
        },
        "type": {
            "type": "keyword"
        },
        "updated_at": {
            "type": "date"
        }
    }
},
"settings": {
    "index": {
        "hidden": "true",
        "number_of_shards": "1",
        "number_of_replicas": "0"
    }
}
}'

Reindex the content: curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_001" }, "dest": { "index": ".kibana_7.13.0_002" } }'
Remove the old index and recreate the aliases

curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
  "actions": [
    { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana_7.13.0_001" } },
    { "remove_index": {"index": ".kibana_7.13.0_001" } },
    { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana_7.13.0" } },
    { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana" } }
  ]
}'

Stop both ES nodes.

Compress both archives

cd .es/node01/8.16.0
rm -rf data/nodes # we need to remove this dir or it fails to start again
zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data  -x "*/\.*"
cd ../../../
cd .es/node02/8.16.0
rm -rf data/nodes # we need to remove this dir or it fails to start again
zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data  -x "*/\.*"
cd ../../../

Run the tests to confirm that the issue is fixed: yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2024-09-24T16:38:51Z

Pinging @elastic/kibana-core (Team:Core)

afharo · 2024-09-24T16:39:10Z

packages/kbn-test/src/es/test_es_cluster.ts

@@ -191,7 +191,10 @@ export function createTestEsCluster<
    `transport.port=${transportPort ?? esTestConfig.getTransportPort()}`,
    // For multi-node clusters, we make all nodes master-eligible by default.
    ...(nodes.length > 1
-      ? ['discovery.type=zen', `cluster.initial_master_nodes=${nodes.map((n) => n.name).join(',')}`]
+      ? [
+          'discovery.type=multi-node',


zen is no longer supported

jbudz · 2024-09-24T18:46:42Z

src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts

@@ -96,7 +96,7 @@ function createRoot({ logFileName, hosts }: RootConfig) {
 }

 // Failing 9.0 version update: https://github.com/elastic/kibana/issues/192624


comment can be removed

Addressed in a225053 (#193899)

jbudz · 2024-09-24T18:48:04Z

src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip

Do we know why the file size increased? mostly curious

#193741 (review)

I'll use ditto, sorry!

Addressed in 6a699b5 (#193899)

Reverted in 97b4937 (#193899). We noticed that the documents were gone 😨

I re-ran it locally and it included them now 🤷
However, somehow, the .zip generated via ditto were even larger than the ones generated with zip, so I stuck to the existing files 😇

This reverts commit 6a699b5.

gsoldevila

LGTM, thanks!

kibana-ci · 2024-09-25T11:30:31Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 97b4937

Failed CI Steps

Jest Tests #11

Test Failures

[job] [logs] Jest Tests #11 / Configuration button renders the tooltip correctly when hovering the button

Metrics [docs]

✅ unchanged

History

💚 Build #236735 succeeded 1c09fe2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @afharo

jbudz

packages/kbn-test/src/es/test_es_cluster.ts

(cherry picked from commit 66f2027)

kibanamachine · 2024-09-25T13:22:11Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [[#192624] `multiple_es_nodes` (#193899)](#193899)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Alejandro Fernández Haro <[email protected]>

## Summary Addresses #167676 Addresses #158318 Addresses #163254 Addresses #163255 #### Fix for `multiple_es_nodes.test.ts` Inspired on #193899 1. Start both nodes of ES 8.17.0 with the affected data-archive on separate terminals: 1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 2. After ES is ready (without starting Kibana), reindex the index `.kibana_7.13.0_002` 1. Retrieve the settings from the original index via `curl -L 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''` 2. Create the target index with those settings: ```shell curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "mappings": { "properties": { "bar": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "coreMigrationVersion": { "type": "keyword" }, "foo": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "migrationVersion": { "dynamic": "true", "properties": { "bar": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "foo": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "settings": { "index": { "hidden": "true", "number_of_shards": "1", "number_of_replicas": "0" } } }' ``` 3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003" } }'` 4. Remove the old index and recreate the aliases ```shell curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "actions": [ { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001" } }, { "remove_index": {"index": ".kibana_7.13.0_002" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } } ] }' ``` 3. Stop both ES nodes. 4. Compress both archives ```shell cd .es/node01/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*" cd ../../../ cd .es/node02/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*" cd ../../../ ``` 5. Run the tests to confirm that the issue is fixed: `yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts` #### Fix for `incompatible_cluster_routing_allocation.test.ts` Inspired on #193741 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts ``` #### Fix for `read_batch_size.test.ts` Inspired on #193899 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts ```

…6641) ## Summary Addresses elastic#167676 Addresses elastic#158318 Addresses elastic#163254 Addresses elastic#163255 #### Fix for `multiple_es_nodes.test.ts` Inspired on elastic#193899 1. Start both nodes of ES 8.17.0 with the affected data-archive on separate terminals: 1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 2. After ES is ready (without starting Kibana), reindex the index `.kibana_7.13.0_002` 1. Retrieve the settings from the original index via `curl -L 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''` 2. Create the target index with those settings: ```shell curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "mappings": { "properties": { "bar": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "coreMigrationVersion": { "type": "keyword" }, "foo": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "migrationVersion": { "dynamic": "true", "properties": { "bar": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "foo": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "settings": { "index": { "hidden": "true", "number_of_shards": "1", "number_of_replicas": "0" } } }' ``` 3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003" } }'` 4. Remove the old index and recreate the aliases ```shell curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "actions": [ { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001" } }, { "remove_index": {"index": ".kibana_7.13.0_002" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } } ] }' ``` 3. Stop both ES nodes. 4. Compress both archives ```shell cd .es/node01/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*" cd ../../../ cd .es/node02/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*" cd ../../../ ``` 5. Run the tests to confirm that the issue is fixed: `yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts` #### Fix for `incompatible_cluster_routing_allocation.test.ts` Inspired on elastic#193741 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts ``` #### Fix for `read_batch_size.test.ts` Inspired on elastic#193899 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts ``` (cherry picked from commit 3d254c2) # Conflicts: # src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts

[elastic#192624] multiple_es_nodes

1c09fe2

afharo self-assigned this Sep 24, 2024

afharo requested review from a team as code owners September 24, 2024 16:38

afharo commented Sep 24, 2024

View reviewed changes

jbudz reviewed Sep 24, 2024

View reviewed changes

afharo added 3 commits September 25, 2024 11:07

Remove comment

a225053

Recompress with ditto

6a699b5

Revert "Recompress with ditto"

97b4937

This reverts commit 6a699b5.

gsoldevila approved these changes Sep 25, 2024

View reviewed changes

afharo enabled auto-merge (squash) September 25, 2024 11:33

jbudz approved these changes Sep 25, 2024

View reviewed changes

afharo merged commit 66f2027 into elastic:main Sep 25, 2024
24 of 25 checks passed

afharo deleted the 192624-multiple_es_nodes branch September 25, 2024 13:17

kibanamachine added the v9.0.0 label Sep 25, 2024

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 25, 2024

[elastic#192624] multiple_es_nodes (elastic#193899)

5755ea7

(cherry picked from commit 66f2027)

kibanamachine mentioned this pull request Sep 25, 2024

[8.x] [#192624] `multiple_es_nodes` (#193899) #194001

Merged

kibanamachine added the v8.16.0 label Sep 25, 2024

gsoldevila mentioned this pull request Oct 17, 2024

Update archives to 8.17.0 binary data format, unskip tests #196641

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#192624] `multiple_es_nodes` #193899

[#192624] `multiple_es_nodes` #193899

afharo commented Sep 24, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Sep 24, 2024

afharo Sep 24, 2024

jbudz Sep 24, 2024

afharo Sep 25, 2024

jbudz Sep 24, 2024

gsoldevila Sep 25, 2024

afharo Sep 25, 2024

afharo Sep 25, 2024

afharo Sep 25, 2024

afharo Sep 25, 2024

gsoldevila left a comment

kibana-ci commented Sep 25, 2024

jbudz left a comment

kibanamachine commented Sep 25, 2024

		@@ -96,7 +96,7 @@ function createRoot({ logFileName, hosts }: RootConfig) {
		}

		// Failing 9.0 version update: https://github.com/elastic/kibana/issues/192624

[#192624] multiple_es_nodes #193899

[#192624] multiple_es_nodes #193899

Conversation

afharo commented Sep 24, 2024 • edited by kibanamachine Loading

Summary

For maintainers

elasticmachine commented Sep 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsoldevila left a comment

Choose a reason for hiding this comment

kibana-ci commented Sep 25, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

jbudz left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 25, 2024

💚 All backports created successfully

Questions ?

[#192624] `multiple_es_nodes` #193899

[#192624] `multiple_es_nodes` #193899

afharo commented Sep 24, 2024 •

edited by kibanamachine

Loading