Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - migration v2 migrates saved objects normally with multiple ES nodes #167676

Open
kibanamachine opened this issue Sep 29, 2023 · 34 comments
Assignees
Labels
blocker failed-es-promotion failed-test A test failure on a tracked branch, potentially flaky-test skipped-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v9.0.0

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Sep 29, 2023

A test failed on a tracked branch

Error: Command failed with exit code 1: ./bin/elasticsearch-keystore create
Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from [elasticsearch.yml]
	at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1249)
	at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:126)
	at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:52)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.Command.main(Command.java:50)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
Caused by: org.elasticsearch.xcontent.XContentParseException: [85:23] Duplicate field 'xpack.security.enabled'
 at [Source: (ByteArrayInputStream); line: 85, column: 23]
	at org.elasticsearch.xcontent.provider.json.JsonXContentParser.newXContentParseException(JsonXContentParser.java:52)
	at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:71)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:768)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:736)
	at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1245)
	... 9 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'xpack.security.enabled'
 at [Source: (ByteArrayInputStream); line: 85, column: 23]
	at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:243)
	at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:237)
	at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:482)
	at com.fasterxml.jackson.core.JsonParser.nextFieldName(JsonParser.java:1091)
	at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:69)
	... 12 more
    at makeError (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at configureKeystore (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/packages/kbn-es/src/install/install_archive.ts:100:3)
    at installArchive (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/packages/kbn-es/src/install/install_archive.ts:75:5)
    at installSnapshot (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/packages/kbn-es/src/install/install_snapshot.ts:64:10)
    at /var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/packages/kbn-es/src/cluster.ts:118:31
    at /var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/packages/kbn-tooling-log/src/tooling_log.ts:84:18
    at Cluster.installSnapshot (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/packages/kbn-es/src/cluster.ts:117:12)
    at TestCluster.start (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/packages/kbn-test/src/es/test_es_cluster.ts:242:24)
    at startES (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/packages/core/test-helpers/core-test-helpers-kbn-server/src/create_root.ts:268:7)
    at Object.<anonymous> (/var/lib/buildkite-agent/builds/kb-n2-4-spot-43c6d541a4061875/elastic/kibana-on-merge/kibana/src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts:148:16)

First failure: CI Build - main

@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Sep 29, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Sep 29, 2023
@kibanamachine kibanamachine added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Sep 29, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Sep 29, 2023
@kibanamachine kibanamachine reopened this Oct 16, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@mistic
Copy link
Member

mistic commented Oct 17, 2024

Skipped.

main: dd65ffa

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

gsoldevila added a commit that referenced this issue Oct 17, 2024
## Summary

Addresses #167676
Addresses #158318
Addresses #163254
Addresses #163255

#### Fix for `multiple_es_nodes.test.ts`
Inspired on #193899

1. Start both nodes of ES 8.17.0 with the affected data-archive on
separate terminals:
1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip
--base-path .es/node01`
2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip
--base-path .es/node02`

2. After ES is ready (without starting Kibana), reindex the index
`.kibana_7.13.0_002`
1. Retrieve the settings from the original index via `curl -L
'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''`
	2. Create the target index with those settings: 
	```shell
curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H
'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:
Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
      "mappings": {
        "properties": {
          "bar": {
            "properties": {
              "status": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "coreMigrationVersion": {
            "type": "keyword"
          },
          "foo": {
            "properties": {
              "status": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "migrationVersion": {
            "dynamic": "true",
            "properties": {
              "bar": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "foo": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "namespace": {
            "type": "keyword"
          },
          "namespaces": {
            "type": "keyword"
          },
          "originId": {
            "type": "keyword"
          },
          "references": {
            "type": "nested",
            "properties": {
              "id": {
                "type": "keyword"
              },
              "name": {
                "type": "keyword"
              },
              "type": {
                "type": "keyword"
              }
            }
          },
          "type": {
            "type": "keyword"
          },
          "updated_at": {
            "type": "date"
          }
        }
      },
      "settings": {
        "index": {
            "hidden": "true",
            "number_of_shards": "1",
            "number_of_replicas": "0"
        }
      }
    }'
         ```
3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H
'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:
Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": {
"index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003"
} }'`
    4. Remove the old index and recreate the aliases
    ```shell
curl -L 'http://localhost:9200/_aliases' -H 'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
  "actions": [
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001"
} },
    { "remove_index": {"index": ".kibana_7.13.0_002" } },
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } },
    { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } }
  ]
}'
	```
3. Stop both ES nodes.
4. Compress both archives
	```shell
	cd .es/node01/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start
again
zip -r
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip
data -x "*/\.*"
	cd ../../../
	cd .es/node02/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start
again
zip -r
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip
data -x "*/\.*"
	cd ../../../
	```
5. Run the tests to confirm that the issue is fixed: `yarn
test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`

#### Fix for `incompatible_cluster_routing_allocation.test.ts`
Inspired on #193741

```shell
# 1. Start ES 8.17.0 with the affected data-archive
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip
# ... after ES has completely started up, stop it.

# 2. Compress the archive
cd .es/8.17.0
zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data  -x "*/\.*"
cd ../../

# 3. Run the tests to confirm that the issue is fixed.
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts
```

#### Fix for `read_batch_size.test.ts`
Inspired on #193899

```shell
# 1. Start ES 8.17.0 with the affected data-archive
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip
# ... after ES has completely started up, stop it.

# 2. Compress the archive
cd .es/8.17.0
zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data  -x "*/\.*"
cd ../../

# 3. Run the tests to confirm that the issue is fixed.
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts
```
@gsoldevila
Copy link
Contributor

Addressed by #196641

gsoldevila added a commit to gsoldevila/kibana that referenced this issue Oct 17, 2024
…6641)

## Summary

Addresses elastic#167676
Addresses elastic#158318
Addresses elastic#163254
Addresses elastic#163255

#### Fix for `multiple_es_nodes.test.ts`
Inspired on elastic#193899

1. Start both nodes of ES 8.17.0 with the affected data-archive on
separate terminals:
1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip
--base-path .es/node01`
2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip
--base-path .es/node02`

2. After ES is ready (without starting Kibana), reindex the index
`.kibana_7.13.0_002`
1. Retrieve the settings from the original index via `curl -L
'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''`
	2. Create the target index with those settings:
	```shell
curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H
'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:
Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
      "mappings": {
        "properties": {
          "bar": {
            "properties": {
              "status": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "coreMigrationVersion": {
            "type": "keyword"
          },
          "foo": {
            "properties": {
              "status": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "migrationVersion": {
            "dynamic": "true",
            "properties": {
              "bar": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "foo": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "namespace": {
            "type": "keyword"
          },
          "namespaces": {
            "type": "keyword"
          },
          "originId": {
            "type": "keyword"
          },
          "references": {
            "type": "nested",
            "properties": {
              "id": {
                "type": "keyword"
              },
              "name": {
                "type": "keyword"
              },
              "type": {
                "type": "keyword"
              }
            }
          },
          "type": {
            "type": "keyword"
          },
          "updated_at": {
            "type": "date"
          }
        }
      },
      "settings": {
        "index": {
            "hidden": "true",
            "number_of_shards": "1",
            "number_of_replicas": "0"
        }
      }
    }'
         ```
3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H
'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:
Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": {
"index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003"
} }'`
    4. Remove the old index and recreate the aliases
    ```shell
curl -L 'http://localhost:9200/_aliases' -H 'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
  "actions": [
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001"
} },
    { "remove_index": {"index": ".kibana_7.13.0_002" } },
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } },
    { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } }
  ]
}'
	```
3. Stop both ES nodes.
4. Compress both archives
	```shell
	cd .es/node01/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start
again
zip -r
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip
data -x "*/\.*"
	cd ../../../
	cd .es/node02/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start
again
zip -r
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip
data -x "*/\.*"
	cd ../../../
	```
5. Run the tests to confirm that the issue is fixed: `yarn
test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`

#### Fix for `incompatible_cluster_routing_allocation.test.ts`
Inspired on elastic#193741

```shell
# 1. Start ES 8.17.0 with the affected data-archive
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip
# ... after ES has completely started up, stop it.

# 2. Compress the archive
cd .es/8.17.0
zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data  -x "*/\.*"
cd ../../

# 3. Run the tests to confirm that the issue is fixed.
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts
```

#### Fix for `read_batch_size.test.ts`
Inspired on elastic#193899

```shell
# 1. Start ES 8.17.0 with the affected data-archive
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip
# ... after ES has completely started up, stop it.

# 2. Compress the archive
cd .es/8.17.0
zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data  -x "*/\.*"
cd ../../

# 3. Run the tests to confirm that the issue is fixed.
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts
```

(cherry picked from commit 3d254c2)

# Conflicts:
#	src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts
@kibanamachine kibanamachine reopened this Nov 22, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-elasticsearch-snapshot-verify - main

@jbudz
Copy link
Member

jbudz commented Nov 25, 2024

Skipped

main: d44f70f

@kibanamachine
Copy link
Contributor Author

@kibanamachine
Copy link
Contributor Author

@kibanamachine
Copy link
Contributor Author

@kibanamachine
Copy link
Contributor Author

@kibanamachine
Copy link
Contributor Author

@kibanamachine
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker failed-es-promotion failed-test A test failure on a tracked branch, potentially flaky-test skipped-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v9.0.0
Projects
None yet
Development

No branches or pull requests

7 participants