Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Migrations] [illegal_argument_exception]: mapper [property] cannot be changed from type [x] to [y] #163289

Closed
gsoldevila opened this issue Aug 7, 2023 · 4 comments
Closed

[Migrations] [illegal_argument_exception]: mapper [property] cannot be changed from type [x] to [y] #163289

gsoldevila opened this issue Aug 7, 2023 · 4 comments
Labels
Feature:Migrations Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@gsoldevila
Copy link
Contributor

gsoldevila commented Aug 7, 2023 •
edited
Loading

We have spotted a couple of occurrences of that error during saved object migrations:

The first one, on a flaky test: #127545

[00:02:13] proc [kibana] [2022-03-11T20:45:43.189+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_task_manager_8.2.0_001/_mapping?timeout=60s error: [illegal_argument_exception]: mapper [task.status] cannot be changed from type [text] to [keyword],
[00:02:13] proc [kibana] [2022-03-11T20:45:43.190+00:00][ERROR][http] Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_task_manager_8.2.0_001/_mapping?timeout=60s error: [illegal_argument_exception]: mapper [task.status] cannot be changed from type [text] to [keyword],

The second one, on an SDH: https://github.com/elastic/sdh-kibana/issues/3976

Unable to complete saved object migrations for the [.kibana] index. Error: [{"type":"illegal_argument_exception","reason":"mapper [canvas-workpad.pages.elements.position.left] cannot be changed from type [long] to [float]"}]

For the SDH occurrence, the issue seemed to occur during the REINDEX_SOURCE_TO_TEMP_INDEX_BULK step, when the migrators are indexing documents into the temporary indices.
@botelastic botelastic bot added the needs-team Issues missing a team label label Aug 7, 2023
@gsoldevila gsoldevila added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Aug 7, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Aug 7, 2023
This was referenced Aug 7, 2023
Closed
Merged
@pgayvallet
Copy link
Contributor

pgayvallet commented Aug 23, 2023 •
edited
Loading

For the SDH occurrence, the issue seemed to occur during the REINDEX_SOURCE_TO_TEMP_INDEX_BULK step, when the migrators are indexing documents into the temporary indices.

Even if the temp index mapping is dynamic: false, it seems that internally, ES still needs to identify the type of the field.

From my understanding, this error between long and float is caused by the fact that the first document using this field has a "shorter" value, creating the field as a float, but then another document has a value that can't be contained in a float, so using a long value in java, causing the mapper exception, as the field already exists as a float type.

We should probably try to confirm this hypothesis, but if confirmed, I'm not 100% sure what we could do about it, aside from actually using the mappings from the types (in addition to "dynamic: false") for the temp index?

Note that this wouldn't explain the FTR error between text and keyword, so maybe my guess is just wrong...

gsoldevila added a commit that referenced this issue Sep 1, 2023
@gsoldevila
[esArchiver] Do not perform SO migration on cleanup (#163302)
f8303ec 
## Summary

Attempt at fixing #127545

The failing tests use `esArchiver` to cleanup Saved Object indices
between tests.
Strangely, the cleanup method calls the migration logic to update the SO
indices.

In this particular source of flakiness, a test fails during the
migration (problem is tracked by
#163289).

Performing a migration as part of the cleanup does not make too much
sense (at least in this test suite), so the goal of this PR is to
simplify the cleanup operation, getting rid of the call to
`migrateSavedObjectIndices`.

50 runs flaky test runner
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850
✅
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Sep 1, 2023
@gsoldevila
[esArchiver] Do not perform SO migration on cleanup (elastic#163302)
be5b715 
## Summary

Attempt at fixing elastic#127545

The failing tests use `esArchiver` to cleanup Saved Object indices
between tests.
Strangely, the cleanup method calls the migration logic to update the SO
indices.

In this particular source of flakiness, a test fails during the
migration (problem is tracked by
elastic#163289).

Performing a migration as part of the cleanup does not make too much
sense (at least in this test suite), so the goal of this PR is to
simplify the cleanup operation, getting rid of the call to
`migrateSavedObjectIndices`.

50 runs flaky test runner
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850
✅

(cherry picked from commit f8303ec)
kibanamachine added a commit that referenced this issue Sep 1, 2023
@kibanamachine @gsoldevila
[8.10] [esArchiver] Do not perform SO migration on cleanup (#163302) (#…
8ea7cee 
…165451)

# Backport

This will backport the following commits from `main` to `8.10`:
- [[esArchiver] Do not perform SO migration on cleanup
(#163302)](#163302)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-09-01T10:41:20Z","message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com//issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n✅","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","Team:Core","Team:Operations","release_note:skip","test-failure-flaky","backport:all-open","v8.11.0"],"number":163302,"url":"https://github.com/elastic/kibana/pull/163302","mergeCommit":{"message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com//issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n✅","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/163302","number":163302,"mergeCommit":{"message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com//issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n✅","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9"}}]}]
BACKPORT-->

Co-authored-by: Gerard Soldevila <[email protected]>
@gsoldevila gsoldevila mentioned this issue Sep 4, 2023
Merged
gsoldevila added a commit to gsoldevila/kibana that referenced this issue Sep 4, 2023
@gsoldevila
[esArchiver] Do not perform SO migration on cleanup (elastic#163302)
174e3c5 
## Summary

Attempt at fixing elastic#127545

The failing tests use `esArchiver` to cleanup Saved Object indices
between tests.
Strangely, the cleanup method calls the migration logic to update the SO
indices.

In this particular source of flakiness, a test fails during the
migration (problem is tracked by
elastic#163289).

Performing a migration as part of the cleanup does not make too much
sense (at least in this test suite), so the goal of this PR is to
simplify the cleanup operation, getting rid of the call to
`migrateSavedObjectIndices`.

50 runs flaky test runner
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850
✅

(cherry picked from commit f8303ec)

# Conflicts:
#	packages/kbn-es-archiver/src/actions/empty_kibana_index.ts
#	test/examples/search/warnings.ts
gsoldevila added a commit that referenced this issue Sep 25, 2023
@gsoldevila
[7.17] [esArchiver] Do not perform SO migration on cleanup (#163302) (#…
757e867 
…165566)

# Backport

This will backport the following commits from `main` to `7.17`:
- [[esArchiver] Do not perform SO migration on cleanup
(#163302)](#163302)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-09-01T10:41:20Z","message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com//issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n✅","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","Team:Core","Team:Operations","release_note:skip","test-failure-flaky","backport:all-open","v8.10.0","v8.11.0"],"number":163302,"url":"https://github.com/elastic/kibana/pull/163302","mergeCommit":{"message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com//issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n✅","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/165451","number":165451,"state":"MERGED","mergeCommit":{"sha":"8ea7cee5003c3bfb550da9ab1fab8e9bfe820250","message":"[8.10]
[esArchiver] Do not perform SO migration on cleanup (#163302)
(#165451)\n\n# Backport\n\nThis will backport the following commits from
`main` to `8.10`:\n- [[esArchiver] Do not perform SO migration on
cleanup\n(#163302)](https://github.com/elastic/kibana/pull/163302)\n\n<!---
Backport version: 8.9.7 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Gerard\nSoldevila\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2023-09-01T10:41:20Z\",\"message\":\"[esArchiver]\nDo
not perform SO migration on cleanup
(#163302)\\n\\n##\nSummary\\r\\n\\r\\nAttempt at
fixing\nhttps://github.com//issues/127545\\r\\n\\r\\nThe
failing tests\nuse `esArchiver` to cleanup Saved Object
indices\\r\\nbetween\ntests.\\r\\nStrangely, the cleanup method calls
the migration logic to\nupdate the SO\\r\\nindices.\\r\\n\\r\\nIn this
particular source of flakiness,\na test fails during the\\r\\nmigration
(problem is
tracked\nby\\r\\nhttps://github.com//issues/163289).\\r\\n\\r\\nPerforming\na
migration as part of the cleanup does not make too much\\r\\nsense
(at\nleast in this test suite), so the goal of this PR is
to\\r\\nsimplify the\ncleanup operation, getting rid of the
call\nto\\r\\n`migrateSavedObjectIndices`.\\r\\n\\r\\n50 runs flaky
test\nrunner\\r\\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\\r\\n✅\",\"sha\":\"f8303ec7cf57aa8787331ada397dfa853de969c9\",\"branchLabelMapping\":{\"^v8.11.0$\":\"main\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"bug\",\"Team:Core\",\"Team:Operations\",\"release_note:skip\",\"test-failure-flaky\",\"backport:all-open\",\"v8.11.0\"],\"number\":163302,\"url\":\"https://github.com/elastic/kibana/pull/163302\",\"mergeCommit\":{\"message\":\"[esArchiver]\nDo
not perform SO migration on cleanup
(#163302)\\n\\n##\nSummary\\r\\n\\r\\nAttempt at
fixing\nhttps://github.com//issues/127545\\r\\n\\r\\nThe
failing tests\nuse `esArchiver` to cleanup Saved Object
indices\\r\\nbetween\ntests.\\r\\nStrangely, the cleanup method calls
the migration logic to\nupdate the SO\\r\\nindices.\\r\\n\\r\\nIn this
particular source of flakiness,\na test fails during the\\r\\nmigration
(problem is
tracked\nby\\r\\nhttps://github.com//issues/163289).\\r\\n\\r\\nPerforming\na
migration as part of the cleanup does not make too much\\r\\nsense
(at\nleast in this test suite), so the goal of this PR is
to\\r\\nsimplify the\ncleanup operation, getting rid of the
call\nto\\r\\n`migrateSavedObjectIndices`.\\r\\n\\r\\n50 runs flaky
test\nrunner\\r\\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\\r\\n✅\",\"sha\":\"f8303ec7cf57aa8787331ada397dfa853de969c9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v8.11.0\",\"labelRegex\":\"^v8.11.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/163302\",\"number\":163302,\"mergeCommit\":{\"message\":\"[esArchiver]\nDo
not perform SO migration on cleanup
(#163302)\\n\\n##\nSummary\\r\\n\\r\\nAttempt at
fixing\nhttps://github.com//issues/127545\\r\\n\\r\\nThe
failing tests\nuse `esArchiver` to cleanup Saved Object
indices\\r\\nbetween\ntests.\\r\\nStrangely, the cleanup method calls
the migration logic to\nupdate the SO\\r\\nindices.\\r\\n\\r\\nIn this
particular source of flakiness,\na test fails during the\\r\\nmigration
(problem is
tracked\nby\\r\\nhttps://github.com//issues/163289).\\r\\n\\r\\nPerforming\na
migration as part of the cleanup does not make too much\\r\\nsense
(at\nleast in this test suite), so the goal of this PR is
to\\r\\nsimplify the\ncleanup operation, getting rid of the
call\nto\\r\\n`migrateSavedObjectIndices`.\\r\\n\\r\\n50 runs flaky
test\nrunner\\r\\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\\r\\n✅\",\"sha\":\"f8303ec7cf57aa8787331ada397dfa853de969c9\"}}]}]\nBACKPORT-->\n\nCo-authored-by:
Gerard Soldevila
<[email protected]>"}},{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/163302","number":163302,"mergeCommit":{"message":"[esArchiver]
Do not perform SO migration on cleanup (#163302)\n\n##
Summary\r\n\r\nAttempt at fixing
https://github.com/elastic/kibana/issues/127545\r\n\r\nThe failing tests
use `esArchiver` to cleanup Saved Object indices\r\nbetween
tests.\r\nStrangely, the cleanup method calls the migration logic to
update the SO\r\nindices.\r\n\r\nIn this particular source of flakiness,
a test fails during the\r\nmigration (problem is tracked
by\r\nhttps://github.com//issues/163289).\r\n\r\nPerforming
a migration as part of the cleanup does not make too much\r\nsense (at
least in this test suite), so the goal of this PR is to\r\nsimplify the
cleanup operation, getting rid of the call
to\r\n`migrateSavedObjectIndices`.\r\n\r\n50 runs flaky test
runner\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2850\r\n✅","sha":"f8303ec7cf57aa8787331ada397dfa853de969c9"}}]}]
BACKPORT-->
@pgayvallet
Copy link
Contributor

@gsoldevila did we do anything for this issue / shall we keep it open or close it?

@rudolf rudolf mentioned this issue Jul 23, 2024
Merged
9 tasks
@rudolf
Copy link
Contributor

rudolf commented Jul 23, 2024

I think this was another case of #158182 where one node is bulk indexing and another finishes and deletes the temp index. The node that continues to bulk index then causes ES to auto-create an index with the incorrect mappings.

@rudolf rudolf closed this as completed Jul 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature:Migrations Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rudolf @pgayvallet @elasticmachine @gsoldevila