Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Index template resolution does not wait for the Cluster State to be recovered #49499

Closed
polyfractal opened this issue Nov 22, 2019 · 2 comments · Fixed by #99797
Closed

Index template resolution does not wait for the Cluster State to be recovered #49499

polyfractal opened this issue Nov 22, 2019 · 2 comments · Fixed by #99797
Assignees
@ywangd
Labels
>bug :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@polyfractal
Copy link
Contributor

If a bulk request hits a node that has not recovered the CS yet, index metadata will not be available. The actual bulk request will wait/retry until the CS is recovered (due to retry mechanism in BulkOperation), but the resolution of index templates has no retry mechanism. This means default-pipeline processors will be omitted and this can result in incorrect documents being indexed. It may affect other aspects that deal with index templates as well.

There was an attempt to fix this (#46085), but the PR stalled. I closed that PR so that the issue is "unclaimed" in case someone else wants to fix the issue. I might revisit, but don't have time right now. :(
@polyfractal polyfractal added >bug :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. labels Nov 22, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/CRUD)

@rjernst rjernst added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label May 4, 2020
@Leaf-Lin Leaf-Lin added the good first issue low hanging fruit label Jun 2, 2022
@Leaf-Lin Leaf-Lin removed the good first issue low hanging fruit label Jun 15, 2022
@ywangd ywangd self-assigned this Sep 21, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

ywangd added a commit to ywangd/elasticsearch that referenced this issue Sep 22, 2023
@ywangd
Wait for cluster to recover before resolving index template
3302980 
This PR makes bulk index action to wait for cluster to recover before
resolving index templates so that ingest pipelines are correctly
processed when the cluster is recovering.

Resolves: elastic#49499
Supercedes: elastic#46085
@ywangd ywangd mentioned this issue Sep 22, 2023
Merged
ywangd added a commit that referenced this issue Sep 22, 2023
@ywangd
Wait for cluster to recover before resolving index template (#99797)
dd1eb20 
This PR makes bulk index action wait for cluster to recover before
resolving index templates so that ingest pipelines are correctly
processed when the cluster is recovering.

Resolves: #49499
Supercedes: #46085
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ywangd ywangd

Labels
>bug :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Wait for cluster to recover before resolving index template ywangd/elasticsearch
6 participants
@rjernst @polyfractal @ywangd @elasticmachine @Leaf-Lin @elasticsearchmachine