-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding ability to auto-install ingest pipelines #95782
Conversation
Hi @eyalkoren, I've created a changelog YAML for you. |
Pinging @elastic/es-data-management (Team:Data Management) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small comment for the forbidden APIs check - It does look like the CI is failing on some templates not being ready, might be worth trying to reproduce locally?
...k/plugin/core/src/main/java/org/elasticsearch/xpack/core/template/IndexTemplateRegistry.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doh! Put the previous comment on the wrong file 😵💫
...gin/core/src/test/java/org/elasticsearch/xpack/core/template/IndexTemplateRegistryTests.java
Outdated
Show resolved
Hide resolved
@elastic/enterprise-search this PR adds the same capability you added through #95198, only to core. It does pretty much the exact same thing, but I also added a validation step to component templates, so that those are not allowed to be registered if they are dependent on an ingest pipeline that is not yet in place. I assume this doesn't break any assumption you are relying on, however it did break |
@eyalkoren I appreciate that you pick this change to be part of the core. I was about to prepare a PR for this now I have more bandwidth. I will review the issue later today but would like to highlight two recently merged PRs to get sure these fetaures are part of the new implementation:
|
@afoucret thanks for taking a look! My take on the two:
|
The only failing test now is @jbaiera @dakrone @felixbarny - your thoughts on this will be appreciated. |
I don't think #95780 will be required for the index registry work since the StackTemplateRegistry requires the items to be installed on the master node only, which is guaranteed to be the latest version in the cluster based on our upgrade guidance. As for the pipeline jam - I'm not sure I understand how this works today. Is the data stream created via an explicit API call or when a document is first ingested? When a document arrives for the data stream and it doesn't exist, the ingest service will resolve the template for the index the document would go into and uses the pipelines defined on the template, but if the pipelines only exist when the data stream exists then there will be no pipeline definition for the document and the ingest will fail, thus never creating the data stream. I'm assuming that the data stream is created via some other API call or else I'm missing something. @afoucret can you clarify this? Assuming everything does work just fine otherwise, then we could modify the |
Hmm, is that what we want though? Can this (applying pipeline only on master node) cause a scalability issue? Does this mean that the master is the only node with |
.../qa/rest/src/yamlRestTest/java/org/elasticsearch/xpack/entsearch/EnterpriseSearchRestIT.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for the iterations!
Adding the ability to auto-install ingest pipelines and refer to them from index templates through
index.default_pipeline
andindex.final_pipeline
.This would enable required capabilities like #95551 and #95522.