-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ingest: processors configuration can too easily be misconfigured resulting in non-deterministic processor execution order #36134
Comments
Date
ingest processor cannot access fields created by other processors in a chain
@turchanov - Thanks for reporting this. It appears that the root cause of your issue is the missing a couple The following works correctly:
I am going to change the description on this issue and call this a bug that we don't fail with the configuration you provided. It is way too easy to get this wrong, and I had to stare at this for a while before I saw the issue. The problem here is that the provided configuration results a List of 1 with 2 Hashmap entries, where the order the processors are executed is non-deterministic (just got luck it works with convert, but not date). The corrected configuration results in List of 2 each with 1 Hashmap entry where the order of the processors are executed deterministically. |
Pinging @elastic/es-core-features |
The fix here is to fail faster, and don't allow configuration that will result in non-deterministic processor execution. |
Oh my.... Indeed that was my misstake. |
Closing in favor #41837 (that one's a duplicate of this one, but I slightly prefer the write up there). |
Elasticsearch version (
bin/elasticsearch --version
):Plugins installed: []
None
JVM version (
java -version
):OS version (
uname -a
if on a Unix-like system):Description of the problem including expected versus actual behavior:
Date
ingest processor doesn't support chaining. That is it cannot access fields that were created by preceeding processors of processors chain.Convert
processors works ok in this situation.Steps to reproduce:
This fails with
java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [time] not present as part of path [time]
although using
Convert
processor works fineProvide logs (if relevant):
The text was updated successfully, but these errors were encountered: