Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ignore_missing parameter has no effect in rename ingest processor #9725

Merged
merged 3 commits into from
Sep 14, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Removed

### Fixed
- Fix ignore_missing parameter has no effect when using template snippet in rename ingest processor ([#9725](https://github.com/opensearch-project/OpenSearch/pull/9725))

### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.11...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.11...2.x
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@

package org.opensearch.ingest.common;

import org.opensearch.core.common.Strings;
import org.opensearch.ingest.AbstractProcessor;
import org.opensearch.ingest.ConfigurationUtils;
import org.opensearch.ingest.IngestDocument;
Expand Down Expand Up @@ -80,9 +81,12 @@ boolean isIgnoreMissing() {
@Override
public IngestDocument execute(IngestDocument document) {
String path = document.renderTemplate(field);
if (document.hasField(path, true) == false) {
boolean fieldPathIsNullOrEmpty = Strings.isEmpty(path);
gaobinlong marked this conversation as resolved.
Show resolved Hide resolved
if (fieldPathIsNullOrEmpty || document.hasField(path, true) == false) {
if (ignoreMissing) {
return document;
} else if (fieldPathIsNullOrEmpty) {
throw new IllegalArgumentException("field path cannot be null nor empty");
} else {
throw new IllegalArgumentException("field [" + path + "] doesn't exist");
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,15 @@ public void testRenameNonExistingField() throws Exception {
} catch (IllegalArgumentException e) {
assertThat(e.getMessage(), equalTo("field [" + fieldName + "] doesn't exist"));
}

// when using template snippet, the resolved field path maybe empty
processor = createRenameProcessor("", RandomDocumentPicks.randomFieldName(random()), false);
reta marked this conversation as resolved.
Show resolved Hide resolved
try {
processor.execute(ingestDocument);
fail("processor execute should have failed");
} catch (IllegalArgumentException e) {
assertThat(e.getMessage(), equalTo("field path cannot be null nor empty"));
}
}

public void testRenameNonExistingFieldWithIgnoreMissing() throws Exception {
Expand All @@ -121,6 +130,11 @@ public void testRenameNonExistingFieldWithIgnoreMissing() throws Exception {
Processor processor = createRenameProcessor(fieldName, RandomDocumentPicks.randomFieldName(random()), true);
processor.execute(ingestDocument);
assertIngestDocument(originalIngestDocument, ingestDocument);

// when using template snippet, the resolved field path maybe empty
processor = createRenameProcessor("", RandomDocumentPicks.randomFieldName(random()), true);
processor.execute(ingestDocument);
assertIngestDocument(originalIngestDocument, ingestDocument);
}

public void testRenameNewFieldAlreadyExists() throws Exception {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
teardown:
- do:
ingest.delete_pipeline:
id: "my_pipeline"
ignore: 404

---
"Test rename processor with non-existing field and without ignore_missing":
- do:
ingest.put_pipeline:
id: "my_pipeline"
body: >
{
"description": "_description",
"processors": [
{
"rename" : {
"field" : "{{field_foo}}",
"target_field" : "bar"
}
}
]
}
- match: { acknowledged: true }

- do:
catch: '/field path cannot be null nor empty/'
index:
index: test
id: 1
pipeline: "my_pipeline"
body: { message: "foo bar baz" }

---
"Test rename processor with non-existing field and ignore_missing":
- do:
ingest.put_pipeline:
id: "my_pipeline"
body: >
{
"description": "_description",
"processors": [
{
"rename" : {
"field" : "{{field_foo}}",
"target_field" : "bar",
"ignore_missing" : true
}
}
]
}
- match: { acknowledged: true }

- do:
index:
index: test
id: 1
pipeline: "my_pipeline"
body: { message: "foo bar baz" }

- do:
get:
index: test
id: 1
- match: { _source.message: "foo bar baz" }