Ingest: Enable Templated Fieldnames in Rename #31690

original-brownbear · 2018-06-29T15:14:39Z

Adds templating to rename's field and target field to fix #29657

Adds templating to rename's field and target field to fix elastic#29657

elasticmachine · 2018-06-29T15:14:40Z

Pinging @elastic/es-core-infra

original-brownbear · 2018-07-06T09:05:22Z

@rjernst ping for review if and when you have a sec :)

martijnvg

Looks good. I left a few minor comments. Also this change goes in to 6.x, right?

martijnvg · 2018-07-09T08:29:52Z

modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/IngestCommonPlugin.java

@@ -96,12 +96,12 @@ public IngestCommonPlugin() {
                                             Supplier<DiscoveryNodes> nodesInCluster) {
        return Arrays.asList(new GrokProcessorGetAction.RestAction(settings, restController));
    }
-    
+


unneeded whitespace changes?

martijnvg · 2018-07-09T08:31:08Z

modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RenameProcessor.java

@@ -58,28 +60,30 @@ boolean isIgnoreMissing() {

    @Override
    public void execute(IngestDocument document) {
-        if (document.hasField(field, true) == false) {
+        String source = document.renderTemplate(field);


maybe rename source to path?

martijnvg · 2018-07-09T08:33:26Z

...ingest-common/src/test/java/org/elasticsearch/ingest/common/RenameProcessorFactoryTests.java

-        assertThat(renameProcessor.getField(), equalTo("old_field"));
-        assertThat(renameProcessor.getTargetField(), equalTo("new_field"));
+        assertThat(renameProcessor.getField().newInstance(Collections.emptyMap()).execute(), equalTo("old_field"));
+        assertThat(renameProcessor.getTargetField().newInstance(Collections.emptyMap()).execute(), equalTo("new_field"));


Maybe add a test that does a rename with a {{var}} snippet and pass down a map with actual key/value in it?

@martijnvg I think I can't do that (maybe I'm wrong here?) easily because I don't have the Mustache script engine available in these tests.
With the org.elasticsearch.ingest.TestTemplateService all the tests are using org.elasticsearch.ingest.TestTemplateService.MockTemplateScript.Factory#newInstance is called which simply ignores the params Map.

whoops, I forgot that.

original-brownbear · 2018-07-09T09:31:51Z

@martijnvg handling the first 2 points in 0423dd3 and 53b14f6 and commented on the last point.

And yes, I think this should go into 6.x

martijnvg

LGTM

original-brownbear · 2018-07-09T09:50:57Z

@martijnvg thanks!, will merge once green :)

* Ingest: Enable Templated Fieldnames in Rename

* master: [TEST] Mute SlackMessageTests.testTemplateRender Docs: Explain closing the high level client [ML] Re-enable memory limit integration tests (#31328) [test] disable packaging tests for suse boxes Add nio transport to security plugin (#31942) XContentTests : Insert random fields at random positions (#30867) Force execution of fetch tasks (#31974) Fix unreachable error condition in AmazonS3Fixture (#32005) Tests: Fix SearchFieldsIT.testDocValueFields (#31995) Add Expected Reciprocal Rank metric (#31891) [ML] Get ForecastRequestStats doc in RestoreModelSnapshotIT (#31973) SQL: Add support for single parameter text manipulating functions (#31874) [ML] Ensure immutability of MlMetadata (#31957) Tests: Mute SearchFieldsIT.testDocValueFields() muted tests due to #31940 Work around reported problem in eclipse (#31960) Move build integration tests out of :buildSrc project (#31961) Tests: Remove use of joda time in some tests (#31922) [Test] Reactive 3rd party tests on CI (#31919) SQL: Support for escape sequences (#31884) SQL: HAVING clause should accept only aggregates (#31872) Docs: fix typo in datehistogram (#31972) Switch url repository rest tests to new style requests (#31944) Switch reindex tests to new style requests (#31941) Docs: Added note about cloud service to installation and getting started [DOCS] Removes alternative docker pull example (#31934) Add Snapshots Status API to High Level Rest Client (#31515) ingest: date_index_name processor template resolution (#31841) Test: fix null failure in watcher test (#31968) Switch test framework to new style requests (#31939) Switch low level rest tests to new style Requests (#31938) Switch high level rest tests to new style requests (#31937) [ML] Mute test failing due to Java 11 date time format parsing bug (#31899) [TEST] Mute SlackMessageTests.testTemplateRender Fix assertIngestDocument wrongfully passing (#31913) Remove unused reference to filePermissionsCache (#31923) rolling upgrade should use a replica to prevent relocations while running a scroll HLREST: Bundle the x-pack protocol project (#31904) Increase logging level for testStressMaybeFlush Added lenient flag for synonym token filter (#31484) [X-Pack] Beats centralized management: security role + licensing (#30520) HLRest: Move xPackInfo() to xPack().info() (#31905) Docs: add security delete role to api call table (#31907) [test] port archive distribution packaging tests (#31314) Watcher: Slack message empty text (#31596) [ML] Mute failing DetectionRulesIT.testCondition() test Fix broken NaN check in MovingFunctions#stdDev() (#31888) Date: Add DateFormatters class that uses java.time (#31856) [ML] Switch native QA tests to a 3 node cluster (#31757) Change trappy float comparison (#31889) Fix building AD URL from domain name (#31849) Add opaque_id to audit logging (#31878) re-enable backcompat tests add support for is_write_index in put-alias body parsing (#31674) Improve release notes script (#31833) [DOCS] Fix broken link in painless example Handle missing values in painless (#30975) Remove the ability to index or query context suggestions without context (#31007) Ingest: Enable Templated Fieldnames in Rename (#31690) [Docs] Fix typo in the Rollup API Quick Reference (#31855) Ingest: Add ignore_missing option to RemoveProc (#31693) Add template config for Beat state to X-Pack Monitoring (#31809) Watcher: Add ssl.trust email account setting (#31684) Remove link to oss-MSI (#31844) Painless: Restructure Definition/Whitelist (#31879) HLREST: Add x-pack-info API (#31870)

* 6.x: Force execution of fetch tasks (#31974) [TEST] Mute SlackMessageTests.testTemplateRender Docs: Explain closing the high level client [test] disable java packaging tests for suse XContentTests : Insert random fields at random positions (#30867) Add Get Snapshots High Level REST API (#31980) Fix unreachable error condition in AmazonS3Fixture (#32005) [6.x][ML] Ensure immutability of MlMetadata (#31994) Add Expected Reciprocal Rank metric (#31891) SQL: Add support for single parameter text manipulating functions (#31874) muted tests due to #31940 Work around reported problem in eclipse (#31960) Move build integration tests out of :buildSrc project (#31961) [Test] Reactive 3rd party tests on CI (#31919) Fix assertIngestDocument wrongfully passing (#31913) (#31951) SQL: Support for escape sequences (#31884) SQL: HAVING clause should accept only aggregates (#31872) Docs: fix typo in datehistogram (#31972) Switch url repository rest tests to new style requests (#31944) Switch reindex tests to new style requests (#31941) Switch test framework to new style requests (#31939) Docs: Added note about cloud service to installation and getting started [DOCS] Removes alternative docker pull example (#31934) ingest: date_index_name processor template resolution (#31841) Test: fix null failure in watcher test (#31968) Watcher: Slack message empty text (#31596) Switch low level rest tests to new style Requests (#31938) Switch high level rest tests to new style requests (#31937) HLREST: Bundle the x-pack protocol project (#31904) [ML] Mute test failing due to Java 11 date time format parsing bug (#31899) Increase logging level for testStressMaybeFlush rolling upgrade should use a replica to prevent relocations while running a scroll [test] port archive distribution packaging tests (#31314) HLRest: Move xPackInfo() to xPack().info() (#31905) Increase logging level for :qa:rolling-upgrade Backport: Add template config for Beat state to X-Pack Monitoring (#31809) (#31893) Fix building AD URL from domain name (#31849) Fix broken NaN check in MovingFunctions#stdDev() (#31888) Change trappy float comparison (#31889) Add opaque_id to audit logging (#31878) add support for is_write_index in put-alias body parsing (#31674) Ingest: Enable Templated Fieldnames in Rename (#31690) (#31896) Ingest: Add ignore_missing option to RemoveProc (#31693) (#31892) [Docs] Fix typo in the Rollup API Quick Reference (#31855) Watcher: Add ssl.trust email account setting (#31684) [PkiRealm] Invalidate cache on role mappings change (#31510) [Security] Check auth scheme case insensitively (#31490) HLREST: Add x-pack-info API (#31870) Remove link to oss-MSI (#31844) Painless: Restructure Definition/Whitelist (#31879)

Ingest: Enable Templated Fieldnames in Rename

2fdb855

Adds templating to rename's field and target field to fix elastic#29657

original-brownbear added the :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP label Jun 29, 2018

original-brownbear requested review from rjernst and talevy June 29, 2018 15:14

original-brownbear added the review label Jul 3, 2018

martijnvg reviewed Jul 9, 2018

View reviewed changes

original-brownbear added 3 commits July 9, 2018 11:10

Merge remote-tracking branch 'elastic/master' into 29657

efe3d10

Fix redundant whitespace change

0423dd3

Rename local variable

53b14f6

martijnvg approved these changes Jul 9, 2018

View reviewed changes

original-brownbear added v7.0.0 v6.4.0 labels Jul 9, 2018

original-brownbear merged commit 5f5157a into elastic:master Jul 9, 2018

original-brownbear deleted the 29657 branch July 9, 2018 11:50

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request Jul 9, 2018

Ingest: Enable Templated Fieldnames in Rename (elastic#31690)

7770959

* Ingest: Enable Templated Fieldnames in Rename

original-brownbear mentioned this pull request Jul 9, 2018

Ingest: Enable Templated Fieldnames in Rename (#31690) #31896

Merged

original-brownbear added a commit that referenced this pull request Jul 9, 2018

Ingest: Enable Templated Fieldnames in Rename (#31690) (#31896)

156d580

* Ingest: Enable Templated Fieldnames in Rename

lcawl added the >enhancement label Aug 20, 2018

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ingest: Enable Templated Fieldnames in Rename #31690

Ingest: Enable Templated Fieldnames in Rename #31690

original-brownbear commented Jun 29, 2018

elasticmachine commented Jun 29, 2018

original-brownbear commented Jul 6, 2018

martijnvg left a comment

martijnvg Jul 9, 2018

martijnvg Jul 9, 2018 •

edited

Loading

martijnvg Jul 9, 2018

original-brownbear Jul 9, 2018

martijnvg Jul 9, 2018

original-brownbear commented Jul 9, 2018

martijnvg left a comment

original-brownbear commented Jul 9, 2018

Ingest: Enable Templated Fieldnames in Rename #31690

Ingest: Enable Templated Fieldnames in Rename #31690

Conversation

original-brownbear commented Jun 29, 2018

elasticmachine commented Jun 29, 2018

original-brownbear commented Jul 6, 2018

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Jul 9, 2018

Choose a reason for hiding this comment

martijnvg Jul 9, 2018 • edited Loading

Choose a reason for hiding this comment

martijnvg Jul 9, 2018

Choose a reason for hiding this comment

original-brownbear Jul 9, 2018

Choose a reason for hiding this comment

martijnvg Jul 9, 2018

Choose a reason for hiding this comment

original-brownbear commented Jul 9, 2018

martijnvg left a comment

Choose a reason for hiding this comment

original-brownbear commented Jul 9, 2018

martijnvg Jul 9, 2018 •

edited

Loading