Introduce runtime section in mappings #62906

javanna · 2020-09-24T23:10:47Z

The runtime section is at the same level as the existing properties section. Its purpose is to hold runtime fields only. With the introduction of the runtime section, a runtime field can be defined by specifying its type (previously called runtime_type) and script.

PUT /my-index/_mappings
{
    "runtime" : {
        "day_of_week" : {
            "type" : "keyword",
            "script" : {
                "source" : "emit(doc['timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
            }
        }
    },
    "properties" : {
        "timestamp" : {
            "type" : "date"
        }
    }
}

Fields defined in the runtime section can be updated at any time as they are not present in the lucene index. They get replaced entirely when they get updated.

Thanks to the introduction of the runtime section, runtime fields override existing mapped fields defined with the same name, similarly to runtime fields defined in the search request.

Note that the get field mappings API does not return runtime fields as it iterates through the mappers directly, which have no visibility over runtime fields.

Relates to #59332

nik9000

👍 If we want to go this way I think this is how we should do it.

nik9000 · 2020-09-27T21:24:28Z

+1 If we want to go this way I think this is how we should do it.

And by that I mean, I don't see any problems with the code. I'm not 100% sure we want the new section, but if we want it then this implements it how I expect.

server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java

The runtime section is at the same level as the existing properties section. Its purpose is to hold runtime fields only. With the introduction of the runtime section, a runtime field can be defined by specifying its type (previously called runtime_type) and script. Fields defined in the runtime section can be updated at any time as they are not present in the lucene index. They get replaced entirely when they get updated. Thanks to the introduction of the runtime section, runtime fields can override existing mapped fields defined with the same name, similarly to runtime fields defined in the search request.

javanna · 2020-11-11T10:40:16Z

server/src/main/java/org/elasticsearch/index/mapper/RuntimeFieldType.java

+            throws MapperParsingException;
+    }
+
+    public static void parseRuntimeFields(Map<String, Object> node,


I am not a big fan of this public static method here, but I think it's important to share it between QueryShardContext and RootObjectMapper

server/src/test/java/org/elasticsearch/index/IndexSortSettingsTests.java

javanna · 2020-11-11T10:47:03Z

.../test/java/org/elasticsearch/xpack/runtimefields/mapper/AbstractScriptFieldTypeTestCase.java

+        );
+    }
+
+    public final void testMinimalMappingToMaximal() throws IOException {


these three tests I borrowed from MapperTestCase , they were previously run as part of RuntimeFieldMapperTests

javanna · 2020-11-11T10:47:25Z

.../test/java/org/elasticsearch/xpack/runtimefields/mapper/AbstractScriptFieldTypeTestCase.java

+        assertEquals("Failed to parse mapping: stored scripts are not supported for runtime field [field]", exception.getMessage());
+    }
+
+    public void testFieldCaps() throws Exception {


these were moved from RuntimeFieldMapperTests

javanna · 2020-11-11T10:49:32Z

...lds/src/test/java/org/elasticsearch/xpack/runtimefields/mapper/DateScriptFieldTypeTests.java

+        MappedFieldType fieldType = mapperService.fieldType("field");
+        assertThat(fieldType, instanceOf(DateScriptFieldType.class));
+        assertEquals(Strings.toString(mapping.get()), Strings.toString(mapperService.documentMapper()));
+    }


these were moved from RuntimeFieldMapperTests

.../test/java/org/elasticsearch/xpack/runtimefields/mapper/AbstractScriptFieldTypeTestCase.java

nik9000

I left some formatting stuff but nothing big at all. Let me know if you'd like a consult on the sneaky tests.

server/src/test/java/org/elasticsearch/index/IndexSortSettingsTests.java

server/src/test/java/org/elasticsearch/index/mapper/RootObjectMapperTests.java

...plugin/runtime-fields/src/main/java/org/elasticsearch/xpack/runtimefields/RuntimeFields.java

...elds/src/main/java/org/elasticsearch/xpack/runtimefields/mapper/AbstractScriptFieldType.java

nik9000 · 2020-11-11T13:55:09Z

...elds/src/main/java/org/elasticsearch/xpack/runtimefields/mapper/AbstractScriptFieldType.java

+     *  but {@link Builder#init(FieldMapper)} and {@link Builder#build(ContentPath)} are never called as
+     *  {@link RuntimeFieldTypeParser#parse(String, Map, Mapper.TypeParser.ParserContext)} calls
+     *  {@link Builder#parse(String, Mapper.TypeParser.ParserContext, Map)} and returns the corresponding
+     *  {@link org.elasticsearch.index.mapper.MappedFieldType}.
     */


Is this something we should keep doing forever or just a way to get this in without making 123213214 changes? Like, should we stop doing this in a follow up change?

ehm, doing what?

Using the Builder at all. It feels like it makes sense because we have them now, but maybe its overkill?

I like using Parameters to define what gets parsed and to reuse that part of the existing mappings infra. I think that not using them would mean going back to manual parsing and printing.

Me too! I just wonder if we can avoid building the FieldMapper then. Like, in a follow up? Or not. I dunno. It just fields weird to build a dummy field mapper to make parsing easier. No big deal.

we do not build the field mapper at all!

pressed enter too quickly: we skip calling build on the builder. we make the builder create the right field type and return that, without creating the field mapper.

OH! I see you even said it. I should read more closely!

one follow-up we could think of to clean things up is to extract the parsing bits from FieldMapper.Builder as well as the Parameter class so that parsing can be done outside of a Builder.

nik9000 · 2020-11-11T13:59:56Z

x-pack/plugin/src/test/resources/rest-api-spec/test/runtime_fields/10_keyword.yml

                script:
                  source: |
                    for (String node : params._fields.node.values) {
                      emit(params.prefix + node);
                    }
                  params:
                    prefix: node_
+            properties:


Maybe keep the traditional properties at the top? I think the runtime properties will tend to consume the traditional ones so I'd try to read the traditional ones first.

I actually never considered runtime fields could not be on top :) I would expect them to appear first because they can override ordinary fields, I guess.

romseygeek

A couple of queries, but LGTM as a whole.

romseygeek · 2020-11-12T10:24:24Z

server/src/test/java/org/elasticsearch/index/mapper/FieldTypeLookupTests.java

+        {
+            Set<String> sourcePaths = fieldTypeLookup.sourcePaths("field2");
+            assertEquals(1, sourcePaths.size());
+            assertTrue(sourcePaths.contains("field2"));


This is an interesting one. I'm not really sure what sourcePaths should return for runtime fields - I feel like maybe it should be an empty set? The ValueFetcher isn't using sourcePaths to load data for these fields, so maybe it doesn't matter.

I see what you mean, though the logic in the method is very simple and I am reluctant to special case runtime fields, under the assumption that they are never in source. Also runtime fields can be plugged in and maybe we should not make assumptions about how they will fetch their values?

I figure there is a chance runtime fields will use the source path, yeah. But at least for now they don't so it doesn't really matter. I guess emptySet is more correct, but, yeah, it doesn't really matter.

server/src/test/java/org/elasticsearch/index/mapper/RootObjectMapperTests.java

romseygeek · 2020-11-12T10:57:34Z

...elds/src/main/java/org/elasticsearch/xpack/runtimefields/mapper/AbstractScriptFieldType.java

-     * {@code null} because most fields don't support formats.
-     */
-    protected String format() {
-        return null;


Much nicer to only have this on the DateFieldType, thanks!

javanna · 2020-11-12T13:17:57Z

...va/org/elasticsearch/xpack/runtimefields/test/search/CoreTestsWithSearchRuntimeFieldsIT.java

@@ -116,20 +111,22 @@ protected boolean modifySearch(ApiCallSection search) {
                        return mergeMappings(new String[] { "*" });
                    }
                    String[] patterns = Arrays.stream(index.split(",")).map(m -> m.equals("_all") ? "*" : m).toArray(String[]::new);
+                    // TODO this is always false?


@nik9000 this is for you I think :)

ooops. I think I mean length == 1. I can take a look in a follow up.

nik9000 · 2020-11-11T16:44:48Z

...elds/src/main/java/org/elasticsearch/xpack/runtimefields/mapper/AbstractScriptFieldType.java

+     *  but {@link Builder#init(FieldMapper)} and {@link Builder#build(ContentPath)} are never called as
+     *  {@link RuntimeFieldTypeParser#parse(String, Map, Mapper.TypeParser.ParserContext)} calls
+     *  {@link Builder#parse(String, Mapper.TypeParser.ParserContext, Map)} and returns the corresponding
+     *  {@link org.elasticsearch.index.mapper.MappedFieldType}.
     */


OH! I see you even said it. I should read more closely!

nik9000 · 2020-11-12T13:38:22Z

server/src/test/java/org/elasticsearch/index/mapper/FieldTypeLookupTests.java

+        {
+            Set<String> sourcePaths = fieldTypeLookup.sourcePaths("field2");
+            assertEquals(1, sourcePaths.size());
+            assertTrue(sourcePaths.contains("field2"));


I figure there is a chance runtime fields will use the source path, yeah. But at least for now they don't so it doesn't really matter. I guess emptySet is more correct, but, yeah, it doesn't really matter.

nik9000 · 2020-11-12T13:38:55Z

server/src/test/java/org/elasticsearch/index/mapper/RootObjectMapperTests.java

-                        .endObject()
-                    .endObject()
-                .endObject()
+            .startObject()


This one got harder to read!

nik9000 · 2020-11-12T13:39:56Z

x-pack/plugin/runtime-fields/qa/build.gradle

@@ -1,4 +1,4 @@
-// Shared infratructure
+// Shared infrastructure


nik9000 · 2020-11-12T13:42:08Z

...va/org/elasticsearch/xpack/runtimefields/test/search/CoreTestsWithSearchRuntimeFieldsIT.java

@@ -116,20 +111,22 @@ protected boolean modifySearch(ApiCallSection search) {
                        return mergeMappings(new String[] { "*" });
                    }
                    String[] patterns = Arrays.stream(index.split(",")).map(m -> m.equals("_all") ? "*" : m).toArray(String[]::new);
+                    // TODO this is always false?


ooops. I think I mean length == 1. I can take a look in a follow up.

nik9000 · 2020-11-12T13:42:30Z

...plugin/runtime-fields/src/main/java/org/elasticsearch/xpack/runtimefields/RuntimeFields.java

-    public Map<String, Mapper.TypeParser> getMappers() {
-        return Collections.singletonMap(RuntimeFieldMapper.CONTENT_TYPE, RuntimeFieldMapper.PARSER);
+    public Map<String, RuntimeFieldType.Parser> getRuntimeFieldTypes() {
+        return Map.ofEntries(


The runtime section is at the same level as the existing properties section. Its purpose is to hold runtime fields only. With the introduction of the runtime section, a runtime field can be defined by specifying its type (previously called runtime_type) and script. ``` PUT /my-index/_mappings { "runtime" : { "day_of_week" : { "type" : "keyword", "script" : { "source" : "emit(doc['timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" } } }, "properties" : { "timestamp" : { "type" : "date" } } } ``` Fields defined in the runtime section can be updated at any time as they are not present in the lucene index. They get replaced entirely when they get updated. Thanks to the introduction of the runtime section, runtime fields override existing mapped fields defined with the same name, similarly to runtime fields defined in the search request. Relates to elastic#59332

The runtime section is at the same level as the existing properties section. Its purpose is to hold runtime fields only. With the introduction of the runtime section, a runtime field can be defined by specifying its type (previously called runtime_type) and script. ``` PUT /my-index/_mappings { "runtime" : { "day_of_week" : { "type" : "keyword", "script" : { "source" : "emit(doc['timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" } } }, "properties" : { "timestamp" : { "type" : "date" } } } ``` Fields defined in the runtime section can be updated at any time as they are not present in the lucene index. They get replaced entirely when they get updated. Thanks to the introduction of the runtime section, runtime fields override existing mapped fields defined with the same name, similarly to runtime fields defined in the search request. Relates to #59332

Runtime fields are defined in a separate runtime section in the mappings. Since the runtime section was introduced, runtime fields are not taken into account when parsing documents. That means that if a document gets indexed that holds a field that's already defined as a runtime field, the field gets dynamically mapped as a concrete field although it will always be shadowed by the runtime field defined with the same name. A more sensible default would be to instead consider runtime fields like ordinary mapped fields, so a dynamic update is not necessary whenever a field is defined as part of the runtime section. As a consequence, the field does not get indexed. If users prefer to keep indexing the field although it is shadowed, we consider this an exception, and they can do so by mapping the field under properties explicitly. Relates to elastic#62906

Runtime fields are defined in a separate runtime section in the mappings. Since the runtime section was introduced, runtime fields are not taken into account when parsing documents. That means that if a document gets indexed that holds a field that's already defined as a runtime field, the field gets dynamically mapped as a concrete field although it will always be shadowed by the runtime field defined with the same name. A more sensible default would be to instead consider runtime fields like ordinary mapped fields, so a dynamic update is not necessary whenever a field is defined as part of the runtime section. As a consequence, the field does not get indexed. If users prefer to keep indexing the field although it is shadowed, we consider this an exception, and they can do so by mapping the field under properties explicitly. Relates to #62906

Runtime fields are defined in a separate runtime section in the mappings. Since the runtime section was introduced, runtime fields are not taken into account when parsing documents. That means that if a document gets indexed that holds a field that's already defined as a runtime field, the field gets dynamically mapped as a concrete field although it will always be shadowed by the runtime field defined with the same name. A more sensible default would be to instead consider runtime fields like ordinary mapped fields, so a dynamic update is not necessary whenever a field is defined as part of the runtime section. As a consequence, the field does not get indexed. If users prefer to keep indexing the field although it is shadowed, we consider this an exception, and they can do so by mapping the field under properties explicitly. Relates to elastic#62906

Runtime fields are defined in a separate runtime section in the mappings. Since the runtime section was introduced, runtime fields are not taken into account when parsing documents. That means that if a document gets indexed that holds a field that's already defined as a runtime field, the field gets dynamically mapped as a concrete field although it will always be shadowed by the runtime field defined with the same name. A more sensible default would be to instead consider runtime fields like ordinary mapped fields, so a dynamic update is not necessary whenever a field is defined as part of the runtime section. As a consequence, the field does not get indexed. If users prefer to keep indexing the field although it is shadowed, we consider this an exception, and they can do so by mapping the field under properties explicitly. Relates to #62906

javanna requested review from nik9000 and romseygeek September 24, 2020 23:10

nik9000 reviewed Sep 27, 2020

View reviewed changes

romseygeek reviewed Sep 28, 2020

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java Outdated Show resolved Hide resolved

javanna force-pushed the enhancement/mapping_runtime_section branch from d7cbcce to dd5f648 Compare November 10, 2020 16:47

javanna changed the title ~~WIP: introduce runtime section in mappings~~ Introduce runtime section in mappings Nov 10, 2020

javanna added :Search Foundations/Mapping Index mappings, including merging and defining field types v7.11.0 v8.0.0 >enhancement labels Nov 10, 2020

javanna added 4 commits November 10, 2020 18:12

iter

8bad8db

fix PermissionsIT

5f05998

Merge branch 'master' into enhancement/mapping_runtime_section

740425f

Merge branch 'master' into enhancement/mapping_runtime_section

ae57332

javanna commented Nov 11, 2020

View reviewed changes

server/src/test/java/org/elasticsearch/index/IndexSortSettingsTests.java Show resolved Hide resolved

javanna commented Nov 11, 2020

View reviewed changes

javanna added 2 commits November 11, 2020 11:51

iter

353dcd8

import

357f821

romseygeek reviewed Nov 11, 2020

View reviewed changes

.../test/java/org/elasticsearch/xpack/runtimefields/mapper/AbstractScriptFieldTypeTestCase.java Outdated Show resolved Hide resolved

fix tests

7844567

nik9000 reviewed Nov 11, 2020

View reviewed changes

javanna added 4 commits November 11, 2020 15:55

iter

58c07f7

iter

131f83c

Merge branch 'master' into enhancement/mapping_runtime_section

9998821

spotless

380a133

javanna added 4 commits November 11, 2020 17:29

formatting

6c4edd4

fix test

b53bf03

Merge branch 'master' into enhancement/mapping_runtime_section

1701c2e

Merge branch 'master' into enhancement/mapping_runtime_section

3e64d01

javanna requested review from romseygeek and nik9000 November 12, 2020 09:13

fix test

21a2f47

romseygeek approved these changes Nov 12, 2020

View reviewed changes

javanna added 4 commits November 12, 2020 12:36

expand tests

9b38b20

Merge branch 'master' into enhancement/mapping_runtime_section

7fc73eb

add comment

4b8361e

fix test

e081564

javanna commented Nov 12, 2020

View reviewed changes

nik9000 reviewed Nov 12, 2020

View reviewed changes

nik9000 approved these changes Nov 12, 2020

View reviewed changes

restore formatting

567ff82

javanna merged commit 3101293 into elastic:master Nov 12, 2020

wylieconlon mentioned this pull request Nov 16, 2020

[Lens] Fix runtime mappings by using new API elastic/kibana#83350

Closed

pheyos mentioned this pull request Nov 17, 2020

Functional tests - fix esArchive mappings with runtime fields elastic/kibana#83530

Merged

javanna mentioned this pull request Nov 18, 2020

Make document parsing aware of runtime fields #65210

Merged

javanna mentioned this pull request Nov 19, 2020

Make document parsing aware of runtime fields (#65210) #65282

Merged

stevejgordon mentioned this pull request Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce runtime section in mappings #62906

Introduce runtime section in mappings #62906

javanna commented Sep 24, 2020 •

edited

Loading

nik9000 left a comment

nik9000 commented Sep 27, 2020

javanna Nov 11, 2020

javanna Nov 11, 2020

javanna Nov 11, 2020

javanna Nov 11, 2020

nik9000 left a comment

nik9000 Nov 11, 2020

javanna Nov 11, 2020

nik9000 Nov 11, 2020

javanna Nov 11, 2020

nik9000 Nov 11, 2020

javanna Nov 11, 2020

javanna Nov 11, 2020

nik9000 Nov 11, 2020

javanna Nov 12, 2020 •

edited

Loading

nik9000 Nov 11, 2020

javanna Nov 11, 2020

romseygeek left a comment

romseygeek Nov 12, 2020

javanna Nov 12, 2020

nik9000 Nov 12, 2020

romseygeek Nov 12, 2020

javanna Nov 12, 2020

nik9000 Nov 12, 2020

nik9000 Nov 11, 2020

nik9000 Nov 12, 2020

nik9000 Nov 12, 2020

javanna Nov 12, 2020

nik9000 Nov 12, 2020

nik9000 Nov 12, 2020

nik9000 Nov 12, 2020

		@@ -1,4 +1,4 @@
		// Shared infratructure
		// Shared infrastructure

Introduce runtime section in mappings #62906

Introduce runtime section in mappings #62906

Conversation

javanna commented Sep 24, 2020 • edited Loading

nik9000 left a comment

Choose a reason for hiding this comment

nik9000 commented Sep 27, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javanna Nov 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

romseygeek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javanna commented Sep 24, 2020 •

edited

Loading

javanna Nov 12, 2020 •

edited

Loading