Add support for runtime fields #61776
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Put together some example script contexts for scripted fields. This is almost certainly wrong, but it gets us something to think about.
This reworks the script contexts to support multiple values for each hit. Just like before, these are almost certainly not the final implementation, but they give us something to iterate on.
Painless extensions work differently now.
Uses a temporary hack to allow us to use painless in unit tests.
This is the first draft implementation of scripted field of type keyword. Still lots of TODOs to address, this PR targets a feature branch.
This adds what I think is just about the simplest possible `term` query implementation for `keyword` script fields and wires it into the field mapper that we build for them.
Adds tests for the `keyword` typed `script` field's fielddata implementation. One unit test which runs quickly and an integretion test for fetching and an integration test for the `terms` agg.
This adds the `exits` and `terms` query for the `keyword` style of scripted field.
Adds the remaining queries for scripted keyword fields and handles a few leftover TODOs, mostly around `hashCode` and `equals`.
This updates a test so that runtime fields depend on one another and fixes support for it.
Instead of relying no createComponents to get the script service, we rely on the query shard context supplier that's available in the parser context, yet we only carry around as callback that allows us to compile the script
This will cause them to fail if you don't have permission to execute expensive queries.
There seems to be no longer a way to require a field in mapper paramaters. This needs to be fixed upstream.
The error was calling toString on the parameter object. we need to instead print out the returned value by calling getValue
This addresses a TODO around using the script params, which are now parsed from the mappings. It also expand existing tests to verify that params are carried around and accessible in script for both fielddata and queries. Relates to #59332
This adds the `long` runtime type to `script` fields and implements doc values, field data, the `term` and `exists` query.
javanna
commented
Sep 1, 2020
...mlRestTest/java/org/elasticsearch/xpack/runtimefields/rest/CoreTestsWithRuntimeFieldsIT.java
Show resolved
Hide resolved
nik9000
approved these changes
Sep 1, 2020
There was a problem hiding this comment.
I recognize this code!
Like you told me yesterday, most of the stuff outside of the plugin is about plumbing the script service and handling ranges. Thanks to all of your hard work getting the rest of what we needed into master already.
I found a couple of small things to add to my personal TODO list that I'd forgot about but none of those should block merging.
LGTM
server/src/main/java/org/elasticsearch/index/query/functionscore/DecayFunctionBuilder.java
Show resolved
Hide resolved
x-pack/plugin/runtime-fields/src/main/plugin-metadata/plugin-security.policy
Outdated
Show resolved
Hide resolved
|
Looks like there are some core tests failing with runtime fields: I'm cool skipping those for now to unblock this. |
This adds support for the `format` parameter on `long` and `double` valued long fields. It only adds support for it in the request - in the mapping it is not yet supported.
Runtime fields needs to see the point in time api for its tests.
We had it for testing early on and forgot to remove it.
I pushed 1ea67d2 to fix this. |
Adds explicit tests for nested fields to the runtime fields project. And skip all of the core tests for nested fields. They don't really have a chance of resolving the right fields from the source.
…pts (#61775) This commit exposes stored fields to runtime fields (through params._fields) and replaces the current way to access _source (source['field']) in favour of the documented way for all existing scripts: params._source .
This commit introduces integration tests for Document level security with queries against runtime fields, as well as field level security used in combination with runtime fields. Relates to #59332
) This commit adds tests around field_caps and search against multiple indices, where one index has some fields defined as runtime fields, while another index has the same fields defined as concrete fields. We want to make sure that as long as the data type matches, the two can integrate seamlessly. Relates to #59332
This commit extends the coverage around max chain depth in runtime fields. We test that direct references are not counted, that 5 references are allowed while 6 are not, and we test that the same error is returned no matter where the runtime field is used. Relates to #59332
|
run elasticsearch-ci/docs |
Adds a test for `runtime_script` who's script contains `currentTimeMillis`.
|
run elasticsearch-ci/packaging-sample-windows |
javanna
added a commit
that referenced
this pull request
Sep 7, 2020
This commit includes the work that has been done on the runtime fields feature branch until now. The high level tasks are listed in #59332. The tasks that have not yet been completed can be worked on after merging the feature branch. We are adding a new x-pack plugin called runtime-fields that plugs in a custom mapper which allows to define runtime fields based on a script. The changes included in this commit that were made outside of the x-pack/plugin/runtime-fields directory are minimal and revolve around 1) making the ScriptService available while parsing index mappings so that the scripts associated to runtime fields can be compiled 2) sharing code to manipulate ranges etc. as it can be reused in runtime fields. Co-authored-by: Nik Everett <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes the work that has been done on the runtime fields feature branch until now. The high level tasks are listed in #59332. The tasks that have not yet been completed can be worked on after merging the feature branch to master.
We are adding a new x-pack plugin called
runtime-fieldsthat plugs in a custom mapper which allows to define runtime fields based on a script.The changes included in this PR that were made outside of the
x-pack/plugin/runtime-fieldsdirectory are minimal and revolve around 1) making theScriptServiceavailable while parsing index mappings so that the scripts associated to runtime fields can be compiled 2) sharing code to manipulate ranges etc. as it can be reused in runtime fields .