Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Expression pushdown optimization #663

Conversation

dai-chen
Copy link
Member

@dai-chen dai-chen commented Aug 5, 2020

Issue #, if available:

Description of changes:

  1. New script language: Register the expression in new query engine as a script language in Elasticsearch. In this case, we don't need to maintain painless source for each expression.
  2. The first half: serialize the expression and send it along with DSL request.
  3. The second half: the code in DSL is deserialized to run on each node as if built-in painless script.
  4. Expression visitor: Add expression node visitor because both first and second half needs to visit expression tree.

Limitation:

  1. Lucene query optimization: For now, the entire expression would be pushed down to a script query. There will be another PR for Lucene query API pushdown soon.
  2. Multi-field name: Assumption is made that the inner field name is always "keyword", so "text.keyword" is used to get doc value for multi-field.
  3. Serialization: for now JDK serializer is in use. Due to not all expressions serializable, ex. arithmetic expression, pipelining filter operator will be used as before if serialization error happens.

Testing: Add UT and pass all ITs. Because of explain API missing in new engine, new IT will be added later.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@dai-chen dai-chen self-assigned this Aug 5, 2020
@dai-chen dai-chen marked this pull request as ready for review August 6, 2020 20:29
@dai-chen dai-chen requested review from penghuo and chloe-zh August 6, 2020 20:29
Copy link
Contributor

@penghuo penghuo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for change!

@dai-chen dai-chen merged commit 032e3ba into opendistro-for-elasticsearch:develop Aug 11, 2020
@dai-chen dai-chen deleted the expression-pushdown-optimization branch August 11, 2020 15:42
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants