-
Notifications
You must be signed in to change notification settings - Fork 75
[NSE-581] Improve GetArrayItem(Split()) performance #933
Conversation
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/oap-project/native-sql-engine/issues Then could you also rename commit message and pull request title in the following format?
See also: |
Signed-off-by: Yuan Zhou <[email protected]>
8810e41
to
5a41c36
Compare
copy(str = newFirst, regex = newSecond, limit = newThird) | ||
} | ||
|
||
case class StringSplitPart(str: Expression, regex: Expression, limit: Expression, index: Expression) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I prefer to register StringSplitPart through UDFRegistration, rather than adding whole expression code to gazelle, which will affect the difficulty of upgrading spark versions later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, these codes will be moved to Spark321 shim layers or dynamically register this on session starts - will also try to upstream SplitPart
to upstream Spark
StringSplit(str.expr, Literal(pattern), Literal(limit)) | ||
} | ||
def splitpart(str: Column, pattern: String, limit: Int, index: Int): Column = withExpr { | ||
StringSplitPart(str.expr, Literal(pattern), Literal(limit), Literal(index)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
Signed-off-by: Yuan Zhou <[email protected]>
To align with spark's behavior, please apply this patch to arrow: oap-project/arrow#107. |
Signed-off-by: Yuan Zhou <[email protected]>
Signed-off-by: Yuan Zhou <[email protected]>
Signed-off-by: Yuan Zhou <[email protected]>
This reverts commit 4cda7d5.
What changes were proposed in this pull request?
The implementation converts
GetArrayItem(Split())
intoSplitPart()
as Gazelle does not support Array based functions yet.How was this patch tested?
pass jenkins