Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better UDF support #152

Merged
merged 29 commits into from
Jun 21, 2022
Merged

Better UDF support #152

merged 29 commits into from
Jun 21, 2022

Conversation

Jolanrensen
Copy link
Collaborator

from issue: #143

One gotcha I noticed is that

val toNormalClass2 by udf.register { a: String, b: Int ->
    NormalClass(b, a)
}
shouldThrow<AnalysisException> { // toNormalClass2 is never accessed, so the delegate getValue function is not executed
    spark.sql("select toNormalClass2(first, second) from test2").show()
}

won't work since toNormalClass2 is never accessed, so the register function is not executed. A simple val a = toNormalClass2 already fixes this, but it's not optimal...

@Jolanrensen Jolanrensen added this to the 1.1.1 milestone May 30, 2022
@Jolanrensen Jolanrensen changed the base branch from spark-3.2 to main May 30, 2022 11:54
@Jolanrensen
WIP, col.typed<>(), refactoring register function
40cde30
@Jolanrensen
wip new udf define methods etc. working on register()
e32e89f
@Jolanrensen
register now works with typedUDFs and directly in the call
aef84d0
@Jolanrensen
working on unnamed and normal udf version
2f3be59
@Jolanrensen
wip
51b3260
@Jolanrensen
Good luck Qodana.
3f0f8cd 
Created UserDefinedFunction and NamedUserDefinedFunction and 22 instances of each.
udf() and udf.register() functions with/without name for lambda, function reference, functional value reference.
.register() extension function in KSparkSession as well
aggregatorOf() function and udaf(). Can also be done using udf.register()
@Jolanrensen Jolanrensen changed the title Created udf delegates and tests Better UDF support Jun 7, 2022
@Jolanrensen
Copy link
Collaborator Author

Maybe a vararg typed udf as final one?

@Jolanrensen
added vararg udf version which can be created by making a single arra…
a00d230 
…y udf (since only WrappedArrays are allowed in udfs anyways).

Added TypedColumn.asWrappedArray() functions for list-likes
@Jolanrensen
updated to kotlin 1.7.0
2c8bd5f 
working on vararg udfs
Vararg unwrapper for spark (wip)
@Jolanrensen
moved vararg unwrapper to scala since inheriting 20+ scala traits fro…
2b801e9 
…m kotlin makes OOM hell break loose.
@Jolanrensen
reworked column functions to prefer typed, so select() can be used in…
cb68fc8 
…stead of selectTyped()
@Jolanrensen
singleCol()
2ab2436
@Jolanrensen
added deprecations for UDFRegister.kt
574018e 
fixes overall
tests, tests, tests (wip)
@Jolanrensen
Merge branch 'spark-3.2' into udf+
7d13bc9 
# Conflicts:
#	pom.xml
@Jolanrensen
Copy link
Collaborator Author

To do: Finish tests, add examples, add in wiki/readme, probably merge udt first since column functions are changed here as well.

This was referenced Jun 15, 2022
Merged
Closed
@Jolanrensen
Merge branch 'main' into udf+
cc39f1d 
# Conflicts:
#	jupyter/src/main/kotlin/org/jetbrains/kotlinx/spark/api/jupyter/SparkIntegration.kt
#	kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/SparkSession.kt
@Jolanrensen
udaf examples
55d14ee
@Jolanrensen
vararg examples
81c692b
@Jolanrensen
readme and examples
aee1a79
@Jolanrensen
Copy link
Collaborator Author

updated wiki https://github.com/Kotlin/kotlin-spark-api/wiki/UDF

@Jolanrensen Jolanrensen requested a review from asm0dey June 20, 2022 13:20
@Jolanrensen
Copy link
Collaborator Author

@asm0dey whenever you have time to check it :)

asm0dey
asm0dey reviewed Jun 20, 2022
View reviewed changes
kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/UDFDelegates.kt Outdated
// }
//
///** Creates delegate of [UDFRegistration]. Use like:
// * ```kotlin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this file should be deleted

asm0dey
asm0dey reviewed Jun 20, 2022
View reviewed changes
kotlin-spark-api/3.2/src/main/kotlin/org/jetbrains/kotlinx/spark/api/UDFRegister.kt
@@ -91,6 +51,7 @@ class UDFWrapper0(private val udfName: String) {
* Registers the [func] with its [name] in [this].
*/
@OptIn(ExperimentalStdlibApi::class)
@Deprecated("Use new UDF notation", ReplaceWith("this.register(name, func)"), DeprecationLevel.HIDDEN)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this!

@asm0dey
Copy link
Contributor

asm0dey commented Jun 20, 2022

It's definitely 1.2.0, not 1.1.1

@Jolanrensen Jolanrensen merged commit c85eee0 into main Jun 21, 2022
@Jolanrensen Jolanrensen deleted the udf+ branch June 21, 2022 19:22
This was referenced Jun 21, 2022
Closed
Merged
Jolanrensen added a commit that referenced this pull request Jun 21, 2022
@Jolanrensen
hotfix for #152
c8c702e
Jolanrensen added a commit that referenced this pull request Jun 22, 2022
@Jolanrensen
hotfix for #152
96f0440
@Jolanrensen
Copy link
Collaborator Author

For future reference, these files were used to generate the UDFs:

generateUDF.zip

@Jolanrensen Jolanrensen mentioned this pull request Jul 29, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asm0dey asm0dey asm0dey left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.2.1
Development

Successfully merging this pull request may close these issues.
2 participants
@Jolanrensen @asm0dey