[CORE] [de]serialize issue with spark shim

[baibaichen]

Backend

All

Bug description

We occasionally encounter the following errors, it should be caused by driver and worker load diffierent implementaion, i.e. vanilla version and gluten version.

To adjust class loading order can workaround this issue, but is there any more robust solution？

Caused by: java.lang.IllegalArgumentException: Illegal lambda deserialization
	at scala.runtime.LambdaDeserializer$.makeCallSite$1(LambdaDeserializer.scala:89) ~[scala-library-2.12.15.jar:?]
	at scala.runtime.LambdaDeserializer$.deserializeLambda(LambdaDeserializer.scala:113) ~[scala-library-2.12.15.jar:?]
	at scala.runtime.LambdaDeserialize.deserializeLambda(LambdaDeserialize.java:38) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$deserializeLambda$(FileFormatWriter.scala) ~[gluten.jar:3.3.0-kylin-5.2.0-SNAPSHOT]

Spark version

None

Spark configurations

All

System information

No response

Relevant logs

No response