Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[KYUUBI #6577] Add spark sql engine plugin module and define a sql stringify plugin #6578

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

[KYUUBI #6577] Add spark sql engine plugin module and define a sql stringify plugin #6578

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c3ce8dd
[KYUUBI #6577] Define a PlanOnlyExecutor interface to extend plan onl…
wForget Aug 2, 2024
a4f9dd3
newline
wForget Aug 2, 2024
24f1f79
license
wForget Aug 2, 2024
5129317
fix style
wForget Aug 2, 2024
8d65c7e
narrow interface definition
wForget Aug 16, 2024
870a9b4
refactor
wForget Aug 16, 2024
4aca8ea
address comments
wForget Aug 16, 2024
f8fdafe
add readme
wForget Aug 16, 2024
567e185
fix
wForget Sep 5, 2024
5f5494e
fix readme
wForget Sep 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 0 additions & 35 deletions ...k-engine-plugin/src/main/java/org/apache/kyuubi/engine/spark/plugin/PlanOnlyExecutor.java
View file
Open in desktop

This file was deleted.

4 changes: 2 additions & 2 deletions .../spark/kyuubi-spark-engine-plugin/pom.xml → ...rk/kyuubi-spark-sql-engine-plugin/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@
<relativePath>../../../pom.xml</relativePath>
</parent>

<artifactId>kyuubi-spark-engine-plugin</artifactId>
<artifactId>kyuubi-spark-sql-engine-plugin</artifactId>
<packaging>jar</packaging>
<name>Kyuubi Project Spark Engine Plugin</name>
<name>Kyuubi Project Spark SQL Engine Plugin</name>
<url>https://kyuubi.apache.org/</url>

<dependencies>
Expand Down
13 changes: 4 additions & 9 deletions ...yuubi/engine/spark/plugin/SparkPlans.java → ...engine/spark/plugin/PlanOnlyExecutor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,13 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.kyuubi.engine.spark.plugin;

import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
import org.apache.spark.sql.execution.SparkPlan;
import org.apache.spark.sql.SparkSession;

public interface SparkPlans {
public interface PlanOnlyExecutor {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about SQLStringifyPlugin? Since output does not have to be a plan

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a developer guide?


LogicalPlan parsedPlan();
LogicalPlan analyzedPlan();
LogicalPlan optimizedPlan();
SparkPlan sparkPlan();
SparkPlan executedPlan();
String mode();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we move the mode to execute?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will use SPI to load plugins, so we need to define a qualified name for the plugin to distinguish them.


String execute(SparkSession spark, String statement);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe toString

}
2 changes: 1 addition & 1 deletion externals/kyuubi-spark-sql-engine/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@

<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-engine-plugin</artifactId>
<artifactId>kyuubi-spark-sql-engine-plugin</artifactId>
<version>${project.version}</version>
</dependency>

Expand Down
21 changes: 9 additions & 12 deletions ...ql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/PlanOnlyStatement.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@

package org.apache.kyuubi.engine.spark.operation

import java.util.Locale

import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import org.apache.spark.kyuubi.SparkUtilsHelper
Expand All @@ -30,7 +28,7 @@ import org.apache.spark.sql.types.StructType
import org.apache.kyuubi.KyuubiSQLException
import org.apache.kyuubi.config.KyuubiConf.{LINEAGE_PARSER_PLUGIN_PROVIDER, OPERATION_PLAN_ONLY_EXCLUDES, OPERATION_PLAN_ONLY_OUT_STYLE}
import org.apache.kyuubi.engine.spark.KyuubiSparkUtil.getSessionConf
import org.apache.kyuubi.engine.spark.operation.planonly.{PlanOnlyExecutors, SparkPlansImpl}
import org.apache.kyuubi.engine.spark.operation.planonly.PlanOnlyExecutors
import org.apache.kyuubi.operation.{AnalyzeMode, ArrayFetchIterator, ExecutionMode, IterableFetchIterator, JsonStyle, LineageMode, OperationHandle, OptimizeMode, OptimizeWithStatsMode, ParseMode, PhysicalMode, PlainStyle, PlanOnlyMode, PlanOnlyStyle, UnknownMode, UnknownStyle}
import org.apache.kyuubi.operation.PlanOnlyMode.{notSupportedModeError, unknownModeError}
import org.apache.kyuubi.operation.PlanOnlyStyle.{notSupportedStyleError, unknownStyleError}
Expand Down Expand Up @@ -81,11 +79,14 @@ class PlanOnlyStatement(
result = spark.sql(statement)
iter = new ArrayFetchIterator(result.collect())

case plan => style match {
case PlainStyle => explainWithPlainStyle(plan)
case JsonStyle => explainWithJsonStyle(plan)
case UnknownStyle => unknownStyleError(style)
case other => throw notSupportedStyleError(other, "Spark SQL")
case plan =>
// TODO: remove style configuration, keep only mode configuration
(mode, style) match {
case (PlanOnlyExecutors(executor), _) => executor.execute(spark, statement)
case (_, PlainStyle) => explainWithPlainStyle(plan)
case (_, JsonStyle) => explainWithJsonStyle(plan)
case (_, UnknownStyle) => unknownStyleError(style)
case (_, other) => throw notSupportedStyleError(other, "Spark SQL")
}
}
}
Expand Down Expand Up @@ -153,10 +154,6 @@ class PlanOnlyStatement(
case LineageMode =>
val result = parseLineage(spark, plan)
iter = new IterableFetchIterator(Seq(Row(result)))
case PlanOnlyExecutors(executor) =>
val sparkPlans = SparkPlansImpl(spark, plan)
val result = executor.execute(sparkPlans, style.name.toLowerCase(Locale.ROOT))
iter = new IterableFetchIterator(Seq(Row(result)))
case UnknownMode => throw unknownModeError(mode)
case _ =>
throw KyuubiSQLException(s"The operation mode $mode" +
Expand Down
52 changes: 0 additions & 52 deletions ...ine/src/main/scala/org/apache/kyuubi/engine/spark/operation/planonly/SparkPlansImpl.scala
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
<module>extensions/spark/kyuubi-spark-connector-common</module>
<module>extensions/spark/kyuubi-spark-connector-tpcds</module>
<module>extensions/spark/kyuubi-spark-connector-tpch</module>
<module>extensions/spark/kyuubi-spark-engine-plugin</module>
<module>extensions/spark/kyuubi-spark-sql-engine-plugin</module>
<module>extensions/spark/kyuubi-spark-lineage</module>
<module>externals/kyuubi-chat-engine</module>
<module>externals/kyuubi-download</module>
Expand Down
Loading