Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up GetTables operation for Spark session catalog #6018

Closed
wants to merge 3 commits into from
Closed

Speed up GetTables operation for Spark session catalog #6018

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
615b747
Speed up GetTables operation
pan3793 Jan 25, 2024
405b124
fix
pan3793 Jan 25, 2024
058001c
fix
pan3793 Jan 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/configuration/settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,7 +392,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co

| Key | Default | Meaning | Type | Since |
|--------------------------------------------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.operation.getTables.ignoreTableProperties | false | Speed up the `GetTables` operation by returning table identities only. | boolean | 1.8.0 |
| kyuubi.operation.getTables.ignoreTableProperties | false | Speed up the `GetTables` operation by ignoring `tableTypes` query criteria, and returning table identities only. | boolean | 1.8.0 |
| kyuubi.operation.idle.timeout | PT3H | Operation will be closed when it's not accessed for this duration of time | duration | 1.0.0 |
| kyuubi.operation.interrupt.on.cancel | true | When true, all running tasks will be interrupted if one cancels a query. When false, all running tasks will remain until finished. | boolean | 1.2.0 |
| kyuubi.operation.language | SQL | Choose a programing language for the following inputs<ul><li>SQL: (Default) Run all following statements as SQL queries.</li><li>SCALA: Run all following input as scala codes</li><li>PYTHON: (Experimental) Run all following input as Python codes with Spark engine</li></ul> | string | 1.5.0 |
Expand Down
37 changes: 27 additions & 10 deletions ...ark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/util/SparkCatalogUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,31 +163,48 @@ object SparkCatalogUtils extends Logging {
val namespaces = listNamespacesWithPattern(catalog, schemaPattern)
catalog match {
case builtin if builtin.name() == SESSION_CATALOG =>
val catalog = spark.sessionState.catalog
val databases = catalog.listDatabases(schemaPattern)
val sessionCatalog = spark.sessionState.catalog
val databases = sessionCatalog.listDatabases(schemaPattern)

def isMatchedTableType(tableTypes: Set[String], tableType: String): Boolean = {
val typ = if (tableType.equalsIgnoreCase(VIEW)) VIEW else TABLE
tableTypes.exists(typ.equalsIgnoreCase)
}

databases.flatMap { db =>
val identifiers = catalog.listTables(db, tablePattern, includeLocalTempViews = false)
catalog.getTablesByName(identifiers)
.filter(t => isMatchedTableType(tableTypes, t.tableType.name)).map { t =>
val typ = if (t.tableType.name == VIEW) VIEW else TABLE
val identifiers =
sessionCatalog.listTables(db, tablePattern, includeLocalTempViews = false)
if (ignoreTableProperties) {
identifiers.map { ti: TableIdentifier =>
Row(
catalogName,
t.database,
t.identifier.table,
typ,
t.comment.getOrElse(""),
ti.database.getOrElse("default"),
ti.table,
TABLE, // ignore tableTypes criteria and simply treat all table type as TABLE
"",
null,
null,
null,
null,
null)
}
} else {
sessionCatalog.getTablesByName(identifiers)
.filter(t => isMatchedTableType(tableTypes, t.tableType.name)).map { t =>
val typ = if (t.tableType.name == VIEW) VIEW else TABLE
Row(
catalogName,
t.database,
t.identifier.table,
typ,
t.comment.getOrElse(""),
null,
null,
null,
null,
null)
}
}
}
case tc: TableCatalog =>
val tp = tablePattern.r.pattern
Expand Down
3 changes: 2 additions & 1 deletion kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3433,7 +3433,8 @@ object KyuubiConf {

val OPERATION_GET_TABLES_IGNORE_TABLE_PROPERTIES: ConfigEntry[Boolean] =
buildConf("kyuubi.operation.getTables.ignoreTableProperties")
.doc("Speed up the `GetTables` operation by returning table identities only.")
.doc("Speed up the `GetTables` operation by ignoring `tableTypes` query criteria, " +
"and returning table identities only.")
.version("1.8.0")
.booleanConf
.createWithDefault(false)
Expand Down
Loading