`. | string | 1.6.0 |
+| kyuubi.metadata.store.jdbc.user || The username for server JDBC metadata store. | string | 1.6.0 |
### Metrics
diff --git a/docs/monitor/metrics.md b/docs/monitor/metrics.md
index 561014c370c..8043fa08102 100644
--- a/docs/monitor/metrics.md
+++ b/docs/monitor/metrics.md
@@ -40,56 +40,57 @@ The metrics system is configured via `$KYUUBI_HOME/conf/kyuubi-defaults.conf`.
These metrics include:
-| Metrics Prefix | Metrics Suffix | Type | Since | Description |
-|--------------------------------------------------|----------------------------------------|-----------|-------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `kyuubi.exec.pool.threads.alive` | | gauge | 1.2.0 | threads keepAlive in the backend executive thread pool
|
-| `kyuubi.exec.pool.threads.active` | | gauge | 1.2.0 | threads active in the backend executive thread pool
|
-| `kyuubi.exec.pool.work_queue.size` | | gauge | 1.7.0 | work queue size in the backend executive thread pool
|
-| `kyuubi.connection.total` | | counter | 1.2.0 | cumulative connection count
|
-| `kyuubi.connection.total` | `${sessionType}` | counter | 1.7.0 | cumulative connection count with session type `${sessionType}`
|
-| `kyuubi.connection.opened` | | gauge | 1.2.0 | current active connection count
|
-| `kyuubi.connection.opened` | `${user}` | counter | 1.2.0 | current active connections count requested by a `${user}`
|
-| `kyuubi.connection.opened` | `${user}``${sessionType}` | counter | 1.7.0 | current active connections count requested by a `${user}` with session type `${sessionType}`
|
-| `kyuubi.connection.opened` | `${sessionType}` | counter | 1.7.0 | current active connections count with session type `${sessionType}`
|
-| `kyuubi.connection.failed` | | counter | 1.2.0 | cumulative failed connection count
|
-| `kyuubi.connection.failed` | `${user}` | counter | 1.2.0 | cumulative failed connections for a `${user}`
|
-| `kyuubi.connection.failed` | `${sessionType}` | counter | 1.7.0 | cumulative failed connection count with session type `${sessionType}`
|
-| `kyuubi.operation.total` | | counter | 1.5.0 | cumulative opened operation count
|
-| `kyuubi.operation.total` | `${operationType}` | counter | 1.5.0 | cumulative opened count for the operation `${operationType}`
|
-| `kyuubi.operation.opened` | | gauge | 1.5.0 | current opened operation count
|
-| `kyuubi.operation.opened` | `${operationType}` | counter | 1.5.0 | current opened count for the operation `${operationType}`
|
-| `kyuubi.operation.failed` | `${operationType}`
`.${errorType}` | counter | 1.5.0 | cumulative failed count for the operation `${operationType}` with a particular `${errorType}`, e.g. `execute_statement.AnalysisException`
|
-| `kyuubi.operation.state` | `${operationState}` | meter | 1.5.0 | kyuubi operation state rate
|
-| `kyuubi.operation.exec_time` | `${operationType}` | histogram | 1.7.0 | execution time histogram for the operation `${operationType}`, now only `ExecuteStatement` is enabled.
|
-| `kyuubi.engine.total` | | counter | 1.2.0 | cumulative created engines
|
-| `kyuubi.engine.timeout` | | counter | 1.2.0 | cumulative timeout engines
|
-| `kyuubi.engine.failed` | `${user}` | counter | 1.2.0 | cumulative explicitly failed engine count for a `${user}`
|
-| `kyuubi.engine.failed` | `${errorType}` | counter | 1.2.0 | cumulative explicitly failed engine count for a particular `${errorType}`, e.g. `ClassNotFoundException`
|
-| `kyuubi.backend_service.open_session` | | timer | 1.5.0 | kyuubi backend service `openSession` method execution time and rate
|
-| `kyuubi.backend_service.close_session` | | timer | 1.5.0 | kyuubi backend service `closeSession` method execution time and rate
|
-| `kyuubi.backend_service.get_info` | | timer | 1.5.0 | kyuubi backend service `getInfo` method execution time and rate
|
-| `kyuubi.backend_service.execute_statement` | | timer | 1.5.0 | kyuubi backend service `executeStatement` method execution time and rate
|
-| `kyuubi.backend_service.get_type_info` | | timer | 1.5.0 | kyuubi backend service `getTypeInfo` method execution time and rate
|
-| `kyuubi.backend_service.get_catalogs` | | timer | 1.5.0 | kyuubi backend service `getCatalogs` method execution time and rate
|
-| `kyuubi.backend_service.get_schemas` | | timer | 1.5.0 | kyuubi backend service `getSchemas` method execution time and rate
|
-| `kyuubi.backend_service.get_tables` | | timer | 1.5.0 | kyuubi backend service `getTables` method execution time and rate
|
-| `kyuubi.backend_service.get_table_types` | | timer | 1.5.0 | kyuubi backend service `getTableTypes` method execution time and rate
|
-| `kyuubi.backend_service.get_columns` | | timer | 1.5.0 | kyuubi backend service `getColumns` method execution time and rate
|
-| `kyuubi.backend_service.get_functions` | | timer | 1.5.0 | kyuubi backend service `getFunctions` method execution time and rate
|
-| `kyuubi.backend_service.get_operation_status` | | timer | 1.5.0 | kyuubi backend service `getOperationStatus` method execution time and rate
|
-| `kyuubi.backend_service.cancel_operation` | | timer | 1.5.0 | kyuubi backend service `cancelOperation` method execution time and rate
|
-| `kyuubi.backend_service.close_operation` | | timer | 1.5.0 | kyuubi backend service `closeOperation` method execution time and rate
|
-| `kyuubi.backend_service.get_result_set_metadata` | | timer | 1.5.0 | kyuubi backend service `getResultSetMetadata` method execution time and rate
|
-| `kyuubi.backend_service.fetch_results` | | timer | 1.5.0 | kyuubi backend service `fetchResults` method execution time and rate
|
-| `kyuubi.backend_service.fetch_log_rows_rate` | | meter | 1.5.0 | kyuubi backend service `fetchResults` method that fetch log rows rate
|
-| `kyuubi.backend_service.fetch_result_rows_rate` | | meter | 1.5.0 | kyuubi backend service `fetchResults` method that fetch result rows rate
|
-| `kyuubi.backend_service.get_primary_keys` | | meter | 1.6.0 | kyuubi backend service `get_primary_keys` method execution time and rate
|
-| `kyuubi.backend_service.get_cross_reference` | | meter | 1.6.0 | kyuubi backend service `get_cross_reference` method execution time and rate
|
-| `kyuubi.operation.state` | `${operationType}`
`.${state}` | meter | 1.6.0 | The `${operationType}` with a particular `${state}` rate, e.g. `BatchJobSubmission.pending`, `BatchJobSubmission.finished`. Note that, the terminal states are cumulative, but the intermediate ones are not.
|
-| `kyuubi.metadata.request.opened` | | counter | 1.6.1 | current opened count for the metadata requests
|
-| `kyuubi.metadata.request.total` | | meter | 1.6.0 | metadata requests time and rate
|
-| `kyuubi.metadata.request.failed` | | meter | 1.6.0 | metadata requests failure time and rate
|
-| `kyuubi.metadata.request.retrying` | | meter | 1.6.0 | retrying metadata requests time and rate, it is not cumulative
|
+| Metrics Prefix | Metrics Suffix | Type | Since | Description |
+|--------------------------------------------------|----------------------------------------|-----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `kyuubi.exec.pool.threads.alive` | | gauge | 1.2.0 | threads keepAlive in the backend executive thread pool
|
+| `kyuubi.exec.pool.threads.active` | | gauge | 1.2.0 | threads active in the backend executive thread pool
|
+| `kyuubi.exec.pool.work_queue.size` | | gauge | 1.7.0 | work queue size in the backend executive thread pool
|
+| `kyuubi.connection.total` | | counter | 1.2.0 | cumulative connection count
|
+| `kyuubi.connection.total` | `${sessionType}` | counter | 1.7.0 | cumulative connection count with session type `${sessionType}`
|
+| `kyuubi.connection.opened` | | gauge | 1.2.0 | current active connection count
|
+| `kyuubi.connection.opened` | `${user}` | counter | 1.2.0 | current active connections count requested by a `${user}`
|
+| `kyuubi.connection.opened` | `${user}``${sessionType}` | counter | 1.7.0 | current active connections count requested by a `${user}` with session type `${sessionType}`
|
+| `kyuubi.connection.opened` | `${sessionType}` | counter | 1.7.0 | current active connections count with session type `${sessionType}`
|
+| `kyuubi.connection.failed` | | counter | 1.2.0 | cumulative failed connection count
|
+| `kyuubi.connection.failed` | `${user}` | counter | 1.2.0 | cumulative failed connections for a `${user}`
|
+| `kyuubi.connection.failed` | `${sessionType}` | counter | 1.7.0 | cumulative failed connection count with session type `${sessionType}`
|
+| `kyuubi.operation.total` | | counter | 1.5.0 | cumulative opened operation count
|
+| `kyuubi.operation.total` | `${operationType}` | counter | 1.5.0 | cumulative opened count for the operation `${operationType}`
|
+| `kyuubi.operation.opened` | | gauge | 1.5.0 | current opened operation count
|
+| `kyuubi.operation.opened` | `${operationType}` | counter | 1.5.0 | current opened count for the operation `${operationType}`
|
+| `kyuubi.operation.failed` | `${operationType}`
`.${errorType}` | counter | 1.5.0 | cumulative failed count for the operation `${operationType}` with a particular `${errorType}`, e.g. `execute_statement.AnalysisException`
|
+| `kyuubi.operation.state` | `${operationState}` | meter | 1.5.0 | kyuubi operation state rate
|
+| `kyuubi.operation.exec_time` | `${operationType}` | histogram | 1.7.0 | execution time histogram for the operation `${operationType}`, now only `ExecuteStatement` is enabled.
|
+| `kyuubi.engine.total` | | counter | 1.2.0 | cumulative created engines
|
+| `kyuubi.engine.timeout` | | counter | 1.2.0 | cumulative timeout engines
|
+| `kyuubi.engine.failed` | `${user}` | counter | 1.2.0 | cumulative explicitly failed engine count for a `${user}`
|
+| `kyuubi.engine.failed` | `${errorType}` | counter | 1.2.0 | cumulative explicitly failed engine count for a particular `${errorType}`, e.g. `ClassNotFoundException`
|
+| `kyuubi.backend_service.open_session` | | timer | 1.5.0 | kyuubi backend service `openSession` method execution time and rate
|
+| `kyuubi.backend_service.close_session` | | timer | 1.5.0 | kyuubi backend service `closeSession` method execution time and rate
|
+| `kyuubi.backend_service.get_info` | | timer | 1.5.0 | kyuubi backend service `getInfo` method execution time and rate
|
+| `kyuubi.backend_service.execute_statement` | | timer | 1.5.0 | kyuubi backend service `executeStatement` method execution time and rate
|
+| `kyuubi.backend_service.get_type_info` | | timer | 1.5.0 | kyuubi backend service `getTypeInfo` method execution time and rate
|
+| `kyuubi.backend_service.get_catalogs` | | timer | 1.5.0 | kyuubi backend service `getCatalogs` method execution time and rate
|
+| `kyuubi.backend_service.get_schemas` | | timer | 1.5.0 | kyuubi backend service `getSchemas` method execution time and rate
|
+| `kyuubi.backend_service.get_tables` | | timer | 1.5.0 | kyuubi backend service `getTables` method execution time and rate
|
+| `kyuubi.backend_service.get_table_types` | | timer | 1.5.0 | kyuubi backend service `getTableTypes` method execution time and rate
|
+| `kyuubi.backend_service.get_columns` | | timer | 1.5.0 | kyuubi backend service `getColumns` method execution time and rate
|
+| `kyuubi.backend_service.get_functions` | | timer | 1.5.0 | kyuubi backend service `getFunctions` method execution time and rate
|
+| `kyuubi.backend_service.get_operation_status` | | timer | 1.5.0 | kyuubi backend service `getOperationStatus` method execution time and rate
|
+| `kyuubi.backend_service.cancel_operation` | | timer | 1.5.0 | kyuubi backend service `cancelOperation` method execution time and rate
|
+| `kyuubi.backend_service.close_operation` | | timer | 1.5.0 | kyuubi backend service `closeOperation` method execution time and rate
|
+| `kyuubi.backend_service.get_result_set_metadata` | | timer | 1.5.0 | kyuubi backend service `getResultSetMetadata` method execution time and rate
|
+| `kyuubi.backend_service.fetch_results` | | timer | 1.5.0 | kyuubi backend service `fetchResults` method execution time and rate
|
+| `kyuubi.backend_service.fetch_log_rows_rate` | | meter | 1.5.0 | kyuubi backend service `fetchResults` method that fetch log rows rate
|
+| `kyuubi.backend_service.fetch_result_rows_rate` | | meter | 1.5.0 | kyuubi backend service `fetchResults` method that fetch result rows rate
|
+| `kyuubi.backend_service.get_primary_keys` | | meter | 1.6.0 | kyuubi backend service `get_primary_keys` method execution time and rate
|
+| `kyuubi.backend_service.get_cross_reference` | | meter | 1.6.0 | kyuubi backend service `get_cross_reference` method execution time and rate
|
+| `kyuubi.operation.state` | `${operationType}`
`.${state}` | meter | 1.6.0 | The `${operationType}` with a particular `${state}` rate, e.g. `BatchJobSubmission.pending`, `BatchJobSubmission.finished`. Note that, the terminal states are cumulative, but the intermediate ones are not.
|
+| `kyuubi.metadata.request.opened` | | counter | 1.6.1 | current opened count for the metadata requests
|
+| `kyuubi.metadata.request.total` | | meter | 1.6.0 | metadata requests time and rate
|
+| `kyuubi.metadata.request.failed` | | meter | 1.6.0 | metadata requests failure time and rate
|
+| `kyuubi.metadata.request.retrying` | | meter | 1.6.0 | retrying metadata requests time and rate, it is not cumulative
|
+| `kyuubi.operartion.batch_pending_max_elapse` | | gauge | 1.10.1 | the batch pending max elapsed time on current kyuubi instance
|
Before v1.5.0, if you use these metrics:
- `kyuubi.statement.total`
diff --git a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index e26033bf0f8..a493d7c4578 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -2023,6 +2023,19 @@ object KyuubiConf {
.intConf
.createWithDefault(65536)
+ val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
+ buildConf("kyuubi.metadata.search.window")
+ .doc("The time window to restrict user queries to metadata within a specific period, " +
+ "starting from the current time to the past. It only affects `GET /api/v1/batches` API. " +
+ "You may want to set this to short period to improve query performance and reduce load " +
+ "on the metadata store when administer want to reserve the metadata for long time. " +
+ "The side-effects is that, the metadata created outside the window will not be " +
+ "invisible to users. If it is undefined, all metadata will be visible for users.")
+ .version("1.10.1")
+ .timeConf
+ .checkValue(_ > 0, "must be positive number")
+ .createOptional
+
val ENGINE_EXEC_WAIT_QUEUE_SIZE: ConfigEntry[Int] =
buildConf("kyuubi.backend.engine.exec.pool.wait.queue.size")
.doc("Size of the wait queue for the operation execution thread pool in SQL engine" +
diff --git a/kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala b/kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala
index f615467f3f0..336060e8f70 100644
--- a/kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala
+++ b/kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala
@@ -64,6 +64,7 @@ object MetricsConstants {
final val OPERATION_TOTAL: String = OPERATION + "total"
final val OPERATION_STATE: String = OPERATION + "state"
final val OPERATION_EXEC_TIME: String = OPERATION + "exec_time"
+ final val OPERATION_BATCH_PENDING_MAX_ELAPSE: String = OPERATION + "batch_pending_max_elapse"
final private val BACKEND_SERVICE = KYUUBI + "backend_service."
final val BS_FETCH_LOG_ROWS_RATE = BACKEND_SERVICE + "fetch_log_rows_rate"
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala
index 129fbc8d9aa..aa17c5436a3 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala
@@ -424,6 +424,14 @@ class BatchJobSubmission(
Utils.deleteDirectoryRecursively(session.resourceUploadFolderPath.toFile)
}
}
+
+ def getPendingElapsedTime: Long = {
+ if (state == OperationState.PENDING) {
+ System.currentTimeMillis() - createTime
+ } else {
+ 0L
+ }
+ }
}
object BatchJobSubmission {
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
index 5b8b1686adc..b93dcf7b5b8 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
@@ -31,12 +31,14 @@ import org.eclipse.jetty.servlet.{ErrorPageErrorHandler, FilterHolder}
import org.apache.kyuubi.{KyuubiException, Utils}
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf._
+import org.apache.kyuubi.metrics.MetricsConstants.OPERATION_BATCH_PENDING_MAX_ELAPSE
+import org.apache.kyuubi.metrics.MetricsSystem
import org.apache.kyuubi.server.api.v1.ApiRootResource
import org.apache.kyuubi.server.http.authentication.{AuthenticationFilter, KyuubiHttpAuthenticationFactory}
import org.apache.kyuubi.server.ui.{JettyServer, JettyUtils}
import org.apache.kyuubi.service.{AbstractFrontendService, Serverable, Service, ServiceUtils}
import org.apache.kyuubi.service.authentication.{AuthTypes, AuthUtils}
-import org.apache.kyuubi.session.{KyuubiSessionManager, SessionHandle}
+import org.apache.kyuubi.session.{KyuubiBatchSession, KyuubiSessionManager, SessionHandle}
import org.apache.kyuubi.util.{JavaUtils, ThreadUtils}
import org.apache.kyuubi.util.ThreadUtils.scheduleTolerableRunnableWithFixedDelay
@@ -202,6 +204,14 @@ class KyuubiRestFrontendService(override val serverable: Serverable)
}
}
+ private def getBatchPendingMaxElapse(): Long = {
+ val batchPendingElapseTimes = sessionManager.allSessions().map {
+ case session: KyuubiBatchSession => session.batchJobSubmissionOp.getPendingElapsedTime
+ case _ => 0L
+ }
+ if (batchPendingElapseTimes.isEmpty) 0L else batchPendingElapseTimes.max
+ }
+
def waitForServerStarted(): Unit = {
// block until the HTTP server is started, otherwise, we may get
// the wrong HTTP server port -1
@@ -220,6 +230,9 @@ class KyuubiRestFrontendService(override val serverable: Serverable)
isStarted.set(true)
startBatchChecker()
recoverBatchSessions()
+ MetricsSystem.tracing { ms =>
+ ms.registerGauge(OPERATION_BATCH_PENDING_MAX_ELAPSE, getBatchPendingMaxElapse, 0)
+ }
} catch {
case e: Exception => throw new KyuubiException(s"Cannot start $getName", e)
}
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
index e3e981abdc0..499110cf68c 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
@@ -67,6 +67,7 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
fe.getConf.get(ENGINE_SECURITY_ENABLED)
private lazy val resourceFileMaxSize = fe.getConf.get(BATCH_RESOURCE_FILE_MAX_SIZE)
private lazy val extraResourceFileMaxSize = fe.getConf.get(BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE)
+ private lazy val metadataSearchWindow = fe.getConf.get(METADATA_SEARCH_WINDOW)
private def batchV2Enabled(reqConf: Map[String, String]): Boolean = {
fe.getConf.get(BATCH_SUBMITTER_ENABLED) &&
@@ -420,13 +421,15 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
s"The valid batch state can be one of the following: ${VALID_BATCH_STATES.mkString(",")}")
}
+ val createTimeFilter =
+ math.max(createTime, metadataSearchWindow.map(System.currentTimeMillis() - _).getOrElse(0L))
val filter = MetadataFilter(
sessionType = SessionType.BATCH,
engineType = batchType,
username = batchUser,
state = batchState,
requestName = batchName,
- createTime = createTime,
+ createTime = createTimeFilter,
endTime = endTime)
val batches = sessionManager.getBatchesFromMetadataStore(filter, from, size, desc)
new GetBatchesResponse(from, batches.size, batches.asJava)
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala
index 8d939aefc0c..9ca1948dec6 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala
@@ -139,8 +139,13 @@ class MetadataManager extends AbstractService("MetadataManager") {
from: Int,
size: Int,
desc: Boolean = false): Seq[Batch] = {
- withMetadataRequestMetrics(_metadataStore.getMetadataList(filter, from, size, desc)).map(
- buildBatch)
+ withMetadataRequestMetrics(_metadataStore.getMetadataList(
+ filter,
+ from,
+ size,
+ // if create_file field is set, order by create_time, which is faster, otherwise by key_id
+ orderBy = if (filter.createTime > 0) Some("create_time") else Some("key_id"),
+ direction = if (desc) "DESC" else "ASC")).map(buildBatch)
}
def countBatch(
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataStore.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataStore.scala
index d350050f142..b9fb52f779a 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataStore.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataStore.scala
@@ -58,13 +58,16 @@ trait MetadataStore extends Closeable {
* @param from the metadata offset.
* @param size the size to get.
* @param desc the order of metadata list.
+ * @param orderBy the order by column, default is the auto increment primary key, `key_id`.
+ * @param direction the order direction, default is `ASC`.
* @return selected metadata list.
*/
def getMetadataList(
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata]
+ orderBy: Option[String] = Some("key_id"),
+ direction: String = "ASC"): Seq[Metadata]
/**
* Count the metadata list with filter conditions.
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
index e3db01f87eb..af965a220f6 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala
@@ -257,15 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
filter: MetadataFilter,
from: Int,
size: Int,
- desc: Boolean = false): Seq[Metadata] = {
+ orderBy: Option[String] = Some("key_id"),
+ direction: String = "ASC"): Seq[Metadata] = {
val queryBuilder = new StringBuilder
val params = ListBuffer[Any]()
queryBuilder.append("SELECT ")
queryBuilder.append(METADATA_COLUMNS)
queryBuilder.append(s" FROM $METADATA_TABLE")
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
- queryBuilder.append(" ORDER BY key_id ")
- queryBuilder.append(if (desc) "DESC " else "ASC ")
+ orderBy.foreach(o => queryBuilder.append(s" ORDER BY $o $direction "))
queryBuilder.append(dialect.limitClause(size, from))
val query = queryBuilder.toString
JdbcUtils.withConnection { connection =>
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
index 1a376e93e5a..4d32de70262 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
@@ -486,6 +486,7 @@ abstract class BatchesResourceSuiteBase extends KyuubiFunSuite
.queryParam("from", "0")
.queryParam("size", "1")
.queryParam("desc", "true")
+ .queryParam("createTime", "1")
.request(MediaType.APPLICATION_JSON_TYPE)
.header(AUTHORIZATION_HEADER, basicAuthorizationHeader("anonymous"))
.get()
diff --git a/pom.xml b/pom.xml
index c6055b1b166..73146363243 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1053,7 +1053,20 @@
org.apache.kafka
kafka-clients
${kafka.version}
- true
+
+
+ com.github.luben
+ zstd-jni
+
+
+ org.lz4
+ lz4-java
+
+
+ org.xerial.snappy
+ snappy-java
+
+