Skip to content

Commit

Permalink
fix
Browse files Browse the repository at this point in the history
  • Loading branch information
@turboFei
turboFei committed Dec 1, 2024
1 parent 34c7b3f commit acc99b1
Show file tree
Hide file tree
Showing 5 changed files with 11 additions and 8 deletions.
2 changes: 2 additions & 0 deletions docs/configuration/settings.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,9 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.batch.application.starvation.timeout | PT3M | Threshold above which to warn batch application may be starved. | duration | 1.7.0 |
| kyuubi.batch.conf.ignore.list || A comma-separated list of ignored keys for batch conf. If the batch conf contains any of them, the key and the corresponding value will be removed silently during batch job submission. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering. You can also pre-define some config for batch job submission with the prefix: kyuubi.batchConf.[batchType]. For example, you can pre-define `spark.master` for the Spark batch job with key `kyuubi.batchConf.spark.spark.master`. | set | 1.6.0 |
| kyuubi.batch.extra.resource.file.max.size | 0 | The maximum size in bytes of each uploaded extra resource file when creating batch. 0 or negative value means no limit. | long | 1.10.0 |
| kyuubi.batch.pending.check.window | PT24H | The window to check the batch pending max elapse time from metadata store. | duration | 1.10.1 |
| kyuubi.batch.resource.file.max.size | 0 | The maximum size in bytes of the uploaded resource file when creating batch. 0 or negative value means no limit. | long | 1.10.0 |
| kyuubi.batch.search.window | PT168H | The window to search the batch from metadata store. | duration | 1.10.1 |
| kyuubi.batch.session.idle.timeout | PT6H | Batch session idle timeout, it will be closed when it's not accessed for this duration | duration | 1.6.2 |

### Credentials
Expand Down
2 changes: 1 addition & 1 deletion docs/monitor/metrics.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ These metrics include:
| `kyuubi.metadata.request.total` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> metadata requests time and rate </div> |
| `kyuubi.metadata.request.failed` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> metadata requests failure time and rate </div> |
| `kyuubi.metadata.request.retrying` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> retrying metadata requests time and rate, it is not cumulative </div> |
| `kyuubi.operartion.batch_pending_time` | | gauge | 1.10.1 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> the batch pending max elapsed time on current kyuubi instance </div> |
| `kyuubi.operartion.batch_pending_max_elapse` | | gauge | 1.10.1 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> the batch pending max elapsed time on current kyuubi instance </div> |

Before v1.5.0, if you use these metrics:
- `kyuubi.statement.total`
Expand Down
6 changes: 2 additions & 4 deletions kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1932,17 +1932,15 @@ object KyuubiConf {

val BATCH_PENDING_CHECK_WINDOW: ConfigEntry[Long] =
buildConf("kyuubi.batch.pending.check.window")
.internal
.doc("The window to check the pending batch.")
.doc("The window to check the batch pending max elapse time from metadata store.")
.version("1.10.1")
.timeConf
.checkValue(_ > 0, "must be positive number")
.createWithDefault(Duration.ofDays(1).toMillis)

val BATCH_SEARCH_WINDOW: ConfigEntry[Long] =
buildConf("kyuubi.batch.search.window")
.internal
.doc("The window to search the batch.")
.doc("The window to search the batch from metadata store.")
.version("1.10.1")
.timeConf
.checkValue(_ > 0, "must be positive number")
Expand Down
2 changes: 1 addition & 1 deletion kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConstants.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ object MetricsConstants {
final val OPERATION_TOTAL: String = OPERATION + "total"
final val OPERATION_STATE: String = OPERATION + "state"
final val OPERATION_EXEC_TIME: String = OPERATION + "exec_time"
final val OPERATION_BATCH_PENDING_TIME: String = OPERATION + "batch_pending_time"
final val OPERATION_BATCH_PENDING_MAX_ELAPSE: String = OPERATION + "batch_pending_max_elapse"

final private val BACKEND_SERVICE = KYUUBI + "backend_service."
final val BS_FETCH_LOG_ROWS_RATE = BACKEND_SERVICE + "fetch_log_rows_rate"
Expand Down
7 changes: 5 additions & 2 deletions kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiRestFrontendService.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ class KyuubiRestFrontendService(override val serverable: Serverable)
}
}

private def getBatchPendingTime(): Long = {
private def getBatchPendingMaxElapse(): Long = {
val filter = MetadataFilter(
state = OperationState.PENDING.toString,
kyuubiInstance = connectionUrl,
Expand Down Expand Up @@ -234,7 +234,10 @@ class KyuubiRestFrontendService(override val serverable: Serverable)
startBatchChecker()
recoverBatchSessions()
MetricsSystem.tracing { ms =>
ms.registerGauge(MetricsConstants.OPERATION_BATCH_PENDING_TIME, getBatchPendingTime, 0)
ms.registerGauge(
MetricsConstants.OPERATION_BATCH_PENDING_MAX_ELAPSE,
getBatchPendingMaxElapse,
0)
}
} catch {
case e: Exception => throw new KyuubiException(s"Cannot start $getName", e)
Expand Down

0 comments on commit acc99b1

Please sign in to comment.