Skip to content

Commit

Permalink
Merge branch 'master' into KYUUBI-5594
Browse files Browse the repository at this point in the history
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Nov 6, 2023
2 parents 86fc475 + fdacd23 commit c34f32e
Show file tree
Hide file tree
Showing 61 changed files with 1,035 additions and 628 deletions.
28 changes: 21 additions & 7 deletions .github/ISSUE_TEMPLATE/code-contrib-task.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,19 @@ body:
have good logical thinking and the ability to solve complex problems, be proficient in programming
skills or algorithms
- type: dropdown
id: level
attributes:
label: What's the level of this task?
multiple: false
options:
- TRIVIAL
- EASY
- MEDIUM
- CHALLENGE
validations:
required: true

- type: checkboxes
attributes:
label: Code of Conduct
Expand All @@ -65,19 +78,19 @@ body:
description: Mentor is required for MEDIUM and CHALLENGE tasks, to guide contributors to complete the task.
options:
- label: >
I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task
to guide contributors to complete the task.
I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide
contributors through the task.
required: false
- type: textarea
attributes:
label: Skill requirements
description: Which stills are required for contributors who want to take this task?
description: What skills are required for contributors who want to take this task?
placeholder: |
e.g.
- Basic knowledge on Scala Programing Language
- Basic knowledge on Scala programming language
- Familiar with Apache Maven, Docker and GitHub Action
- Basic knowledge on network programing and Apache Thrift RPC framework
- Basic knowledge on network programming and Apache Thrift RPC framework
- Familiar with Apache Spark
- ...
validations:
Expand Down Expand Up @@ -105,8 +118,9 @@ body:
- type: textarea
attributes:
label: Additional context
placeholder: >
Anything else that related to this task that the contributors need to know.
description: Anything else that related to this task that the contributors need to know.
value: |
Introduction of [2023 Kyuubi Code Contribution Program](https://github.com/apache/kyuubi/issues/5357)
validations:
required: false

Expand Down
1 change: 1 addition & 0 deletions docs/client/jdbc/kyuubi_jdbc.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,7 @@ It's straightforward to use principal and keytab for Kerberos authentication, ju
- kyuubiClientPrincipal: Kerberos ``principal`` for client authentication
- kyuubiClientKeytab: path of Kerberos ``keytab`` file for client authentication
- kyuubiClientTicketCache: path of Kerberos ``ticketCache`` file for client authentication, available since 1.8.0.
- kyuubiServerPrincipal: Kerberos ``principal`` configured by `kyuubi.kinit.principal` at the server side. ``kyuubiServerPrincipal`` is available
as an alias of ``principal`` since 1.7.0, use ``principal`` for previous versions.

Expand Down
32 changes: 16 additions & 16 deletions docs/configuration/settings.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -387,22 +387,22 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co

### Server

| Key | Default | Meaning | Type | Since |
|----------------------------------------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.server.administrators || Comma-separated list of Kyuubi service administrators. We use this config to grant admin permission to any service accounts. | set | 1.8.0 |
| kyuubi.server.info.provider | ENGINE | The server information provider name, some clients may rely on this information to check the server compatibilities and functionalities. <li>SERVER: Return Kyuubi server information.</li> <li>ENGINE: Return Kyuubi engine information.</li> | string | 1.6.1 |
| kyuubi.server.limit.batch.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.limit.batch.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server batch connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.limit.batch.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.limit.client.fetch.max.rows | &lt;undefined&gt; | Max rows limit for getting result row set operation. If the max rows specified by client-side is larger than the limit, request will fail directly. | int | 1.8.0 |
| kyuubi.server.limit.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.user.deny.list || The user in the deny list will be denied to connect to kyuubi server, if the user has configured both user.unlimited.list and user.deny.list, the priority of the latter is higher. | set | 1.8.0 |
| kyuubi.server.limit.connections.user.unlimited.list || The maximum connections of the user in the white list will not be limited. | set | 1.7.0 |
| kyuubi.server.name | &lt;undefined&gt; | The name of Kyuubi Server. | string | 1.5.0 |
| kyuubi.server.periodicGC.interval | PT30M | How often to trigger a garbage collection. | duration | 1.7.0 |
| kyuubi.server.redaction.regex | &lt;undefined&gt; | Regex to decide which Kyuubi contain sensitive information. When this regex matches a property key or value, the value is redacted from the various logs. || 1.6.0 |
| Key | Default | Meaning | Type | Since |
|----------------------------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.server.administrators || Comma-separated list of Kyuubi service administrators. We use this config to grant admin permission to any service accounts when security mechanism is enabled. Note, when kyuubi.authentication is configured to NOSASL or NONE, everyone is treated as administrator. | set | 1.8.0 |
| kyuubi.server.info.provider | ENGINE | The server information provider name, some clients may rely on this information to check the server compatibilities and functionalities. <li>SERVER: Return Kyuubi server information.</li> <li>ENGINE: Return Kyuubi engine information.</li> | string | 1.6.1 |
| kyuubi.server.limit.batch.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.limit.batch.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server batch connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.limit.batch.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.limit.client.fetch.max.rows | &lt;undefined&gt; | Max rows limit for getting result row set operation. If the max rows specified by client-side is larger than the limit, request will fail directly. | int | 1.8.0 |
| kyuubi.server.limit.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.user.deny.list || The user in the deny list will be denied to connect to kyuubi server, if the user has configured both user.unlimited.list and user.deny.list, the priority of the latter is higher. | set | 1.8.0 |
| kyuubi.server.limit.connections.user.unlimited.list || The maximum connections of the user in the white list will not be limited. | set | 1.7.0 |
| kyuubi.server.name | &lt;undefined&gt; | The name of Kyuubi Server. | string | 1.5.0 |
| kyuubi.server.periodicGC.interval | PT30M | How often to trigger a garbage collection. | duration | 1.7.0 |
| kyuubi.server.redaction.regex | &lt;undefined&gt; | Regex to decide which Kyuubi contain sensitive information. When this regex matches a property key or value, the value is redacted from the various logs. || 1.6.0 |

### Session

Expand Down
2 changes: 0 additions & 2 deletions docs/tools/kyuubi-admin.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,6 @@ Usage: ``bin/kyuubi-admin list engine [options]``
- The subdomain for the share level of an engine. If not specified, it will read the configuration item kyuubi.engine.share.level.subdomain from kyuubi-defaults.conf.
* - --hs2ProxyUser
- The proxy user to impersonate. When specified, it will list engines for the hs2ProxyUser.
* - -a --all
- All the engine.

.. _list_server:

Expand Down
8 changes: 7 additions & 1 deletion ...xtension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import scala.annotation.tailrec
import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeSet, Expression, NamedExpression, UnaryExpression}
import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys
import org.apache.spark.sql.catalyst.plans.{FullOuter, Inner, LeftAnti, LeftOuter, LeftSemi, RightOuter}
import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, LogicalPlan, Project, Sort, SubqueryAlias, View}
import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Generate, LogicalPlan, Project, Sort, SubqueryAlias, View, Window}

/**
* Infer the columns for Rebalance and Sort to improve the compression ratio.
Expand Down Expand Up @@ -96,6 +96,12 @@ object InferRebalanceAndSortOrders {
case f: Filter => candidateKeys(f.child, output)
case s: SubqueryAlias => candidateKeys(s.child, output)
case v: View => candidateKeys(v.child, output)
case g: Generate => candidateKeys(g.child, AttributeSet(g.requiredChildOutput))
case w: Window =>
val aliasMap = getAliasMap(w.windowExpressions)
Some((
w.partitionSpec.map(p => aliasMap.getOrElse(p.canonicalized, p)),
w.orderSpec.map(_.child).map(o => aliasMap.getOrElse(o.canonicalized, o))))

case _ => None
}
Expand Down
27 changes: 26 additions & 1 deletion ...extension-spark-3-5/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -199,9 +199,10 @@ class RebalanceBeforeWritingSuite extends KyuubiSparkSQLExtensionTest {
}

withView("v") {
withTable("t", "input1", "input2") {
withTable("t", "t2", "input1", "input2") {
withSQLConf(KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS.key -> "true") {
sql(s"CREATE TABLE t (c1 int, c2 long) USING PARQUET PARTITIONED BY (p string)")
sql(s"CREATE TABLE t2 (c1 int, c2 long, c3 long) USING PARQUET PARTITIONED BY (p string)")
sql(s"CREATE TABLE input1 USING PARQUET AS SELECT * FROM VALUES(1,2),(1,3)")
sql(s"CREATE TABLE input2 USING PARQUET AS SELECT * FROM VALUES(1,3),(1,3)")
sql(s"CREATE VIEW v as SELECT col1, count(*) as col2 FROM input1 GROUP BY col1")
Expand Down Expand Up @@ -264,6 +265,30 @@ class RebalanceBeforeWritingSuite extends KyuubiSparkSQLExtensionTest {
|SELECT * FROM v
|""".stripMargin)
checkShuffleAndSort(df5.queryExecution.analyzed, 1, 1)

// generate
val df6 = sql(
s"""
|INSERT INTO TABLE t2 PARTITION(p='a')
|SELECT /*+ broadcast(input2) */ input1.col1, input2.col1, cast(cc.action1 as bigint)
|FROM input1
|JOIN input2
|ON input1.col1 = input2.col1
| lateral view explode(ARRAY(input1.col1, input1.col2)) cc as action1
|""".stripMargin)
checkShuffleAndSort(df6.queryExecution.analyzed, 1, 1)

// window
val df7 = sql(
s"""
|INSERT INTO TABLE t2 PARTITION(p='a')
|SELECT /*+ broadcast(input2) */ input1.col1, input2.col2,
| RANK() OVER (PARTITION BY input2.col2 ORDER BY input1.col1) AS rank
|FROM input1
|JOIN input2
|ON input1.col1 = input2.col1
|""".stripMargin)
checkShuffleAndSort(df7.queryExecution.analyzed, 1, 1)
}
}
}
Expand Down
1 change: 1 addition & 0 deletions ...ain/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableExtractor
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,6 @@ org.apache.kyuubi.plugin.spark.authz.serde.ResolvedDbObjectNameTableExtractor
org.apache.kyuubi.plugin.spark.authz.serde.ResolvedIdentifierTableExtractor
org.apache.kyuubi.plugin.spark.authz.serde.ResolvedTableTableExtractor
org.apache.kyuubi.plugin.spark.authz.serde.StringTableExtractor
org.apache.kyuubi.plugin.spark.authz.serde.SubqueryAliasTableExtractor
org.apache.kyuubi.plugin.spark.authz.serde.TableIdentifierTableExtractor
org.apache.kyuubi.plugin.spark.authz.serde.TableTableExtractor
2 changes: 2 additions & 0 deletions .../main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.URIExtractor
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,6 @@
# limitations under the License.
#

org.apache.kyuubi.plugin.spark.authz.serde.CatalogStorageFormatURIExtractor
org.apache.kyuubi.plugin.spark.authz.serde.OptionsUriExtractor
org.apache.kyuubi.plugin.spark.authz.serde.StringURIExtractor
Loading

0 comments on commit c34f32e

Please sign in to comment.