[KYUUBI #6411] Grpc common support #6412

davidyuan1223 · 2024-05-23T03:24:42Z

🔍 Description

Issue References 🔗

This pull request fixes #6411

Describe Your Solution 🔧

Add a common module common-grpc to support grpc type fronendservice/backendservice, which can compitable with another grpc engine, not only spark connect

Types of changes 🔖

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklist 📝

This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

1. "$@" is a array, we want use string to compare. so update "$@" => "$*" 2. `tty` mean execute the command, we can use $(tty) replace it 3. param $# is a number, compare number should use -gt/-lt,not >/<

1. "$@" is a array, we want use string to compare. so update "$@" => "$*" 2. `tty` mean execute the command, we can use $(tty) replace it 3. param $# is a number, compare number should use -gt/-lt,not >/< 4. not sure the /bin/kyuubi line 63 'exit -1' need modify? so the directory bin only have a shellcheck note in /bin/kyuubi

# Conflicts: # bin/docker-image-tool.sh

Fix apache#4057

fix_4186

Revert "fix_4186"

fix_4186

# Conflicts: # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkConsoleProgressBar.scala

…ommon-support # Conflicts: # kyuubi-grpc/pom.xml

davidyuan1223 · 2024-05-23T11:32:05Z

@wForget @yaooqinn @ulysses-you This pr is to add common-grpc module based on #6365, i add a simple test case with two simple grpc request/response, what do you think this.

codecov-commenter · 2024-05-23T12:38:21Z

Codecov Report

Attention: Patch coverage is 51.93199% with 311 lines in your changes are missing coverage. Please review.

Project coverage is 58.34%. Comparing base (47a1091) to head (c6af8ed).
Report is 207 commits behind head on master.

Files	Patch %	Lines
...rg/apache/spark/ui/JakartaHttpServletRequest.scala	0.00%	71 Missing ⚠️
.../org/apache/spark/ui/JavaxHttpServletRequest.scala	7.04%	66 Missing ⚠️
...ubi/plugin/spark/authz/serde/tableExtractors.scala	76.28%	20 Missing and 3 partials ⚠️
...g/apache/kyuubi/engine/spark/KyuubiSparkUtil.scala	0.00%	21 Missing ⚠️
.../main/scala/org/apache/spark/ui/SparkUIUtils.scala	62.74%	17 Missing and 2 partials ⚠️
...g/apache/spark/sql/kyuubi/SparkDatasetHelper.scala	56.09%	17 Missing and 1 partial ⚠️
...src/main/scala/org/apache/spark/ui/EngineTab.scala	69.38%	14 Missing and 1 partial ⚠️
...rg/apache/kyuubi/engine/spark/SparkSQLEngine.scala	55.55%	7 Missing and 5 partials ⚠️
...uubi/engine/spark/operation/ExecuteStatement.scala	16.66%	10 Missing ⚠️
.../engine/spark/session/SparkSQLSessionManager.scala	23.07%	9 Missing and 1 partial ⚠️
... and 11 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6412      +/-   ##
============================================
- Coverage     61.19%   58.34%   -2.86%     
- Complexity       23       24       +1     
============================================
  Files           623      656      +33     
  Lines         37060    40283    +3223     
  Branches       5024     5498     +474     
============================================
+ Hits          22680    23504     +824     
- Misses        11948    14271    +2323     
- Partials       2432     2508      +76

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ulysses-you · 2024-05-24T09:10:40Z

Thank you @davidyuan1223 , I think we should make the first commit goal clear. Custom grpc things do not help us support Spark connect (the key make ExecutePlanRequest to be compatible with Kyuubi ?). I think we can add a test: use Spark connect client to query/cancel on a Kyuubi grpc server.

davidyuan1223 · 2024-05-24T09:15:44Z

Thank you @davidyuan1223 , I think we should make the first commit goal clear. Custom grpc things do not help us support Spark connect (the key make ExecutePlanRequest to be compatible with Kyuubi ?). I think we can add a test: use Spark connect client to query/cancel on a Kyuubi grpc server.

You are right, I will add a test case to test spark connect client with kyuubi server

davidyuan1223 · 2024-07-23T16:17:39Z

i'm trying to build a simple case to test spark grpc, but it always report Error

*** RUN ABORTED ***
  java.lang.NoClassDefFoundError: org/sparkproject/guava/util/concurrent/internal/InternalFutureFailureAccess
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
  ...
  Cause: java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.InternalFutureFailureAccess
  at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

I have tried to solve this issue many times by searching for a solution on https://issues.apache.org/jira/browse/SPARK-45201, but it seems futile. Can we focus on discussing the structure instead? We can then use the new module kyuubi-grpc-spark to test this. This bug is quite challenging for me.
@yaooqinn @ulysses-you

ulysses-you · 2024-07-24T01:53:31Z

@davidyuan1223 thank you for the work. Do you require this patch apache/spark#45775 ?

This issue is due to connect required a higher guava version which is different with Spark core itself, but connect depends on the Spark core's shaded guava.

I think this issue has been fixed in Spark3.5.2 which is on RCing, we can use the 3.5.2 rc version to verify it and change to use 3.5.2 after released.

davidyuan1223 · 2024-07-24T02:12:57Z

@davidyuan1223 thank you for the work. Do you require this patch apache/spark#45775 ?

This issue is due to connect required a higher guava version which is different with Spark core itself, but connect depends on the Spark core's shaded guava.

I think this issue has been fixed in Spark3.5.2 which is on RCing, we can use the 3.5.2 rc version to verify it and change to use 3.5.2 after released.

thanks, i will try! The spark-connect package has troubled me for a long time. Its proto classes conflict with connect-common, forcing me to shade it. Then I encountered a problem with Guava, and despite trying many methods, it still didn't work.

ulysses-you · 2024-07-24T02:22:05Z

thank you @davidyuan1223 , if you find any other issues related to Spark, just craete PR to Spark community. We can help review and push it.

github-actions · 2024-11-04T00:24:24Z

Thanks for the PR! This PR is being closed due to inactivity. This isn't a judgement on the merit of the PR in any way. If this is still an issue with the latest version of Kyuubi, please reopen it and ask a committer to remove the Stale tag!

Thank you for using Kyuubi!

davidyuan1223 and others added 23 commits January 13, 2023 16:40

fix-apache#4057 info:

dd39efd

1. "$@" is a array, we want use string to compare. so update "$@" => "$*" 2. `tty` mean execute the command, we can use $(tty) replace it 3. param $# is a number, compare number should use -gt/-lt,not >/<

Merge remote-tracking branch 'origin/fix-#4057' into fix-apache#4057

cb11935

# Conflicts: # bin/docker-image-tool.sh

Merge pull request #10 from xiaoyuandajian/fix-#4057

55a0a43

Fix apache#4057

remove the used blank lines

c48ad38

Merge branch 'apache:master' into master

16237c2

Merge branch 'apache:master' into master

bfa6cbf

Merge branch 'apache:master' into master

e244029

fix_4186

b616044

Merge branch 'master' into fix_4186

360d183

Merge pull request #11 from davidyuan1223/fix_4186

c83836b

fix_4186

Revert "fix_4186"

40e80d9

Merge pull request #12 from davidyuan1223/revert-11-fix_4186

0925a4b

Revert "fix_4186"

Merge branch 'apache:master' into master

2beccb6

Merge branch 'apache:master' into master

c8eb9a2

fix_4186

56b91a3

Merge pull request #13 from davidyuan1223/fix

57ec746

fix_4186

Merge branch 'apache:master' into master

72e7aea

Merge remote-tracking branch 'origin/master'

bcb0cf3

# Conflicts: # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkConsoleProgressBar.scala

add common method to get session level config

8b51840

Merge branch 'apache:master' into master

6376466

Merge branch 'apache:master' into master

46001d4

add new module common-grpc

3d6c53b

github-actions bot added module:common kind:build labels May 23, 2024

davidyuan1223 and others added 5 commits May 23, 2024 11:27

fix_4186

e16fd7f

add common method to get session level config

a3a4ebd

add new module common-grpc

631ac9a

fork master new version

0a52c2a

Merge remote-tracking branch 'origin/grpc-common-support' into grpc-c…

d92fdb6

…ommon-support # Conflicts: # kyuubi-grpc/pom.xml

davidyuan1223 added 3 commits May 23, 2024 11:33

fix format error

4c5a4ae

fix format error

3e302aa

update settings.md

c6af8ed

github-actions bot added the kind:documentation Documentation is a feature! label May 23, 2024

davidyuan1223 and others added 6 commits May 28, 2024 16:32

update

a1e0447

update

5bf1814

update

e031c8c

update

55cbae3

update

0c17121

add new module kyuubi-grpc

2b959f9

tigrulya-exe mentioned this pull request Aug 15, 2024

[TASK][CHALLENGE] Support Spark Connect Frontend/Backend #5383

Open

3 tasks

github-actions bot added the Stale label Nov 4, 2024

github-actions bot closed this Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KYUUBI #6411] Grpc common support #6412

[KYUUBI #6411] Grpc common support #6412

davidyuan1223 commented May 23, 2024 •

edited

Loading

davidyuan1223 commented May 23, 2024

codecov-commenter commented May 23, 2024

ulysses-you commented May 24, 2024

davidyuan1223 commented May 24, 2024

davidyuan1223 commented Jul 23, 2024 •

edited

Loading

ulysses-you commented Jul 24, 2024

davidyuan1223 commented Jul 24, 2024

ulysses-you commented Jul 24, 2024

github-actions bot commented Nov 4, 2024

[KYUUBI #6411] Grpc common support #6412

[KYUUBI #6411] Grpc common support #6412

Conversation

davidyuan1223 commented May 23, 2024 • edited Loading

🔍 Description

Issue References 🔗

Describe Your Solution 🔧

Types of changes 🔖

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklist 📝

davidyuan1223 commented May 23, 2024

codecov-commenter commented May 23, 2024

Codecov Report

ulysses-you commented May 24, 2024

davidyuan1223 commented May 24, 2024

davidyuan1223 commented Jul 23, 2024 • edited Loading

ulysses-you commented Jul 24, 2024

davidyuan1223 commented Jul 24, 2024

ulysses-you commented Jul 24, 2024

github-actions bot commented Nov 4, 2024

davidyuan1223 commented May 23, 2024 •

edited

Loading

davidyuan1223 commented Jul 23, 2024 •

edited

Loading