[#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector #4791
Conversation
coolderli
force-pushed
the
add-flink-connector-runtime
branch
from
3017555 to
d591c73
Compare
coolderli
changed the title
|
@FANNG1 PTAL. Thanks. |
|
Hi @coolderli without this runtime jar, how do we run Flink Gravitino connector previously? |
@jerryshao Just use the flink connector jar. But there are some conflicts here. Therefore, a shaded operation was performed in the runtime. |
| @@ -23,7 +23,7 @@ This capability allows users to perform federation queries, accessing data from | |||
|
|
|||
| ## How to use it | |||
|
|
|||
| 1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector jar, and place it to the classpath of Flink. | |||
| 1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector runtime jar, and place it to the classpath of Flink. | |||
There was a problem hiding this comment.
@yuqi please help to verify that the flink runtime jar link will be https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector-runtime ?
There was a problem hiding this comment.
There is no artifact named gravitino-flink-connector-runtime, only gravitino-flink-connector-runtime-1.18.0_2.12 and gravitino-flink_2.12.
There was a problem hiding this comment.
so the download link should be https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector-runtime-1.18.0_2.12?
There was a problem hiding this comment.
we should use 1.18 not 1.18.0 here? and link should be https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector-runtime-1.18 like spark not include scala versions.
val flinkMajorVersion: String = flink.substringBeforeLast(".")
There was a problem hiding this comment.
@FANNG1 But I saw the spark has the scala version(_2.12).
There was a problem hiding this comment.
I mean the runtime jar download link doesn't contain scala version
There was a problem hiding this comment.
|
LGTM, except minor comments, @coolderli could you check it again? |
|
LGTM |
| implementation(project(":api")) | ||
| implementation(project(":catalogs:catalog-common")) | ||
| implementation(project(":common")) | ||
| implementation(project(":core")) |
There was a problem hiding this comment.
you can add { exclude("*") } to shrink the size of lib
There was a problem hiding this comment.
I am confused why you need to depend on core module?
There was a problem hiding this comment.
@mchades could you create a separate issue to track this?
There was a problem hiding this comment.
@mchades could you create a separate issue to track this?
If this is just a minor fix, it is recommended to fix it in this PR.
There was a problem hiding this comment.
I'm afraid there maybe something unexpected, I will merge this PR to kick off 0.6.0
…build flink connector (#4791) ### What changes were proposed in this pull request? - add flink runtime connector ### Why are the changes needed? Fix: #3514 ### Does this PR introduce _any_ user-facing change? - no ### How was this patch tested? - local test --------- Co-authored-by: fanng <[email protected]>
|
@coolderli , thanks for your work! |
…ime lib size from 84MB to 16MB (#4819) ### What changes were proposed in this pull request? - shrink the flink-connector-runtime jar. - Change client-java scope from implementation to CompileOnly in flink module. - use the client-java-runtime in flink-connector-runtime. ### Why are the changes needed? Fix: #5145 - The flink-connector-runtime jar is much big as mentioned in #4791 (comment). ### Does this PR introduce _any_ user-facing change? - no ### How was this patch tested? - local test
What changes were proposed in this pull request?
Why are the changes needed?
Fix: #3514
Does this PR introduce any user-facing change?
How was this patch tested?