Skip to content
This repository has been archived by the owner on Mar 3, 2023. It is now read-only.

Heron Pom files should have proper dependencies listed. #3774

Closed
joshfischer1108 opened this issue Feb 25, 2022 · 0 comments · Fixed by #3778
Closed

Heron Pom files should have proper dependencies listed. #3774

joshfischer1108 opened this issue Feb 25, 2022 · 0 comments · Fixed by #3778
Assignees
Labels
Apache Release For tasks to meet Apache release requirements

Comments

@joshfischer1108
Copy link
Member

One of the issues was that we are shading dependencies into the jar. We should instead be providing the jar with only our code, and the pom.xml should include the dependencies.

The scripts to generate the pom.xml files:
https://github.com/apache/incubator-heron/tree/master/scripts/ci
And the scripts to build the Heron API jar(s):

# Low Level Api
java_library(
name = "api-java-low-level",
srcs = glob(["org/apache/heron/api/**/*.java"]),
javacopts = DOCLINT_HTML_AND_SYNTAX,
deps = api_deps_files + [
"//third_party/java:kryo-neverlink",
],
)
# Functional Api
java_library(
name = "api-java",
srcs = glob(["org/apache/heron/streamlet/**/*.java"]),
javacopts = DOCLINT_HTML_AND_SYNTAX,
deps = api_deps_files + [
":api-java-low-level",
"@maven//:org_apache_commons_commons_lang3",
],
)
# Low level and functional Api
java_library(
name = "api-java-low-level-functional",
srcs = glob([
"org/apache/heron/api/**/*.java",
"org/apache/heron/streamlet/**/*.java",
]),
javacopts = DOCLINT_HTML_AND_SYNTAX,
deps = api_deps_files + [
"//third_party/java:kryo-neverlink",
"@maven//:org_apache_commons_commons_lang3",
],
)
java_binary(
name = "api-unshaded",
srcs = glob([
"org/apache/heron/api/**/*.java",
"org/apache/heron/streamlet/**/*.java",
]),
deps = api_deps_files + [
"//third_party/java:kryo-neverlink",
"@maven//:org_apache_commons_commons_lang3",
],
)
jar_jar(
name = "api-shaded",
input_jar = ":api-unshaded_deploy.jar",
rules = "shade.conf",
)
genrule(
name = "heron-api",
srcs = [":api-shaded"],
outs = ["heron-api.jar"],
cmd = "cp $< $@",
)
java_library(
name = "classification",
srcs = glob(["org/apache/heron/classification/**/*.java"]),
resources = ["//heron/api/src:processor-service-conf"],
)

"api-java-low-level": Normal "Topology API" jar
"api-java": Functional (i.e. Streamlet jar)
"api-java-low-level-functional": Combination of topology and streamlet code
"api-unshaded": Java Binary (why a binary???) with both, but kryo neverlink
dependency added. I think this might be for Storm compatibility?
"api-shaded": Remaps the protobuf classes based on this rule... but the
rule doesn't shade any of the other dependencies... If we will continue
shading, we should fix this.
https://github.com/apache/incubator-heron/blob/master/heron/api/src/java/shade.conf
"heron-api": We create a copy of the heron-api.jar for some reason... no
idea why this is this way and we don't just rename the previouis build task
"classification": No idea what this is... is it part of the Heron API? Why
is it not included in the resulting "heron-api" jar?

Questions I have:

  1. Do we want to keep shading dependencies, or should we include the
    dependencies in the pom.xml?
  2. What is classification jar?
  3. We are publishing a Heron API jar (that final artifact), but our
    examples are referencing the more immediate streamlet or low-level jars.
    Should the examples reference that final jar?
@joshfischer1108 joshfischer1108 self-assigned this Feb 25, 2022
@joshfischer1108 joshfischer1108 added the Apache Release For tasks to meet Apache release requirements label Feb 25, 2022
@joshfischer1108 joshfischer1108 linked a pull request Feb 26, 2022 that will close this issue
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Apache Release For tasks to meet Apache release requirements
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant