HBASE-27340 Artifacts with resolved profiles #4740
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Apache9
reviewed
Aug 28, 2022
| @@ -154,11 +154,6 @@ | |||
| <type>test-jar</type> | |||
| <scope>test</scope> | |||
| </dependency> | |||
| <dependency> | |||
There was a problem hiding this comment.
This is for moving all the hadoop related dependencies to hadoop-2.0 profile or hadoop-3.0 profile?
hbase-mapreduce/pom.xml
Outdated
| </dependencies> | ||
| </profile> | ||
| <!-- | ||
| profile for building against Hadoop 3.0.x. Activate using: |
There was a problem hiding this comment.
'Hadoop 3.0.x' -> 'Hadoop 3.x'
| </dependencies> | ||
| </profile> | ||
| <!-- | ||
| profile for building against Hadoop 3.0.0. Activate using: |
There was a problem hiding this comment.
Will fix in next push...
pom.xml
Outdated
| <plugin> | ||
| <groupId>org.codehaus.mojo</groupId> | ||
| <artifactId>flatten-maven-plugin</artifactId> | ||
| <!--<version>1.3.0</version>--> |
There was a problem hiding this comment.
This is because the version has been declared in the parent apache pom?
There was a problem hiding this comment.
Use 'latest'. Will set it to 1.3.0 explicitly next go around.
There was a problem hiding this comment.
I have mixed feelings about manually managing plugin versions but agree it is best on balance. In our own internal builds we need to manage things up sometimes and the version property definitions for plugin versions are helpful for that. Other downstreamers will have a similar experience I suspect.
|
💔 -1 overall
This message was automatically generated. |
apurtell
approved these changes
Aug 30, 2022
|
Approved pending clean spotless check and successful test run. Current failures are relevant to this change e.g. |
|
Let me fix the failure. For master branch, +1 on removing hadoop profiles, etc. but I'd think we'd still want this patch too because hadoop is not only profile and even if no profiles, the published poms are slim w/ runtime-only dependencies w/ explicit versions filled in. |
+1 |
|
💔 -1 overall
This message was automatically generated. |
1 similar comment
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
saintstack
force-pushed
the
HBASE-27340
branch
from
4fc4008 to
5738345
Compare
|
RELEASENOTE changes come of my running spotless locally. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
saintstack
force-pushed
the
HBASE-27340
branch
from
5738345 to
38b2347
Compare
|
💔 -1 overall
This message was automatically generated. |
1 similar comment
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
saintstack
force-pushed
the
HBASE-27340
branch
from
38b2347 to
a106ae2
Compare
|
💔 -1 overall
This message was automatically generated. |
1 similar comment
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Make it so our published poms carry the minimum needed to run
an hbase; the published pom has no profiles -- the profiles
specified at build time are resolved, their dependencies inlined,
and then they are stripped -- and no build-time, or plugins
dependencies or properties, etc. Resultant poms have explicit
hadoop lib versions baked in -- no more being able to choose
hbase with hadoop2 or haddop3 at downstream build time by setting
a '-Dhadoop.profile=X.0'.
Pattern is to add profiles when none in sub-modules when
the flatten plugin complains it can't resolve an hadoop
dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
Adding the hadoop-2.0 and hadoop-3.0 profiles in the sub-module
make it so the flatten plugin can figure 'hadoop.version'
definitively.
Another spin on the above happens when profiles already exist
in submodule but the flatten plugin is complaining it can't
figure figure version on an hadoop dependency NOT under
profiles. Below, we move the delinquent hadoop dependency under
existing profiles (minikdc was the usual dependency outside
profiles in sub-modules that flatten complained about).
Sometimes, moving an hadoop dependency under a profile, there
would be excludes on the local dependency. If the parent pom
excludes section was missing the local excludes, we added them
up to the parent module so all excluding is done up there in
the parent profile dependencyManagement section.
* hbase-asyncfs/pom.xml
* hbase-endpoint/pom.xml
* hbase-examples/pom.xml
* hbase-http/pom.xml
* hbase-rest/pom.xml
* hbase-server/pom.xml
Move the minikdc under profiles so it picks up appropriate hadoop version
when the flatten plugin runs.
* hbase-hadoop2-compat/pom.xml
Add hadoop2 and hadoop3 profiles and move hadoop-common, etc.
under them so we pick up appropriate hadoop version when flatten
plugin runs.
* hbase-mapreduce/pom.xml
Move hadoop dependencies under profiles so right version is
available when the flatten plugin runs.
* hbase-shaded/hbase-shaded-testing-util/pom.xml
Add profiles for hadoop-2.0 and hadoop-3.0 and move the
hadoop dependencies under them.
pom.xml
Add the flatten plugin with the flatten profiles enabled.
Add a few excludes on hadoop profiles picked up from sub-modules.
E.g. exclude bouncycastle bcprov-jdk15 when we include minikdc.
saintstack
force-pushed
the
HBASE-27340
branch
from
a106ae2 to
11ebb74
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Apache9
approved these changes
Sep 2, 2022
saintstack
added a commit
that referenced
this pull request
Sep 2, 2022
Make it so our published poms carry the minimum needed to run
an hbase; the published pom has no profiles -- the profiles
specified at build time are resolved, their dependencies inlined,
and then they are stripped -- and no build-time, or plugins
dependencies or properties, etc. Resultant poms have explicit
hadoop lib versions baked in -- no more being able to choose
hbase with hadoop2 or haddop3 at downstream build time by setting
a '-Dhadoop.profile=X.0'.
Pattern is to add profiles when none in sub-modules when
the flatten plugin complains it can't resolve an hadoop
dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
Adding the hadoop-2.0 and hadoop-3.0 profiles in the sub-module
make it so the flatten plugin can figure 'hadoop.version'
definitively.
Another spin on the above happens when profiles already exist
in submodule but the flatten plugin is complaining it can't
figure figure version on an hadoop dependency NOT under
profiles. Below, we move the delinquent hadoop dependency under
existing profiles (minikdc was the usual dependency outside
profiles in sub-modules that flatten complained about).
Sometimes, moving an hadoop dependency under a profile, there
would be excludes on the local dependency. If the parent pom
excludes section was missing the local excludes, we added them
up to the parent module so all excluding is done up there in
the parent profile dependencyManagement section.
* hbase-asyncfs/pom.xml
* hbase-endpoint/pom.xml
* hbase-examples/pom.xml
* hbase-http/pom.xml
* hbase-rest/pom.xml
* hbase-server/pom.xml
Move the minikdc under profiles so it picks up appropriate hadoop version
when the flatten plugin runs.
* hbase-hadoop2-compat/pom.xml
Add hadoop2 and hadoop3 profiles and move hadoop-common, etc.
under them so we pick up appropriate hadoop version when flatten
plugin runs.
* hbase-mapreduce/pom.xml
Move hadoop dependencies under profiles so right version is
available when the flatten plugin runs.
* hbase-shaded/hbase-shaded-testing-util/pom.xml
Add profiles for hadoop-2.0 and hadoop-3.0 and move the
hadoop dependencies under them.
pom.xml
Add the flatten plugin with the flatten profiles enabled.
Add a few excludes on hadoop profiles picked up from sub-modules.
E.g. exclude bouncycastle bcprov-jdk15 when we include minikdc.
Signed-off-by: Andrew Purtell <[email protected]>
Signed-off-by: Duo Zhang <[email protected]>
saintstack
added a commit
to saintstack/hbase
that referenced
this pull request
Sep 3, 2022
(Forward port from branch-2; simplified by the fact that there
is no hadoop-2.0 profile on master branch)
Make it so our published poms carry the minimum needed to run
an hbase; the published pom has no profiles -- the profiles
specified at build time are resolved, their dependencies inlined,
and then they are stripped -- and no build-time, or plugins
dependencies or properties, etc. Resultant poms have explicit
hadoop lib versions baked in -- no more being able to choose
hbase with hadoop2 or haddop3 at downstream build time by setting
a '-Dhadoop.profile=X.0'.
Pattern is to add profiles when none in sub-modules when
the flatten plugin complains it can't resolve an hadoop
dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
Adding the profile in the sub-module make it so the flatten
plugin can figure 'hadoop.version' definitively.
(In master there is only the hadoop-3.0 profile).
Another spin on the above happens when profiles already exist
in submodule but the flatten plugin is complaining it can't
figure figure version on an hadoop dependency NOT under
profiles. Below, we move the delinquent hadoop dependency under
existing profiles (minikdc was the usual dependency outside
profiles in sub-modules that flatten complained about).
Sometimes, moving an hadoop dependency under a profile, there
would be excludes on the local dependency. If the parent pom
excludes section was missing the local excludes, we added them
up to the parent module so all excluding is done up there in
the parent profile dependencyManagement section.
Apache9
pushed a commit
that referenced
this pull request
Sep 6, 2022
(Forward port from branch-2; simplified by the fact that there
is no hadoop-2.0 profile on master branch)
Make it so our published poms carry the minimum needed to run
an hbase; the published pom has no profiles -- the profiles
specified at build time are resolved, their dependencies inlined,
and then they are stripped -- and no build-time, or plugins
dependencies or properties, etc. Resultant poms have explicit
hadoop lib versions baked in -- no more being able to choose
hbase with hadoop2 or haddop3 at downstream build time by setting
a '-Dhadoop.profile=X.0'.
Pattern is to add profiles when none in sub-modules when
the flatten plugin complains it can't resolve an hadoop
dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
Adding the profile in the sub-module make it so the flatten
plugin can figure 'hadoop.version' definitively.
(In master there is only the hadoop-3.0 profile).
Another spin on the above happens when profiles already exist
in submodule but the flatten plugin is complaining it can't
figure figure version on an hadoop dependency NOT under
profiles. Below, we move the delinquent hadoop dependency under
existing profiles (minikdc was the usual dependency outside
profiles in sub-modules that flatten complained about).
Sometimes, moving an hadoop dependency under a profile, there
would be excludes on the local dependency. If the parent pom
excludes section was missing the local excludes, we added them
up to the parent module so all excluding is done up there in
the parent profile dependencyManagement section.
Signed-off-by: Duo Zhang <[email protected]>
ndimiduk
reviewed
Sep 27, 2022
| <id>hadoop-2.0</id> | ||
| <activation> | ||
| <property> | ||
| <!--Below formatting for dev-support/generate-hadoopX-poms.sh--> |
There was a problem hiding this comment.
Heya @saintstack -- this comment is about some support scripts, but I don't see them in the commit. Maybe something you intended to commit with this patch but missed the git add ? Thanks boss.
There was a problem hiding this comment.
Thanks for taking a looksee @ndimiduk
It looks like this is a blind copy/paste of hadoop-2.0 profile activation snippet. I see we are repeating the comment in lots of places (See below). The comment is way stale referring to a script that was removed in 2014.
commit 37654f77a60ad0c13a84d6d6b39657789726df09
Author: Michael Stack <[email protected]>
Date: Wed Mar 19 23:08:55 2014 +0000
HBASE-10781 Remove hadoop-one-compat module and all references to hadoop1
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1579449 13f79535-47bb-0310-9956-ffa450edef68
Here is grep for comment over code base:
$ grep -r 'Below formatt' .
./hbase-client/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-asyncfs/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-replication/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-hadoop2-compat/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-zookeeper/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-thrift/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-common/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-examples/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-procedure/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-endpoint/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-testing-util/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-server/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-rest/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-shaded/hbase-shaded-testing-util/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-shaded/hbase-shaded-mapreduce/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
....
Let me file an issue to clean out the comment... HBASE-27396
There was a problem hiding this comment.
Oh okay. Here I was hoping that there was some additional secret sauce I was missing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make it so our published poms carry the minimum needed to run
an hbase; the published pom has no profiles -- the profiles
specified at build time are resolved, their dependencies inlined,
and then they are stripped -- and no build-time, or plugins
dependencies or properties, etc. Resultant poms have explicit
hadoop lib versions baked in -- no more being able to choose
hbase with hadoop2 or haddop3 at downstream build time by setting
a '-Dhadoop.profile=X.0'.
Pattern is to add profiles when none in sub-modules when
the flatten plugin complains it can't resolve an hadoop
dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
Adding the hadoop-2.0 and hadoop-3.0 profiles in the sub-module
make it so the flatten plugin can figure 'hadoop.version'
definitively.
Another spin on the above happens when profiles already exist
in submodule but the flatten plugin is complaining it can't
figure figure version on an hadoop dependency NOT under
profiles. Below, we move the delinquent hadoop dependency under
existing profiles (minikdc was the usual dependency outside
profiles in sub-modules that flatten complained about).
Sometimes, moving an hadoop dependency under a profile, there
would be excludes on the local dependency. If the parent pom
excludes section was missing the local excludes, we added them
up to the parent module so all excluding is done up there in
the parent profile dependencyManagement section.
hbase-asyncfs/pom.xml
hbase-endpoint/pom.xml
hbase-examples/pom.xml
hbase-http/pom.xml
hbase-rest/pom.xml
hbase-server/pom.xml
Move the minikdc under profiles so it picks up appropriate hadoop version
when the flatten plugin runs.
hbase-hadoop2-compat/pom.xml
Add hadoop2 and hadoop3 profiles and move hadoop-common, etc.
under them so we pick up appropriate hadoop version when flatten
plugin runs.
hbase-mapreduce/pom.xml
Move hadoop dependencies under profiles so right version is
available when the flatten plugin runs.
hbase-shaded/hbase-shaded-testing-util/pom.xml
Add profiles for hadoop-2.0 and hadoop-3.0 and move the
hadoop dependencies under them.
pom.xml
Add the flatten plugin with the flatten profiles enabled.
Add a few excludes on hadoop profiles picked up from sub-modules.
E.g. exclude bouncycastle bcprov-jdk15 when we include minikdc.