GH-38264: [Java][Packaging] Add BOM file #38336

jduo · 2023-10-18T23:08:04Z

Rationale for this change

Provide an easy way for users to import Arrow maven packages.

What changes are included in this PR?

Add a BOM module and utilize this module internally for importing Arrow dependencies.

Are these changes tested?

Yes.

Are there any user-facing changes?

Users can import the BOM dependency now.

Closes: [Java][Packaging] Add an Arrow BOM #38264

jduo · 2023-10-18T23:08:25Z

@danepitkin

danepitkin · 2023-10-19T02:25:35Z

Thanks @jduo !

I'm not a BOM file expert, does it make sense to also pin Arrow's 3rd party deps in here as well? And is it possible to pin 3rd party deps based on JDK version? (Java 8 support requires pinning some deps to outdated versions e.g. Mockito). Maybe a follow up PR would be better to add this, though.

jduo · 2023-10-19T13:49:55Z

Thanks @jduo !

I'm not a BOM file expert, does it make sense to also pin Arrow's 3rd party deps in here as well? And is it possible to pin 3rd party deps based on JDK version? (Java 8 support requires pinning some deps to outdated versions e.g. Mockito). Maybe a follow up PR would be better to add this, though.

I took a look at BOMs for a few projects:

Jackson
Spring
Beam
gRPC

and they all only included internally-built modules in their BOM files rather than also including 3rd party dependencies. This sort of makes sense though, if the BOM is intended to describe what libraries are being shipped.

danepitkin · 2023-10-19T15:05:50Z

Makes sense to me, I'm good with the PR as-is then. LGTM. Thanks!

danepitkin · 2023-10-19T15:09:22Z

The release script is failing though. I'm not quite sure what the issue is yet. Seems to be complaining about 14.0.0-SNAPSHOT vs 14.0.0?

jduo · 2023-10-19T15:45:25Z

The release script is failing though. I'm not quite sure what the issue is yet. Seems to be complaining about 14.0.0-SNAPSHOT vs 14.0.0?

Perhaps there's a list of files that a versioning script updates that needs to be updated to include the BOM pom.

jduo · 2023-10-19T17:21:04Z

The release script is failing though. I'm not quite sure what the issue is yet. Seems to be complaining about 14.0.0-SNAPSHOT vs 14.0.0?

Perhaps there's a list of files that a versioning script updates that needs to be updated to include the BOM pom.

I see the problem. In utils-prepare.sh at https://github.com/jduo/arrow/blob/4c329d0da2ec5ce7d24134ac568ec40619358c08/dev/release/utils-prepare.sh#L68C3-L68C20 , the script uses mvn versions:set on the parent pom, which cascades to all child modules. However the Bill of Materials POM is explicitly not a child, but rather a side-module that the parent imports.

The solution is either to run mvn versions:set -DprocessAllModules which includes updating any aggregated modules, or to simply also run against the BOM file directly.

lidavidm · 2023-10-19T17:59:38Z

New packages also need to be added somewhere in ci/ so that they actually get released

lidavidm · 2023-10-19T18:00:58Z

Append all the new artifacts here

arrow/dev/tasks/tasks.yml

Lines 930 to 942 in 02ad5ae

    
           - flight-sql-jdbc-core-{no_rc_snapshot_version}-cyclonedx.json 
        
           - flight-sql-jdbc-core-{no_rc_snapshot_version}-cyclonedx.xml 
        
           - flight-sql-jdbc-core-{no_rc_snapshot_version}-javadoc.jar 
        
           - flight-sql-jdbc-core-{no_rc_snapshot_version}-sources.jar 
        
           - flight-sql-jdbc-core-{no_rc_snapshot_version}-tests.jar 
        
           - flight-sql-jdbc-core-{no_rc_snapshot_version}.jar 
        
           - flight-sql-jdbc-core-{no_rc_snapshot_version}.pom 
        
           - flight-sql-jdbc-driver-{no_rc_snapshot_version}-cyclonedx.json 
        
           - flight-sql-jdbc-driver-{no_rc_snapshot_version}-cyclonedx.xml 
        
           - flight-sql-jdbc-driver-{no_rc_snapshot_version}-sources.jar 
        
           - flight-sql-jdbc-driver-{no_rc_snapshot_version}-tests.jar 
        
           - flight-sql-jdbc-driver-{no_rc_snapshot_version}.jar 
        
           - flight-sql-jdbc-driver-{no_rc_snapshot_version}.pom

danepitkin · 2023-10-19T19:20:47Z

@github-actions crossbow submit java

github-actions · 2023-10-19T19:23:45Z

Revision: 67cef64

Submitted crossbow builds: ursacomputing/crossbow @ actions-4c0762070e

Task	Status
java-jars
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

dev/tasks/tasks.yml

assignUser · 2023-10-20T03:40:49Z

the java-jars job is the important one for the release as those artifact are used for the release. Looking at the raw log it seems the bom module is not being build in the final step? https://github.com/ursacomputing/crossbow/actions/runs/6579276168/job/17875990318#step:6:11

jduo · 2023-10-20T15:59:41Z

the java-jars job is the important one for the release as those artifact are used for the release. Looking at the raw log it seems the bom module is not being build in the final step? https://github.com/ursacomputing/crossbow/actions/runs/6579276168/job/17875990318#step:6:11

It looks like maven does build it:
https://github.com/ursacomputing/crossbow/actions/runs/6579276168/job/17875990318#step:6:2659
but it is not referenced in the Upload step.

java/bom/pom.xml

jbonofre · 2023-10-23T14:52:41Z

java/bom/pom.xml

+  <parent>
+    <groupId>org.apache</groupId>
+    <artifactId>apache</artifactId>
+    <version>18</version>


Why not using apache 30 here ?

FYI, I'm working on a PR to have reproducible build (with several deps and plugins updates).

I guess because that's what our root POM uses still:

arrow/java/pom.xml

Lines 15 to 19 in 67cef64

<parent>

<groupId>org.apache</groupId>

<artifactId>apache</artifactId>

<version>18</version>

</parent>

I don't think we have any reason to stick to it if there's a much newer version, but if there are potential major changes we could open a separate PR to bump things afterwards?

Fair point :)
I propose to keep apache 18 in this PR.
I will do another PR to upgrade to apache 30 (coupling with other updates as said in my previous comment, I should be able to submit several PRs tonight or tomorrow my time 😄 ).

Thanks !

jduo · 2023-10-23T19:51:21Z

@github-actions crossbow submit java

github-actions · 2023-10-23T19:52:46Z

Unable to match any tasks for `java`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6618285398

jduo · 2023-10-23T19:57:20Z

@github-actions crossbow submit java

github-actions · 2023-10-23T19:57:21Z

Unable to match any tasks for `_java_`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6618323310

github-actions · 2023-10-23T19:59:13Z

Unable to match any tasks for `_java_`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6618336251

jduo · 2023-10-23T20:04:44Z

@github-actions crossbow submit java

github-actions · 2023-10-23T20:07:35Z

Revision: f526fd8

Submitted crossbow builds: ursacomputing/crossbow @ actions-765ba35b65

Task	Status
java-jars
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

jduo · 2023-10-23T22:25:52Z

@danepitkin @lidavidm @assignUser this PR is now green on crossbow.

The issue was that I had the artifact group set to org.apache instead of org.apache.arrow and one of the CI scripts was only copying artifacts installed to ~/.m2/org/apache/arrow. I've checked the BOM artifact to use org.apache.arrow now.

assignUser

LGTM but I'll wait for someone with java background to merge :)

lidavidm · 2023-10-24T12:40:52Z

java/bom/pom.xml

+      <dependency>
+        <groupId>org.apache.arrow</groupId>
+        <artifactId>arrow-vector</artifactId>
+        <version>${project.version}</version>
+        <type>test-jar</type>
+      </dependency>


There's still a test-jar vector dependency here, I think we don't want to re-export this to other projects outside of Arrow

Add Bill of Materials to simplify dependency management. Utilize the BOM internally.

lidavidm

Thanks!

conbench-apache-arrow · 2023-10-25T21:26:38Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 006f387.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change Provide an easy way for users to import Arrow maven packages. ### What changes are included in this PR? Add a BOM module and utilize this module internally for importing Arrow dependencies. ### Are these changes tested? Yes. ### Are there any user-facing changes? Users can import the BOM dependency now. * Closes: apache#38264 Authored-by: James Duong <[email protected]> Signed-off-by: David Li <[email protected]>

jduo requested a review from lidavidm as a code owner October 18, 2023 23:08

github-actions bot added Component: Java awaiting review Awaiting review labels Oct 18, 2023

jduo force-pushed the 38264-java-bom branch from ae177cf to 1174f6a Compare October 19, 2023 17:30

jduo requested review from assignUser, kou and raulcd as code owners October 19, 2023 17:30

jduo force-pushed the 38264-java-bom branch from 1174f6a to 67cef64 Compare October 19, 2023 18:51

kou reviewed Oct 19, 2023

View reviewed changes

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Oct 19, 2023

lidavidm reviewed Oct 20, 2023

View reviewed changes

java/bom/pom.xml Outdated Show resolved Hide resolved

jbonofre suggested changes Oct 23, 2023

View reviewed changes

jbonofre approved these changes Oct 23, 2023

View reviewed changes

jduo force-pushed the 38264-java-bom branch from 67cef64 to 9a84b0b Compare October 23, 2023 15:26

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 23, 2023

jduo force-pushed the 38264-java-bom branch from 9a84b0b to 4f0c123 Compare October 23, 2023 16:24

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 23, 2023

jduo force-pushed the 38264-java-bom branch from 4f0c123 to f526fd8 Compare October 23, 2023 19:48

assignUser approved these changes Oct 24, 2023

View reviewed changes

lidavidm reviewed Oct 24, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 24, 2023

jduo force-pushed the 38264-java-bom branch from f526fd8 to 780bf68 Compare October 24, 2023 13:28

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 24, 2023

apacheGH-38264: [Java][Packaging] Add BOM file

46d28f1

Add Bill of Materials to simplify dependency management. Utilize the BOM internally.

jduo force-pushed the 38264-java-bom branch from 780bf68 to 46d28f1 Compare October 24, 2023 13:32

lidavidm approved these changes Oct 24, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Oct 24, 2023

lidavidm merged commit 006f387 into apache:main Oct 24, 2023
16 checks passed

lidavidm removed the awaiting merge Awaiting merge label Oct 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-38264: [Java][Packaging] Add BOM file #38336

GH-38264: [Java][Packaging] Add BOM file #38336

jduo commented Oct 18, 2023 •

edited by github-actions bot

Loading

jduo commented Oct 18, 2023

danepitkin commented Oct 19, 2023

jduo commented Oct 19, 2023

danepitkin commented Oct 19, 2023

danepitkin commented Oct 19, 2023

jduo commented Oct 19, 2023

jduo commented Oct 19, 2023

lidavidm commented Oct 19, 2023

lidavidm commented Oct 19, 2023

danepitkin commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

assignUser commented Oct 20, 2023

jduo commented Oct 20, 2023

jbonofre Oct 23, 2023

lidavidm Oct 23, 2023

jbonofre Oct 23, 2023

jduo commented Oct 23, 2023 •

edited

Loading

github-actions bot commented Oct 23, 2023

jduo commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

jduo commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

jduo commented Oct 23, 2023

assignUser left a comment

lidavidm Oct 24, 2023

lidavidm left a comment

conbench-apache-arrow bot commented Oct 25, 2023

	<parent>
	<groupId>org.apache</groupId>
	<artifactId>apache</artifactId>
	<version>18</version>
	</parent>

GH-38264: [Java][Packaging] Add BOM file #38336

GH-38264: [Java][Packaging] Add BOM file #38336

Conversation

jduo commented Oct 18, 2023 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

jduo commented Oct 18, 2023

danepitkin commented Oct 19, 2023

jduo commented Oct 19, 2023

danepitkin commented Oct 19, 2023

danepitkin commented Oct 19, 2023

jduo commented Oct 19, 2023

jduo commented Oct 19, 2023

lidavidm commented Oct 19, 2023

lidavidm commented Oct 19, 2023

danepitkin commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

assignUser commented Oct 20, 2023

jduo commented Oct 20, 2023

jbonofre Oct 23, 2023

Choose a reason for hiding this comment

lidavidm Oct 23, 2023

Choose a reason for hiding this comment

jbonofre Oct 23, 2023

Choose a reason for hiding this comment

jduo commented Oct 23, 2023 • edited Loading

github-actions bot commented Oct 23, 2023

jduo commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

jduo commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

jduo commented Oct 23, 2023

assignUser left a comment

Choose a reason for hiding this comment

lidavidm Oct 24, 2023

Choose a reason for hiding this comment

lidavidm left a comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Oct 25, 2023

jduo commented Oct 18, 2023 •

edited by github-actions bot

Loading

jduo commented Oct 23, 2023 •

edited

Loading