[Java] java-jars fails with "We have duplicated artifacts attached"

[lidavidm]

Describe the bug, including details regarding any error messages, version, and platform.

[INFO] <<< source:3.3.0:jar (default-cli) < generate-sources @ module-info-compiler-maven-plugin <<<
[INFO] 
[INFO] 
[INFO] --- source:3.3.0:jar (default-cli) @ module-info-compiler-maven-plugin ---
Error:  We have duplicated artifacts attached.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Arrow Java Root POM 17.0.0-SNAPSHOT:
[INFO] 
[INFO] Arrow Maven Plugins ................................ SUCCESS [ 37.141 s]
[INFO] Module Info Compiler Maven Plugin .................. FAILURE [ 16.713 s]

https://github.com/ursacomputing/crossbow/actions/runs/8903205653/job/24451424677#step:6:16247

Component(s)

Java

[lidavidm]

@vibhatha or @jbonofre, any idea? It's unclear when exactly this popped up since the test was blocked by apache/arrow#41470 until just now

[vibhatha]

Would it be the recent maven build changes, my best guess would be the last build PR, but I can't be sure. Though I didn't notice it before that.

[vibhatha]

Checking my recent crossbow usage

8 days ago: passes

apache/arrow#40340 (comment)

6 days ago: fails

apache/arrow#40340 (comment)

[vibhatha]

I know this doesn't help, but at least it should have happened sometime after 8 days ago.

[lidavidm]

Do you think you could bisect it? Or at least figure out which commits are possible causes in that time range?

[jbonofre]

That's due to the source maven plugin update coming with new Apache POM. On some profile, the source and jar maven plugins are executed by the parent Apache POM and by the project pom. So we have a dual execution, so dual artifacts. It's what I saw on Apache projects. I gonna take a look if it's the same here in Arrow.

[vibhatha]

@lidavidm I am looking into it too.
But seems like @jbonofre has a good pointer to it.

[vibhatha]

@lidavidm I feel like this is the PR which could have broken the CIs. Though java-jars pass in this PR itself, a change made in another PR followed by this merge could have triggered it. It is very hard to predict this since in PR apache/arrow#41309 the java-jars failure is clouded by the C++ filesystem issue which was fixed recently. But that PR has some changes in plugins. But I cannot exactly pin-point what change is causing it.

cc @jbonofre WDYT?

[jbonofre]

@vibhatha I don't think that the plugin-plugin update is the root cause of the problem.
IMHO, I'm suspecting more this commit 9090e679da91e0544171c2da9f2b9ce8ba23d389 as it updated to Apache POM 31.

Let me check locally.

[vibhatha]

Yes, this is the PR I am also suspicious about. I am going to wait for your response. Thanks for looking into this @jbonofre

[jbonofre]

I think, we hit the same issue as in my PR (apache/arrow#39215). We had similar issue there (due to Apache POM version bump as well). Let me double check if it's the same and eventually leverage the reproducible build PR to fix that 😄

[lidavidm]

This will become a release blocker at some point but I suppose 17.0.0 is still a bit off

[jbonofre]

@lidavidm agree, so let me tackle that ! At your service 😄

[lidavidm]

It's much appreciated :)

[jbonofre]

@lidavidm my pleasure, always happy to help 😄

[jbonofre]

@lidavidm @vibhatha

The problem is in java_full_build.sh in crossbow: https://github.com/apache/arrow/blob/c8f89d06bb79909dbd52d7d1504bdcadb220fadd/ci/scripts/java_full_build.sh#L50

It explicitely calls source:jar goal whereas it's already done by Apache POM 31.

IMHO, the java_full_build.sh should only do:

mvn clean \
    install \
    -Papache-release \
    -Parrow-c-data \
    -Parrow-jni \
    -Darrow.cpp.build.dir=$dist_dir \
    -Darrow.c.jni.dist.dir=$dist_dir \
    -DdescriptorId=source-release

If agree, I can create a PR to change this pipeline.

[vibhatha]

@jbonofre it could be needed in apache/arrow#41772 PR itself?

[lidavidm]

Ok, we can test in CI and confirm. Thanks for digging into this!