GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests

[raulcd]

Rationale for this change

It does seem that the only way to shadow Netty version is to modify the Pom for previous versions.

What changes are included in this PR?

Try to patch version of Netty on the pom when cloning Spark.

Are these changes tested?

Archery integration tests

Are there any user-facing changes?

No

• Closes: [CI][Java] Integration jobs with Spark fail with NoSuchMethodError:io.netty.buffer.PooledByteBufAllocator #36332

[raulcd]

@github-actions crossbow submit spark

[github-actions]

⚠️ GitHub issue #36332 has been automatically assigned in GitHub to PR creator.

[github-actions]

Revision: 44ec8f8

Submitted crossbow builds: ursacomputing/crossbow @ actions-e71df5add3

Task	Status
test-conda-python-3.10-spark-master
test-conda-python-3.8-spark-v3.1.2
test-conda-python-3.9-spark-v3.2.0

[raulcd]

This is not going to be enough because from my understanding Spark also pins the versions used on Hadoop here:
https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-3-hive-2.3#L186-L203
and I am not sure we want to patch those too.
@kiszk any idea?

[kiszk]

This PR is motivated by #36332

[kiszk]

LGTM.
It looks like a practical solution to resolve a chicken-and-egg problem.
We need to ensure the latest Arrow works well with Spark. Then, we can release the latest Arrow with Netty 4.1.94. After that, Spark would update the version of Arrow.

[kiszk]

cc @BryanCutler

[raulcd]

@kiszk if I understand correctly you are suggesting to also patch the versions here: https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-3-hive-2.3#L186-L203 to see if the job changing the versions succeeds (at least on master spark)?

[kiszk]

@raulcd Sorry for confusing you. I am fine with this PR.

[raulcd]

but the spark jobs still fail unless we also patch those. I mean the current change doesn't seem enough to make our CI integration jobs with spark successful as seen on the crossbow report comment here: #36640 (comment)

[LuciferYang]

@kiszk if I understand correctly you are suggesting to also patch the versions here: https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-3-hive-2.3#L186-L203 to see if the job changing the versions succeeds (at least on master spark)?

I think this just used by dependencies change check(like a golden file), will not affect the actual netty version usage

[LuciferYang]

I run this sed statement locally, but I found that the netty.version in the pom.xml still 4.1.93.Final

Maybe we can test build/mvn versions:set-property -Dproperty=netty.version -DnewVersion=4.1.94.Final -DgenerateBackupPoms=false

[kiszk]

Hmm, this line may also say to use 4.1.93.final

[raulcd]

Thanks, I'll investigate, I might have messed up on my tests :)

[raulcd]

I am closing this PR as upgrading to netty 4.1.96 fixed the issue reverting the regression introduced on 4.1.94.