[SPARK-50177][BUILD] Upgrade Arrow to 18.0.0

### What changes were proposed in this pull request?

This PR aims to upgrade `Arrow` to 18.0.0 for Apache Spark 4.0.0.

### Why are the changes needed?

To bring the latest improvements and bug fixes,
- https://arrow.apache.org/release/18.0.0.html  (28 October 2024)

Note that `Arrow 18` is the first release who dropped Java 8 like Apache Spark 4.0.0.
- apache/arrow#38051

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #48708 from dongjoon-hyun/SPARK-50177.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

dev/deps/spark-deps-hadoop-3-hive-2.3

@@ -16,11 +16,11 @@ antlr4-runtime/4.13.1//antlr4-runtime-4.13.1.jar
 aopalliance-repackaged/3.0.6//aopalliance-repackaged-3.0.6.jar
 arpack/3.0.3//arpack-3.0.3.jar
 arpack_combined_all/0.1//arpack_combined_all-0.1.jar
-arrow-format/17.0.0//arrow-format-17.0.0.jar
-arrow-memory-core/17.0.0//arrow-memory-core-17.0.0.jar
-arrow-memory-netty-buffer-patch/17.0.0//arrow-memory-netty-buffer-patch-17.0.0.jar
-arrow-memory-netty/17.0.0//arrow-memory-netty-17.0.0.jar
-arrow-vector/17.0.0//arrow-vector-17.0.0.jar
+arrow-format/18.0.0//arrow-format-18.0.0.jar
+arrow-memory-core/18.0.0//arrow-memory-core-18.0.0.jar
+arrow-memory-netty-buffer-patch/18.0.0//arrow-memory-netty-buffer-patch-18.0.0.jar
+arrow-memory-netty/18.0.0//arrow-memory-netty-18.0.0.jar
+arrow-vector/18.0.0//arrow-vector-18.0.0.jar
 audience-annotations/0.12.0//audience-annotations-0.12.0.jar
 avro-ipc/1.12.0//avro-ipc-1.12.0.jar
 avro-mapred/1.12.0//avro-mapred-1.12.0.jar

pom.xml

@@ -226,7 +226,7 @@
     ./python/pyspark/sql/pandas/utils.py, ./python/packaging/classic/setup.py
     and ./python/packaging/connect/setup.py too.
     -->
-    <arrow.version>17.0.0</arrow.version>
+    <arrow.version>18.0.0</arrow.version>
     <ammonite.version>3.0.0</ammonite.version>
     <jjwt.version>0.12.6</jjwt.version>