BIGTOP-4278. Upgrade Kafka to 3.4.1.

[sekikn]

Description of PR

This PR upgrades Kafka to 3.9.0 for the Bigtop 3.4.0 release.

How was this patch tested?

Tested on Debian 11 and Rocky 9, as follows:

$ ./gradlew kafka-clean kafka-pkg repo -Dbuildwithdeps=true
$ cd provisioner/docker
$ ./docker-hadoop.sh -d -dcp -C config_debian-11.yaml -F docker-compose-cgroupv2.yml -G -L -k zookeeper,kafka -s kafka -c 3

...

> Task :bigtop-tests:smoke-tests:kafka:test
Finished generating test XML results (0.003 secs) into: /bigtop-home/bigtop-tests/smoke-tests/kafka/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.01 secs) into: /bigtop-home/bigtop-tests/smoke-tests/kafka/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:kafka:test (Thread[Execution worker for ':' Thread 2,5,main]) completed. Took 6.02 secs.

BUILD SUCCESSFUL in 48s
27 actionable tasks: 7 executed, 20 up-to-date

$ ./gradlew kafka-clean kafka-pkg repo -Dbuildwithdeps=true
$ cd provisioner/docker
$ ./docker-hadoop.sh -d -dcp -C config_rockylinux-9.yaml -F docker-compose-cgroupv2.yml -G -L -r file:///bigtop-home/output -k zookeeper,kafka -s kafka -c 3                                                                                                         

> Task :bigtop-tests:smoke-tests:kafka:test
Finished generating test XML results (0.005 secs) into: /bigtop-home/bigtop-tests/smoke-tests/kafka/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.008 secs) into: /bigtop-home/bigtop-tests/smoke-tests/kafka/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:kafka:test (Thread[Daemon worker,5,main]) completed. Took 6.187 secs.

BUILD SUCCESSFUL in 22s
27 actionable tasks: 7 executed, 20 up-to-date

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
• Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

[sekikn]

I've just noticed Spark 3.5.3 has API incompatibility with recent versions of Kafka, so I'll downgrade Kafka to 3.4.1, which Spark 3.5.3 is built against by default. I'll also downgrade Scala to 2.12.x, since some script provided by Spark to switch Scala version doesn't seem to work in Bigtop's build process as it is.