Start using spark4-preview versions

[mathbunnyru]

Describe your changes

Issue ticket if applicable

Checklist (especially for first-time contributors)

• I have performed a self-review of my code
• If it is a core feature, I have added thorough tests
• I will try not to use force-push to make the review process easier for reviewers
• I have updated the documentation for significant changes

[mathbunnyru]

I checked the build logs, and the expected version of spark is installed:

make build/pyspark-notebook

 => => # INFO:__main__:Latest version: 4.0.0-preview2
 => => # INFO:__main__:Downloading and unpacking Spark
 => => # INFO:__main__:Spark directory name: spark-4.0.0-preview2-bin-hadoop3

Then I checked the image itself, and the directories are named properly:

docker run -it --rm quay.io/jupyter/pyspark-notebook bash

(base) jovyan@b6f0e3c1463d:~$ ll /usr/local | grep spark
lrwxrwxrwx  1 root root   43 Oct 20 12:51 spark -> /usr/local/spark-4.0.0-preview2-bin-hadoop3/
drwxr-xr-x 14 root root 4096 Sep 16 04:02 spark-4.0.0-preview2-bin-hadoop3/

And finally, the image tag:

make hook/pyspark-notebook

INFO:__main__:Calculated tag, tagger_name: SparkVersionTagger tag_value: spark-4.0.0-preview2

docker image ls | grep aarch64-spark
quay.io/jupyter/pyspark-notebook   aarch64-spark-4.0.0-preview2   116a1ce7d803   2 minutes ago   4.51GB

So, everything works exactly as expected

[mathbunnyru]

Went with 2.2.2, becuase preview2 supports this version

[mathbunnyru]

I will describe the release schedule in this PR: #2072

[mathbunnyru]

This is the only line we will have to revert to start using stable versions again (that's why I made a separate commit improving spark setup scripts)

[mathbunnyru]

I guess this might disappear when we switch to JDK 21, but that's a separate story and won't be a part of the switch to Python 3.12

[mathbunnyru]

Unfortunately, sparklyr doesn't seem to support spark v4 yet.

When I run spark_available_versions(), it only gives me versions up to 3.5.

I created an upstream issue: sparklyr/sparklyr#3468