How to bump spark version in elyra/kernel-spark-py?

[paf91]

Hi! Is there a simple way to bump spark and pyspark version in image elyra/kernel-spark-py:3.2.3?
Lower version (3.2) is not supported by the best spark clickhouse connector of latest version 0.7.3, Spark 3.4.2 Java 17 Scala 2.12 would be great or at least Spark 3.3.
https://housepower.github.io/spark-clickhouse-connector/#requirements

Looking at this Dockerfile
makes me think I can't just pass new spark version as hadoop2.7 is not included in repo, I need also to change binary to hadoop3
Of course that is not enough and I would like to get more libraries, like hadoop-aws:3.3.4 to get S3 support.

[paf91]

So basically I made Spark 3.4.2 and Spark 3.3.4 images on top of elyra/kernel-py:3.2.3. Created kernel.json based on spark_python_kubernetes kernel in EG 3.2.3.
As a result, Driver+executor pods are initialized, but sparksession for Spark 3.4.2 hangs on init stage and never initializes, meanwhile Spark 3.3.4 works just fine.
Something is wrong but can't figure out what it is, maybe the pod template? I simply copied EG 3.2.3 spark_python_kubernetes kernel folder and changed the image to what I built on top of elyra/kernel-py:3.2.3