machine learning and deep learning job with pyspark > 3.0.
running jobs in container have advantages as follows:
- a seperate runtime and evrionment
- easy to copy cuda environment to all executores
running spark on k8s make it possible to use gpu clusters.
It's easy to create job CR to run job with spark operator