diff --git a/docs/reference/feast-and-spark.md b/docs/reference/feast-and-spark.md index c5a3a6de1d..9b770d68d3 100644 --- a/docs/reference/feast-and-spark.md +++ b/docs/reference/feast-and-spark.md @@ -27,6 +27,7 @@ Currently Feast is tested using `v1beta2-1.1.2-2.4.5`version of the operator ima | `SPARK_LAUNCHER` | `"k8s"` | | `SPARK_K8S_NAMESPACE` | The name of the Kubernetes namespace to run Spark jobs in. This should match the value of `sparkJobNamespace` set on spark-on-k8s-operator Helm chart. Typically this is also the namespace Feast itself will run in. The example above uses `sparkop`. | | `SPARK_STAGING_LOCATION` | S3 URL to use as a staging location, must be readable and writable by Feast. Use `s3a://` prefix here. Ex.: `s3a://some-bucket/some-prefix` | +| `SPARK_K8S_JOB_TEMPLATE_PATH` | Local file path with the template of the SparkApplication resource. No prefix required. Ex.: `/home/jovyan/work/sparkapp-template.yaml`. An example teamplate is [here](https://github.com/feast-dev/feast/blob/4059a21dc4eba9cd27b2d5b0fabe476c07a8b3bd/sdk/python/feast/pyspark/launchers/k8s/k8s_utils.py#L280-L317) and the spec is defined in the [k8s-operator User Guide](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md).| Lastly, make sure that the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below: diff --git a/sdk/python/feast/constants.py b/sdk/python/feast/constants.py index eb451a72cf..259920edae 100644 --- a/sdk/python/feast/constants.py +++ b/sdk/python/feast/constants.py @@ -193,6 +193,7 @@ class ConfigOptions(metaclass=ConfigMeta): SPARK_K8S_USE_INCLUSTER_CONFIG = "True" # SparkApplication resource template + # Eg. '/home/jovyan/work/sparkapp-template.yaml' SPARK_K8S_JOB_TEMPLATE_PATH = None #: File format of historical retrieval features