diff --git a/docs/design.md b/docs/design.md index fb84c74ad..d1955f026 100644 --- a/docs/design.md +++ b/docs/design.md @@ -4,8 +4,7 @@ * [Introduction](#introduction) * [Architecture](#architecture) * [The CRD Controller](#the-crd-controller) -* [Handling Application Restart](#handling-application-restart) -* [Handling Retries of Failed Submissions](#handling-retries-of-failed-submissions) +* [Handling Application Restart and Failures](#handling-application-restart-and-failures) * [Mutating Admission Webhook](#mutating-admission-webhook) * [Command-line Tool: Sparkctl](#command-line-tool-sparkctl) @@ -45,7 +44,7 @@ As part of preparing a submission for a newly created `SparkApplication` object, ## Handling Application Restart And Failures -The operator provides a configurable option through the `RestartPolicy` field of `SparkApplicationSpec` (see the [Configuring Automatic Application Restart and Failure Handling](user-guide.md) for more details) for specifying the application restart policy. The operator determines if an application should be restarted based on its termination state and the restart policy. As discussed above, the termination state of an application is based on the termination state of the driver pod. So effectively the decision is based on the termination state of the driver pod and the restart policy. Specifically, one of the following conditions applies: +The operator provides a configurable option through the `RestartPolicy` field of `SparkApplicationSpec` (see the [Configuring Automatic Application Restart and Failure Handling](user-guide.md#configuring-automatic-application-restart-and-failure-handling) for more details) for specifying the application restart policy. The operator determines if an application should be restarted based on its termination state and the restart policy. As discussed above, the termination state of an application is based on the termination state of the driver pod. So effectively the decision is based on the termination state of the driver pod and the restart policy. Specifically, one of the following conditions applies: * If the restart policy type is `Never`, the application is not restarted upon terminating. * If the restart policy type is `Always`, the application gets restarted regardless of the termination state of the application. Please note that such an Application will never end up in a terminal state of `COMPLETED` or `FAILED`. diff --git a/docs/developer-guide.md b/docs/developer-guide.md index 9a00c06ca..d198d7261 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -92,4 +92,4 @@ When you update the API, or specifically the `SparkApplication` and `ScheduledSp make build-api-docs ``` -Running the aboe command will update the file `docs/api-docs.md`. \ No newline at end of file +Running the above command will update the file `docs/api-docs.md`. \ No newline at end of file diff --git a/docs/user-guide.md b/docs/user-guide.md index 3f2a71190..ed3dc35e8 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -28,7 +28,7 @@ The Kubernetes Operator for Apache Spark ships with a command-line tool called ` * [Using Tolerations](#using-tolerations) * [Using Pod Security Context](#using-pod-security-context) * [Using Sidecar Containers](#using-sidecar-containers) - * [Using Init-Containers](#using-init-Containers) + * [Using Init-Containers](#using-init-containers) * [Using Volume For Scratch Space](#using-volume-for-scratch-space) * [Using Termination Grace Period](#using-termination-grace-period) * [Using Container LifeCycle Hooks](#using-container-lifecycle-hooks) @@ -39,8 +39,7 @@ The Kubernetes Operator for Apache Spark ships with a command-line tool called ` * [Deleting a SparkApplication](#deleting-a-sparkapplication) * [Updating a SparkApplication](#updating-a-sparkapplication) * [Checking a SparkApplication](#checking-a-sparkapplication) - * [Configuring Automatic Application Restart](#configuring-automatic-application-restart) - * [Configuring Automatic Application Re-submission on Submission Failures](#configuring-automatic-application-re-submission-on-submission-failures) + * [Configuring Automatic Application Restart and Failure Handling](#configuring-automatic-application-restart-and-failure-handling) * [Setting TTL for a SparkApplication](#setting-ttl-for-a-sparkapplication) * [Running Spark Applications on a Schedule using a ScheduledSparkApplication](#running-spark-applications-on-a-schedule-using-a-scheduledsparkapplication) * [Enabling Leader Election for High Availability](#enabling-leader-election-for-high-availability) @@ -437,7 +436,7 @@ Note that the mutating admission webhook is needed to use this feature. Please r ### Using Sidecar Containers -A `SparkApplication` can specify one or more optional sidecar containers for the driver or executor pod, using the optional field `.spec.driver.sidecars` or `.spec.executor.containers`. The specification of each sidecar container follows the [Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#container-v1-core) API definition. Below is an example: +A `SparkApplication` can specify one or more optional sidecar containers for the driver or executor pod, using the optional field `.spec.driver.sidecars` or `.spec.executor.sidecars`. The specification of each sidecar container follows the [Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#container-v1-core) API definition. Below is an example: ```yaml spec: @@ -502,17 +501,17 @@ If that storage isn't enough or you want to use a specific path, you can use one The volume names should start with `spark-local-dir-`. -``` +```yaml spec: volumes: - name: "spark-local-dir-1" hostPath: path: "/tmp/spark-local-dir" - executor: - volumeMounts: - - name: "spark-local-dir-1" - mountPath: "/tmp/spark-local-dir" - ... + executor: + volumeMounts: + - name: "spark-local-dir-1" + mountPath: "/tmp/spark-local-dir" + ... ``` Then you will get `SPARK_LOCAL_DIRS` set to `/tmp/spark-local-dir` in the pod like below. @@ -520,7 +519,7 @@ Then you will get `SPARK_LOCAL_DIRS` set to `/tmp/spark-local-dir` in the pod li ```yaml Environment: SPARK_USER: root - SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP) + SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP) SPARK_LOCAL_DIRS: /tmp/spark-local-dir SPARK_CONF_DIR: /opt/spark/conf ``` @@ -528,7 +527,7 @@ Environment: > Note: Multiple volumes can be used together -``` +```yaml spec: volumes: - name: "spark-local-dir-1" @@ -537,13 +536,13 @@ spec: - name: "spark-local-dir-2" hostPath: path: "/mnt/dir2" - executor: - volumeMounts: - - name: "spark-local-dir-1" - mountPath: "/tmp/dir1" - - name: "spark-local-dir-2" - mountPath: "/tmp/dir2" - ... + executor: + volumeMounts: + - name: "spark-local-dir-1" + mountPath: "/tmp/dir1" + - name: "spark-local-dir-2" + mountPath: "/tmp/dir2" + ... ``` > Note: Besides `HostPath`, `PersistentVolumeClaim` can be used as well. @@ -598,7 +597,7 @@ spec: mainApplicationFile: local:///opt/spark/examples/src/main/python/pyfiles.py ``` -Some PySpark applications need additional Python packages to run. Such dependencies are specified using the optional field `.spec.deps.pyFiles` , which translates to the `--py-files` option of the spark-submit command. +Some PySpark applications need additional Python packages to run. Such dependencies are specified using the optional field `.spec.deps.pyFiles`, which translates to the `--py-files` option of the spark-submit command. ```yaml spec: @@ -742,7 +741,7 @@ The operator supports a high-availability (HA) mode, in which there can be more | ------------- | ------------- | ------------- | | `leader-election` | `false` | Whether to enable leader election (or the HA mode) or not. | | `leader-election-lock-namespace` | `spark-operator` | Kubernetes namespace of the lock resource used for leader election. | -| `leader-election-lock-name` | `spark-operator-lock` | Name of the ock resource used for leader election. | +| `leader-election-lock-name` | `spark-operator-lock` | Name of the lock resource used for leader election. | | `leader-election-lease-duration` | 15 seconds | Leader election lease duration. | | `leader-election-renew-deadline` | 14 seconds | Leader election renew deadline. | | `leader-election-retry-period` | 4 seconds | Leader election retry period. | diff --git a/sparkctl/README.md b/sparkctl/README.md index 5d0fe8e67..4f94e69c9 100644 --- a/sparkctl/README.md +++ b/sparkctl/README.md @@ -42,7 +42,7 @@ The `create` command also supports staging local application dependencies, thoug ##### Uploading to GCS For uploading to GCS, the value should be in the form of `gs://`. The bucket must exist and uploading fails if otherwise. The local dependencies will be uploaded to the path -`spark-app-dependencies//` in the given bucket. It replaces the file path of each local dependency with the URI of the remote copy in the parsed `SaprkApplication` object if uploading is successful. +`spark-app-dependencies//` in the given bucket. It replaces the file path of each local dependency with the URI of the remote copy in the parsed `SparkApplication` object if uploading is successful. Note that uploading to GCS requires a GCP service account with the necessary IAM permission to use the GCP project specified by service account JSON key file (`serviceusage.services.use`) and the permission to create GCS objects (`storage.object.create`). The service account JSON key file must be locally available and be pointed to by the environment variable