Skip to content

Commit

Permalink
Fix typos and links in several docs (#853)
Browse files Browse the repository at this point in the history
Trivial fixes on typos, links and formats in several docs.
  • Loading branch information
huskysun authored Apr 1, 2020
1 parent 0e0867f commit 8a4a991
Show file tree
Hide file tree
Showing 4 changed files with 24 additions and 26 deletions.
5 changes: 2 additions & 3 deletions docs/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@
* [Introduction](#introduction)
* [Architecture](#architecture)
* [The CRD Controller](#the-crd-controller)
* [Handling Application Restart](#handling-application-restart)
* [Handling Retries of Failed Submissions](#handling-retries-of-failed-submissions)
* [Handling Application Restart and Failures](#handling-application-restart-and-failures)
* [Mutating Admission Webhook](#mutating-admission-webhook)
* [Command-line Tool: Sparkctl](#command-line-tool-sparkctl)

Expand Down Expand Up @@ -45,7 +44,7 @@ As part of preparing a submission for a newly created `SparkApplication` object,

## Handling Application Restart And Failures

The operator provides a configurable option through the `RestartPolicy` field of `SparkApplicationSpec` (see the [Configuring Automatic Application Restart and Failure Handling](user-guide.md) for more details) for specifying the application restart policy. The operator determines if an application should be restarted based on its termination state and the restart policy. As discussed above, the termination state of an application is based on the termination state of the driver pod. So effectively the decision is based on the termination state of the driver pod and the restart policy. Specifically, one of the following conditions applies:
The operator provides a configurable option through the `RestartPolicy` field of `SparkApplicationSpec` (see the [Configuring Automatic Application Restart and Failure Handling](user-guide.md#configuring-automatic-application-restart-and-failure-handling) for more details) for specifying the application restart policy. The operator determines if an application should be restarted based on its termination state and the restart policy. As discussed above, the termination state of an application is based on the termination state of the driver pod. So effectively the decision is based on the termination state of the driver pod and the restart policy. Specifically, one of the following conditions applies:

* If the restart policy type is `Never`, the application is not restarted upon terminating.
* If the restart policy type is `Always`, the application gets restarted regardless of the termination state of the application. Please note that such an Application will never end up in a terminal state of `COMPLETED` or `FAILED`.
Expand Down
2 changes: 1 addition & 1 deletion docs/developer-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,4 +92,4 @@ When you update the API, or specifically the `SparkApplication` and `ScheduledSp
make build-api-docs
```

Running the aboe command will update the file `docs/api-docs.md`.
Running the above command will update the file `docs/api-docs.md`.
41 changes: 20 additions & 21 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The Kubernetes Operator for Apache Spark ships with a command-line tool called `
* [Using Tolerations](#using-tolerations)
* [Using Pod Security Context](#using-pod-security-context)
* [Using Sidecar Containers](#using-sidecar-containers)
* [Using Init-Containers](#using-init-Containers)
* [Using Init-Containers](#using-init-containers)
* [Using Volume For Scratch Space](#using-volume-for-scratch-space)
* [Using Termination Grace Period](#using-termination-grace-period)
* [Using Container LifeCycle Hooks](#using-container-lifecycle-hooks)
Expand All @@ -39,8 +39,7 @@ The Kubernetes Operator for Apache Spark ships with a command-line tool called `
* [Deleting a SparkApplication](#deleting-a-sparkapplication)
* [Updating a SparkApplication](#updating-a-sparkapplication)
* [Checking a SparkApplication](#checking-a-sparkapplication)
* [Configuring Automatic Application Restart](#configuring-automatic-application-restart)
* [Configuring Automatic Application Re-submission on Submission Failures](#configuring-automatic-application-re-submission-on-submission-failures)
* [Configuring Automatic Application Restart and Failure Handling](#configuring-automatic-application-restart-and-failure-handling)
* [Setting TTL for a SparkApplication](#setting-ttl-for-a-sparkapplication)
* [Running Spark Applications on a Schedule using a ScheduledSparkApplication](#running-spark-applications-on-a-schedule-using-a-scheduledsparkapplication)
* [Enabling Leader Election for High Availability](#enabling-leader-election-for-high-availability)
Expand Down Expand Up @@ -437,7 +436,7 @@ Note that the mutating admission webhook is needed to use this feature. Please r

### Using Sidecar Containers

A `SparkApplication` can specify one or more optional sidecar containers for the driver or executor pod, using the optional field `.spec.driver.sidecars` or `.spec.executor.containers`. The specification of each sidecar container follows the [Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#container-v1-core) API definition. Below is an example:
A `SparkApplication` can specify one or more optional sidecar containers for the driver or executor pod, using the optional field `.spec.driver.sidecars` or `.spec.executor.sidecars`. The specification of each sidecar container follows the [Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#container-v1-core) API definition. Below is an example:

```yaml
spec:
Expand Down Expand Up @@ -502,33 +501,33 @@ If that storage isn't enough or you want to use a specific path, you can use one
The volume names should start with `spark-local-dir-`.


```
```yaml
spec:
volumes:
- name: "spark-local-dir-1"
hostPath:
path: "/tmp/spark-local-dir"
executor:
volumeMounts:
- name: "spark-local-dir-1"
mountPath: "/tmp/spark-local-dir"
...
executor:
volumeMounts:
- name: "spark-local-dir-1"
mountPath: "/tmp/spark-local-dir"
...
```

Then you will get `SPARK_LOCAL_DIRS` set to `/tmp/spark-local-dir` in the pod like below.

```yaml
Environment:
SPARK_USER: root
SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP)
SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP)
SPARK_LOCAL_DIRS: /tmp/spark-local-dir
SPARK_CONF_DIR: /opt/spark/conf
```


> Note: Multiple volumes can be used together

```
```yaml
spec:
volumes:
- name: "spark-local-dir-1"
Expand All @@ -537,13 +536,13 @@ spec:
- name: "spark-local-dir-2"
hostPath:
path: "/mnt/dir2"
executor:
volumeMounts:
- name: "spark-local-dir-1"
mountPath: "/tmp/dir1"
- name: "spark-local-dir-2"
mountPath: "/tmp/dir2"
...
executor:
volumeMounts:
- name: "spark-local-dir-1"
mountPath: "/tmp/dir1"
- name: "spark-local-dir-2"
mountPath: "/tmp/dir2"
...
```

> Note: Besides `HostPath`, `PersistentVolumeClaim` can be used as well.
Expand Down Expand Up @@ -598,7 +597,7 @@ spec:
mainApplicationFile: local:///opt/spark/examples/src/main/python/pyfiles.py
```

Some PySpark applications need additional Python packages to run. Such dependencies are specified using the optional field `.spec.deps.pyFiles` , which translates to the `--py-files` option of the spark-submit command.
Some PySpark applications need additional Python packages to run. Such dependencies are specified using the optional field `.spec.deps.pyFiles`, which translates to the `--py-files` option of the spark-submit command.

```yaml
spec:
Expand Down Expand Up @@ -742,7 +741,7 @@ The operator supports a high-availability (HA) mode, in which there can be more
| ------------- | ------------- | ------------- |
| `leader-election` | `false` | Whether to enable leader election (or the HA mode) or not. |
| `leader-election-lock-namespace` | `spark-operator` | Kubernetes namespace of the lock resource used for leader election. |
| `leader-election-lock-name` | `spark-operator-lock` | Name of the ock resource used for leader election. |
| `leader-election-lock-name` | `spark-operator-lock` | Name of the lock resource used for leader election. |
| `leader-election-lease-duration` | 15 seconds | Leader election lease duration. |
| `leader-election-renew-deadline` | 14 seconds | Leader election renew deadline. |
| `leader-election-retry-period` | 4 seconds | Leader election retry period. |
Expand Down
2 changes: 1 addition & 1 deletion sparkctl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The `create` command also supports staging local application dependencies, thoug
##### Uploading to GCS

For uploading to GCS, the value should be in the form of `gs://<bucket>`. The bucket must exist and uploading fails if otherwise. The local dependencies will be uploaded to the path
`spark-app-dependencies/<SaprkApplication namespace>/<SparkApplication name>` in the given bucket. It replaces the file path of each local dependency with the URI of the remote copy in the parsed `SaprkApplication` object if uploading is successful.
`spark-app-dependencies/<SparkApplication namespace>/<SparkApplication name>` in the given bucket. It replaces the file path of each local dependency with the URI of the remote copy in the parsed `SparkApplication` object if uploading is successful.

Note that uploading to GCS requires a GCP service account with the necessary IAM permission to use the GCP project specified by service account JSON key file (`serviceusage.services.use`) and the permission to create GCS objects (`storage.object.create`).
The service account JSON key file must be locally available and be pointed to by the environment variable
Expand Down

0 comments on commit 8a4a991

Please sign in to comment.