[SPARK-22648] [K8S] Spark on Kubernetes - Documentation #19946

foxish · 2017-12-12T00:27:50Z

What changes were proposed in this pull request?

This PR contains documentation on the usage of Kubernetes scheduler in Spark 2.3, and a shell script to make it easier to build docker images required to use the integration. The changes detailed here are covered by #19717 and #19468 which have merged already.

How was this patch tested?
The script has been in use for releases on our fork. Rest is documentation.

cc @rxin @mateiz (shepherd)
k8s-big-data SIG members & contributors: @foxish @ash211 @mccheah @liyinan926 @erikerlandson @ssuchter @varunkatta @kimoonkim @tnachen @ifilonenko
reviewers: @vanzin @felixcheung @jiangxb1987 @mridulm

TODO:

Add dockerfiles directory to built distribution. ([SPARK-22777][Scheduler] Kubernetes mode dockerfile permission and distribution #20007)
Change references to docker to instead say "container" ([SPARK-22807] [Scheduler] Remove config that says docker and replace with container #19995)
Update configuration table.
Modify spark.kubernetes.allocation.batch.delay to take time instead of int ([SPARK-22845] [Scheduler] Modify spark.kubernetes.allocation.batch.delay to take time instead of int #20032)

holdenk · 2017-12-12T12:08:26Z

Jenkins OK to test

holdenk · 2017-12-12T12:31:39Z

sbin/build-push-docker-images.sh

+  done
+}
+
+function usage {


So likely not for this PR, but I'm wondering for the future what you think about the idea of extending this to package up a usable Python env so we can solve the dependency management issue as well?

Really excited to see the progress :)

We do already do that in our fork - and when we get PySpark and R submitted (in Spark 2.4 hopefully), we will extend this script.

jiangxb1987

Mostly LGTM, only some nits.

jiangxb1987 · 2017-12-12T12:12:17Z

sbin/build-push-docker-images.sh

+    push) push;;
+    *) usage;;
+  esac
+fi


nit: add extra empty line

jiangxb1987 · 2017-12-12T12:35:38Z

docs/running-on-kubernetes.md

+* This will become a table of contents (this text will be scraped).
+{:toc}
+
+Spark can run on clusters managed by [Kubernetes](https://kubernetes.io). This features makes use of the new experimental native


"This features" -> "This feature"

jiangxb1987 · 2017-12-12T12:40:57Z

docs/running-on-kubernetes.md

+spark-submit can be directly used to submit a Spark application to a Kubernetes cluster. The mechanism by which spark-submit happens is as follows:
+
+* Spark creates a spark driver running within a [Kubernetes pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/).
+* The driver creates executors which are also Kubernetes pods and connects to them, and executes application code.


"which are also Kubernetes pods" -> "which are also running within Kubernetes pods"

jiangxb1987 · 2017-12-12T13:50:12Z

docs/running-on-kubernetes.md

+
+To launch Spark Pi in cluster mode,
+
+    bin/spark-submit \


nit: add {% highlight bash %} over the command.

jiangxb1987 · 2017-12-12T13:57:44Z

docs/running-on-kubernetes.md

+master string with `k8s://` will cause the Spark application to launch on the Kubernetes cluster, with the API server
+being contacted at `api_server_url`. If no HTTP protocol is specified in the URL, it defaults to `https`. For example,
+setting the master to `k8s://example.com:443` is equivalent to setting it to `k8s://https://example.com:443`, but to
+connect without TLS on a different port, the master would be set to `k8s://http://example.com:8080`.


Maybe I missed something but where is the logic that handles connect without TLS you mentioned here?

It should just be handled by the fabric8 client. We added this in fabric8io/kubernetes-client#652

jiangxb1987 · 2017-12-12T15:33:50Z

docs/running-on-kubernetes.md

+### Namespaces
+
+Kubernetes has the concept of [namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
+Namespaces are a way to divide cluster resources between multiple users (via resource quota). Spark on Kubernetes can


"a way" -> "ways"

jiangxb1987 · 2017-12-12T15:39:48Z

docs/running-on-kubernetes.md

+
+# Configuration
+
+See the [configuration page](configuration.html) for information on Spark configurations.  The following configuration is


" The following configuration is" -> " The following configurations are"

foxish · 2017-12-12T19:46:54Z

@jiangxb1987, thanks for the round of review - I updated some links and pointers to the docs as well. PTAL.

vanzin · 2017-12-12T21:14:41Z

docs/running-on-kubernetes.md

+[kubectl](https://kubernetes.io/docs/user-guide/prereqs/).  If you do not already have a working Kubernetes cluster,
+you may setup a test cluster on your local machine using
+[minikube](https://kubernetes.io/docs/getting-started-guides/minikube/).
+  * We recommend using the latest releases of minikube be updated to the most recent version with the DNS addon enabled.


This sentence doesn't read right.

vanzin · 2017-12-12T21:15:34Z

docs/running-on-kubernetes.md

+  <img src="img/k8s-cluster-mode.png" title="Spark cluster components" alt="Spark cluster components" />
+</p>
+
+spark-submit can be directly used to submit a Spark application to a Kubernetes cluster. The mechanism by which spark-submit happens is as follows:


<code>spark-submit</code>

"The submission mechanism works as follows:"

vanzin · 2017-12-12T21:16:42Z

docs/running-on-kubernetes.md

+* Spark creates a spark driver running within a [Kubernetes pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/).
+* The driver creates executors which are also running within Kubernetes pods and connects to them, and executes application code.
+* When the application completes, the executor pods terminate and are cleaned up, but the driver pod persists
+logs and remains in "completed" state in the Kubernetes API till it's eventually garbage collected or manually cleaned up.


s/till/until

vanzin · 2017-12-12T21:21:08Z

docs/running-on-kubernetes.md

+
+Kubernetes has the concept of [namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
+Namespaces are ways to divide cluster resources between multiple users (via resource quota). Spark on Kubernetes can
+use namespaces to launch spark applications. This is through the `--conf spark.kubernetes.namespace` argument to spark-submit.


Instead of mentioning it as a spark-submit argument, mention it as a configuration. Users have more than one way to set configuration in their apps.

vanzin · 2017-12-12T21:22:27Z

docs/running-on-kubernetes.md

+service account that has the right role granted. Spark on Kubernetes supports specifying a custom service account to
+be used by the driver pod through the configuration property
+`spark.kubernetes.authenticate.driver.serviceAccountName=<service account name>`. For example to make the driver pod
+to use the `spark` service account, a user simply adds the following option to the `spark-submit` command:


"...to make the driver pod use..."

vanzin · 2017-12-12T21:24:05Z

docs/running-on-kubernetes.md

+  </td>
+</tr>
+<tr>
+  <td><code>spark.kubernetes.executor.docker.image</code></td>


Same question.

Yes, it's required.

vanzin · 2017-12-12T21:24:42Z

docs/running-on-kubernetes.md

+</tr>
+<tr>
+  <td><code>spark.kubernetes.allocation.batch.delay</code></td>
+  <td><code>1</code></td>


Hmm, isn't this a time conf? If so, specify the value with the unit and don't mention units in the description.

This is an int conf at the moment - but good point about making it a time config. I think it makes sense to change that. Will draft up a PR.

Fix in #20032

vanzin · 2017-12-12T21:25:58Z

docs/running-on-kubernetes.md

+  </td>
+</tr>
+<tr>
+  <td><code>spark.kubernetes.authenticate.submission.oauthToken</code></td>


If not yet supported, I recommend adding support for reading this from an env variable; command line args are visible by anyone using ps.

That makes sense. I think the fabric8 client has a way of consuming that from env-vars, but we need to check that it works.
cc @mccheah

Filing an issue to unblock. I think we can work around this for now - but will be important to document for the future.

vanzin · 2017-12-12T21:27:47Z

docs/submitting-applications.md

@@ -155,6 +165,12 @@ The master URL passed to Spark can be in one of the following formats:
        <code>client</code> or <code>cluster</code> mode depending on the value of <code>--deploy-mode</code>.
        The cluster location will be found based on the <code>HADOOP_CONF_DIR</code> or <code>YARN_CONF_DIR</code> variable.
 </td></tr>
+<tr><td> <code>k8s://HOST:PORT</code> </td><td> Connect to a <a href="running-on-kubernetes.html"> Kubernetes </a> cluster in


Remove spaces around "Kubernetes".

vanzin · 2017-12-12T21:28:45Z

docs/running-on-kubernetes.md

+   <td><code>spark.kubernetes.driver.secrets.[SecretName]</code></td>
+   <td>(none)</td>
+   <td>
+     Mounts the Kubernetes secret named <code>SecretName</code> onto the path specified by the value


What is a "Kubernetes secret"? Do you need a link to explain it somewhere? When would you use this?

ueshin · 2017-12-13T11:32:31Z

ok to test

ueshin · 2017-12-13T11:34:25Z

docs/index.md

@@ -112,7 +113,7 @@ options for deployment:
  * [Mesos](running-on-mesos.html): deploy a private cluster using
      [Apache Mesos](http://mesos.apache.org)
  * [YARN](running-on-yarn.html): deploy Spark on top of Hadoop NextGen (YARN)
-  * [Kubernetes (experimental)](https://github.com/apache-spark-on-k8s/spark): deploy Spark on top of Kubernetes
+  * [Kubernetes (experimental)](running-on-kubernetes.html): deploy Spark on top of Kubernetes


We can remove (experimental) here as well?

ueshin · 2017-12-13T12:01:44Z

docs/running-on-kubernetes.md

+Kubernetes master is running at http://127.0.0.1:6443
+```
+
+In the above example, the specific Kubernetes cluster can be used with spark submit by specifying


spark-submit instead of spark submit?

ueshin · 2017-12-13T12:03:31Z

docs/running-on-kubernetes.md

+The local proxy can be started by:
+
+```bash
+ kubectl proxy


nit: we can remove extra space at the beginning of this line.

ueshin · 2017-12-13T12:08:58Z

docs/running-on-kubernetes.md

+
+### Accessing Logs
+
+Logs can be accessed using the kubernetes API and the `kubectl` CLI. When a Spark application is running, it's possible


nit: Kubernetes API instead of kubernetes API (using a capital letter K)?

ueshin · 2017-12-13T12:10:54Z

docs/running-on-kubernetes.md

+
+There may be several kinds of failures. If the Kubernetes API server rejects the request made from spark-submit, or the
+connection is refused for a different reason, the submission logic should indicate the error encountered. However, if there
+are errors during the running of the application, often, the best way to investigate may be through the kubernetes CLI.


nit: Kubernetes CLI instead of kubernetes CLI?

ueshin · 2017-12-13T12:13:01Z

docs/running-on-kubernetes.md

+
+Status and logs of failed executor pods can be checked in similar ways. Finally, deleting the driver pod will clean up the entire spark 
+application, includling all executors, associated service, etc. The driver pod can be thought of as the Kubernetes representation of 
+the spark application.


nit: Spark application instead of spark application?

ueshin · 2017-12-13T12:13:26Z

docs/running-on-kubernetes.md

+
+Kubernetes has the concept of [namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
+Namespaces are ways to divide cluster resources between multiple users (via resource quota). Spark on Kubernetes can
+use namespaces to launch spark applications. This is through the `--conf spark.kubernetes.namespace` argument to spark-submit.


ueshin · 2017-12-13T12:23:44Z

docs/running-on-kubernetes.md

+  <td><code>spark.kubernetes.authenticate.driver.oauthToken</code></td>
+  <td>(none)</td>
+  <td>
+    OAuth token to use when authenticating against the against the Kubernetes API server from the driver pod when


against the against -> against?

SparkQA · 2017-12-13T14:56:06Z

Test build #84850 has finished for PR 19946 at commit 5f24de1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mridulm

I left a few minor comments, overall looks pretty good - thanks for the PR !

mridulm · 2017-12-13T14:26:46Z

docs/running-on-kubernetes.md

+{:toc}
+
+Spark can run on clusters managed by [Kubernetes](https://kubernetes.io). This feature makes use of the new experimental native
+Kubernetes scheduler that has been added to Spark.


Remove experimental ? I think there are other references (a grep should catch them all) and we can remove them all now ?

Done. The other references have been removed. Couldn't find any others.

mridulm · 2017-12-13T14:32:36Z

docs/running-on-kubernetes.md

+* When the application completes, the executor pods terminate and are cleaned up, but the driver pod persists
+logs and remains in "completed" state in the Kubernetes API till it's eventually garbage collected or manually cleaned up.
+
+Note that in the completed state, the driver pod does *not* use any computational or memory resources.


Just curious - what about disk usage ? What is it proportional to ?

Some edit probably buried this query ... would be great if you can opine @foxish, thanks.

mridulm · 2017-12-13T14:35:32Z

docs/running-on-kubernetes.md

+decisions for driver and executor pods using advanced primitives like
+[node selectors](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector)
+and [node/pod affinities](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity)
+in a future release.


As an aside - should have asked about it in scheduler review - how does kubernetes handle prioritization of resources ? In particular, if it is pre-empting containers, is there a relative ordering ?
Will driver will always be the last to go ?
If this has to be user specified, do we need to add doc link to it here ?

As of today, preemption is at random.
priority and preemption are in alpha as of now. As soon as they go to beta (in the Spark 2.4 timeframe), we'll add the required pieces to make it honor the rule as you said - driver is the last to go, etc.

mridulm · 2017-12-13T14:41:52Z

docs/running-on-kubernetes.md

+    --deploy-mode cluster \
+    --class org.apache.spark.examples.SparkPi \
+    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+    --conf spark.kubernetes.namespace=default \


Remove this default value from example ? (or set to non-default value if we want to illustrate its use)

mridulm · 2017-12-13T14:43:21Z

docs/running-on-kubernetes.md

+    --class org.apache.spark.examples.SparkPi \
+    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+    --conf spark.kubernetes.namespace=default \
+    --conf spark.executor.instances=5 \


--num-executors instead ?

That is specified currently as a YARN-only option in spark-submit. Does it make sense to move it out to suit both K8s and YARN?

You are right, I did not realize mesos did not honour it and it was yarn only option !
It might make sense for k8s also to support it too (since it does not right now, my proposal for doc change would be incorrect) .. what do you think @foxish ?

mridulm · 2017-12-13T14:43:56Z

docs/running-on-kubernetes.md

+$ bin/spark-submit \
+    --deploy-mode cluster \
+    --class org.apache.spark.examples.SparkPi \
+    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \


Reorder so that master is above deploy-mode ?

mridulm · 2017-12-13T14:47:58Z

docs/running-on-kubernetes.md

+    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+    --conf spark.kubernetes.namespace=default \
+    --conf spark.executor.instances=5 \
+    --conf spark.app.name=spark-pi \


--name instead ?

mridulm · 2017-12-13T14:49:25Z

docs/running-on-kubernetes.md

+    --conf spark.app.name=spark-pi \
+    --conf spark.kubernetes.driver.docker.image=<driver-image> \
+    --conf spark.kubernetes.executor.docker.image=<executor-image> \
+    local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar


Just the filename ?

I want to make sure it is intuitive and easy for users looking at examples - and it looks familiar with how spark is typically invoked. (Though the scheme is referenced here https://github.com/apache/spark/pull/19946/files#diff-b5527f236b253e0d9f5db5164bdb43e9R108 : I am assuming avoiding it defaults to local)

Fair enough, I just wanted to show that it had to be a path. The protocol local:/// is necessary to point at container-local files, as opposed to local file paths on the submitting user's machine which will both be handled separately. Changed it to local:///path/to/examples.jar. Is that better?

Just to clarify - local:///my/path/jar implies local to the docker container ? and /my/path/jar points to submitters machine ?

mridulm · 2017-12-13T15:03:46Z

docs/running-on-kubernetes.md

+  <td>
+    In cluster mode, whether to wait for the application to finish before exiting the launcher process.  When changed to
+    false, the launcher has a "fire-and-forget" behavior when launching the Spark job.
+  </td>


When false, will terminating spark submit cause spark application to also terminate ?

No, it wouldn't. It would simply stop spark-submit from continuing to watch for status from the cluster. The application runs the same in either case.

SparkQA · 2017-12-13T18:11:46Z

Test build #84865 has finished for PR 19946 at commit 109069f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-13T18:31:09Z

Test build #84861 has finished for PR 19946 at commit f618e8b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-13T18:49:17Z

Test build #84864 has finished for PR 19946 at commit 8c656c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2017-12-14T00:02:34Z

docs/building-spark.md

@@ -49,7 +49,7 @@ To create a Spark distribution like those distributed by the
 to be runnable, use `./dev/make-distribution.sh` in the project root directory. It can be configured
 with Maven profile settings and so on like the direct Maven build. Example:

-    ./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn
+    ./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes


should we use k8s? I kept bringing this up and that's because I can never spell Kubernetes properly.

It's k8s:// in the URL scheme we use and the package names are also k8s - so, users should never have to type the name.
This is one of the last few holdouts. I'd say that it's consistent here, with the use of other cluster manager names in full in their maven projects. I can change it here if you feel strongly about it.

I actually second @rxin, I do get the spelling wrong at times :-)

Are you both also referring to the config options etc, which are still spark.kubernetes.*, or just the maven build target? If it's everything, it would be a fairly large change, doable certainly - but confirming how far the rename should go.

Ping @mridulm, @rxin - can you please confirm the scope of the renaming you were referring to here? Is it just the maven target? Changing all the config options etc would be a considerably large change at this point. Also, a point that was brought up today was - while k8s is common shorthand, it's not universal as is the full name.

I've filed https://issues.apache.org/jira/browse/SPARK-22853 to discuss this and unblock this PR. We should be able to reach consensus by release time. :)

Yea I don't think you need to block this pr with this.

vanzin · 2017-12-14T21:06:17Z

Seems like the PR builders lost their state?

vanzin · 2017-12-14T21:06:21Z

ok to test

vanzin · 2017-12-14T21:14:46Z

docs/running-on-kubernetes.md

+## Docker Images
+
+Kubernetes requires users to supply images that can be deployed into containers within pods. The images are built to
+be run in a container runtime environment that Kubernetes supports. Docker is a container runtime environment that is


a container runtime environment that Kubernetes supports

Does the current code support these other container runtimes, or just docker? From my memory (and the propery names) it seems that only docker is supported?

the images should be runnable by other container runtimes (cri-o, rkt, etc), although I'm not aware of anybody testing that.

@vanzin, we don't make any runtime level assumptions in Spark code. The K8s abstraction layer CRI should in theory allow using a differnet runtime.

My comment is more about the properties being called "docker" and whether that means only docker images are supported. If you can use any image supported by the k8s cluster, than pehaps the properties should be renamed.

I see your point - Although there is flexibility in theory, as of now, it's safe to assume that most people are running docker containers when using k8s - making the name docker much more intuitive. If the other runtimes do see traction in future (and we do some testing around them), we can rename to container.image instead of docker.image. As of now, I can make the documentation clearer that spark on k8s only supports docker images. Sound like a reasonable thing to do here?

I wonder if k8s can tell the container runtime from the image name? If it can, we can use container.image here, but otherwise, I guess we need another config like xxx.container to tell the container runtime and xxx.${container}.image to specify the image, e.g. xxx.container=docker and xxx.docker.image=something.

+1 to @ueshin's comment; if the same property supports more than docker images, it should be renamed to use a more generic name.

Makes sense. I think container.image should be fine then - will send a PR changing that.
I don't think there are plans to support choosing the runtime on the fly, or letting the CRI abstraction leak into the application layer.

I'd expect the container run-time to be transparent to the particular image, and very definitely transparent to spark. Using container.image makes sense from either standpoint.

#19995 PTAL

vanzin · 2017-12-14T21:16:03Z

docs/running-on-kubernetes.md

+    ./sbin/build-push-docker-images.sh -r <repo> -t my-tag build
+    ./sbin/build-push-docker-images.sh -r <repo> -t my-tag push
+
+Docker files are under the `dockerfiles/` and can be customized further before


"the dockerfiles/ directory in the Spark distribution archive"?

(Assuming that's true. Need to clarify where to find dockerfiles/.)

It is not yet true - it's a TODO item on this PR. Will clarify after we get that change in.

This has been addressed and the script has been fixed now.

vanzin · 2017-12-14T21:18:35Z

docs/running-on-kubernetes.md

+
+## Dependency Management
+
+If your application's dependencies are all hosted in remote locations like HDFS or http servers, they may be referred to


"HTTP servers"

vanzin · 2017-12-14T21:23:21Z

docs/running-on-kubernetes.md

+  <td>
+    Docker image to use for the executors. Specify this using the standard
+    <a href="https://docs.docker.com/engine/reference/commandline/tag/">Docker tag</a> format.
+    This configuration is required and must be provided by the user.


Would it make sense to have this default to the same image used for the driver? Or are they fundamentally different in some way?

They use different commands to run driver vs executor, and also have some differing environment expectations, so defaulting them to the same would not work

vanzin · 2017-12-14T21:26:20Z

docs/running-on-yarn.md

@@ -18,7 +18,8 @@ Spark application's configuration (driver, executors, and the AM when running in

 There are two deploy modes that can be used to launch Spark applications on YARN. In `cluster` mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In `client` mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.

-Unlike [Spark standalone](spark-standalone.html) and [Mesos](running-on-mesos.html) modes, in which the master's address is specified in the `--master` parameter, in YARN mode the ResourceManager's address is picked up from the Hadoop configuration. Thus, the `--master` parameter is `yarn`.
+Unlike [Spark standalone](spark-standalone.html), [Mesos](running-on-mesos.html) and [Kubernetes](running-on-kubernetes.html) modes,


Should just say "other cluster managers supported by Spark" at this point.

SparkQA · 2017-12-15T00:33:33Z

Test build #84929 has finished for PR 19946 at commit 109069f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-15T03:19:24Z

Test build #84934 has finished for PR 19946 at commit 873f04d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2017-12-18T17:54:41Z

pending updates from #19995 and #20007

…ith container ## What changes were proposed in this pull request? Changes discussed in #19946 (comment) docker -> container, since with CRI, we are not limited to running only docker images. ## How was this patch tested? Manual testing Author: foxish <[email protected]> Closes #19995 from foxish/make-docker-container.

SparkQA · 2017-12-20T12:30:26Z

Test build #85167 has finished for PR 19946 at commit 74ac5c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-20T13:40:06Z

Test build #85178 has finished for PR 19946 at commit d235847.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-20T13:46:51Z

Test build #85179 has finished for PR 19946 at commit 702162b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

liyinan926 · 2017-12-21T00:10:43Z

docs/running-on-kubernetes.md

+$ bin/spark-submit \
+    --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+    --deploy-mode cluster \
+    --name spark-pi \


We need to explicitly call out that app names in k8s mode cannot have spaces and special characters.

Good point, will update with caveat.

vanzin · 2017-12-21T00:15:42Z

BTW can I ask you guys to start using "K8S" in the PR titles? These are not really scheduler changes.

…ay to take time instead of int ## What changes were proposed in this pull request? Fixing configuration that was taking an int which should take time. Discussion in apache#19946 (comment) Made the granularity milliseconds as opposed to seconds since there's a use-case for sub-second reactions to scale-up rapidly especially with dynamic allocation. ## How was this patch tested? TODO: manual run of integration tests against this PR. PTAL cc/ mccheah liyinan926 kimoonkim vanzin mridulm jiangxb1987 ueshin Author: foxish <[email protected]> Closes apache#20032 from foxish/fix-time-conf.

foxish · 2017-12-21T00:20:15Z

Makes sense, will be sure to categorize appropriately moving forward. Thanks! On Dec 20, 2017 4:16 PM, "Marcelo Vanzin" <[email protected]> wrote: BTW can I ask you guys to start using "K8S" in the PR titles? These are not really scheduler changes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19946 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3U53sCa7IHsWMCjwYxlIWBCmkDrqUKks5tCaNYgaJpZM4Q-PQe> .

foxish · 2017-12-21T00:49:47Z

Addressed comments. This PR should be ready to go, with one separate issue of renaming to be discussed in https://issues.apache.org/jira/browse/SPARK-22853.

PTAL @vanzin @mridulm @ueshin @jiangxb1987 @felixcheung @rxin
This covers 90% of our documentation for Spark 2.3 (pending the documentation needed for #19954).

SparkQA · 2017-12-21T03:38:44Z

Test build #85222 has finished for PR 19946 at commit 8726154.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2017-12-21T06:04:51Z

docs/running-on-yarn.md

@@ -18,7 +18,8 @@ Spark application's configuration (driver, executors, and the AM when running in

 There are two deploy modes that can be used to launch Spark applications on YARN. In `cluster` mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In `client` mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.

-Unlike [Spark standalone](spark-standalone.html) and [Mesos](running-on-mesos.html) modes, in which the master's address is specified in the `--master` parameter, in YARN mode the ResourceManager's address is picked up from the Hadoop configuration. Thus, the `--master` parameter is `yarn`.
+Unlike other cluster managers supported by Spark
+in which the master's address is specified in the `--master` parameter, in YARN mode the ResourceManager's address is picked up from the Hadoop configuration. Thus, the `--master` parameter is `yarn`.


nit: why start a new line here?

SparkQA · 2017-12-21T22:35:55Z

Test build #85278 has finished for PR 19946 at commit 374ddc8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

foxish · 2017-12-21T23:52:49Z

Ping - any other comments?

jiangxb1987 · 2017-12-22T01:04:12Z

LGTM

rxin · 2017-12-22T01:21:01Z

Merging in master.

The path was recently changed in apache#19946, but the dockerfile was not updated.

## What changes were proposed in this pull request? The path was recently changed in #19946, but the dockerfile was not updated. This is a trivial 1 line fix. ## How was this patch tested? `./sbin/build-push-docker-images.sh -r spark-repo -t latest build` cc/ vanzin mridulm rxin jiangxb1987 liyinan926 Author: Anirudh Ramanathan <[email protected]> Author: foxish <[email protected]> Closes #20051 from foxish/patch-1.

foxish mentioned this pull request Dec 12, 2017

Adding running on k8s.md apache-spark-on-k8s/spark#552

Open

holdenk reviewed Dec 12, 2017

View reviewed changes

jiangxb1987 reviewed Dec 12, 2017

View reviewed changes

vanzin reviewed Dec 12, 2017

View reviewed changes

ueshin reviewed Dec 13, 2017

View reviewed changes

mridulm reviewed Dec 13, 2017

View reviewed changes

foxish force-pushed the update-k8s-docs branch from f618e8b to 8c656c5 Compare December 13, 2017 15:30

rxin reviewed Dec 14, 2017

View reviewed changes

vanzin reviewed Dec 14, 2017

View reviewed changes

foxish mentioned this pull request Dec 15, 2017

[SPARK-22807] [Scheduler] Remove config that says docker and replace with container #19995

Closed

ueshin mentioned this pull request Dec 18, 2017

[SPARK-22777][Scheduler] Kubernetes mode dockerfile permission and distribution #20007

Closed

foxish added 6 commits December 20, 2017 00:56

Adding running on k8s.md

2720c88

Address comments

4ccf59b

Adding links to running-on-kubernetes.md

14bee00

Addressed comments (round 2)

b18d1ba

review comments

9594462

Update to --name

679b5c7

foxish force-pushed the update-k8s-docs branch from 873f04d to a7e0c4c Compare December 20, 2017 08:56

foxish added 3 commits December 20, 2017 00:57

Add path to dockerfiles

74ac5c9

Test and update various options

d235847

Add <code>..</code> blocks

702162b

foxish mentioned this pull request Dec 20, 2017

[SPARK-22845] [Scheduler] Modify spark.kubernetes.allocation.batch.delay to take time instead of int #20032

Closed

liyinan926 reviewed Dec 21, 2017

View reviewed changes

foxish changed the title ~~[SPARK-22648] [Scheduler] Spark on Kubernetes - Documentation~~ [SPARK-22648] [K8S] Spark on Kubernetes - Documentation Dec 21, 2017

Update instructions for spark.app.name, modify config type

8726154

jiangxb1987 reviewed Dec 21, 2017

View reviewed changes

Better line breaks

374ddc8

asfgit closed this in 7ab165b Dec 22, 2017

foxish added a commit to foxish/spark that referenced this pull request Dec 22, 2017

[SPARK-22866] [K8S] Fix path issue in Kubernetes dockerfile

f631331

The path was recently changed in apache#19946, but the dockerfile was not updated.

foxish mentioned this pull request Dec 22, 2017

[SPARK-22866] [K8S] Fix path issue in Kubernetes dockerfile #20051

Closed


		# Configuration

		See the [configuration page](configuration.html) for information on Spark configurations. The following configuration is


		### Accessing Logs

		Logs can be accessed using the kubernetes API and the `kubectl` CLI. When a Spark application is running, it's possible


		## Dependency Management

		If your application's dependencies are all hosted in remote locations like HDFS or http servers, they may be referred to

[SPARK-22648] [K8S] Spark on Kubernetes - Documentation #19946

[SPARK-22648] [K8S] Spark on Kubernetes - Documentation #19946

Conversation

foxish commented Dec 12, 2017 • edited Loading

holdenk commented Dec 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiangxb1987 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foxish commented Dec 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foxish Dec 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ueshin commented Dec 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Dec 13, 2017

mridulm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mridulm Dec 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foxish commented Dec 12, 2017 •

edited

Loading

foxish Dec 13, 2017 •

edited

Loading

mridulm Dec 13, 2017 •

edited

Loading