[SPARK-20401][DOC]In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary. #17696

guoxiaolongzte · 2017-04-20T06:36:09Z

What changes were proposed in this pull request?

Use the REST interface submits the spark job.
e.g.
curl -X POST http://10.43.183.120:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data'{
"action": "CreateSubmissionRequest",
"appArgs": [
"myAppArgument"
],
"appResource": "/home/mr/gxl/test.jar",
"clientSparkVersion": "2.2.0",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"
},
"mainClass": "cn.zte.HdfsTest",
"sparkProperties": {
"spark.jars": "/home/mr/gxl/test.jar",
"spark.driver.supervise": "true",
"spark.app.name": "HdfsTest",
"spark.eventLog.enabled": "false",
"spark.submit.deployMode": "cluster",
"spark.master": "spark://10.43.183.120:6066"
}
}'

I hope that make sure that the driver is automatically restarted if it fails with non-zero exit code.
But I can not find the 'spark.driver.supervise' configuration parameter specification and default values from the spark official document.

How was this patch tested?

manual tests

Please review http://spark.apache.org/contributing.html before opening a pull request.

…anges.

…ucceeded|failed|unknown]

…0177

…ion.

…remove redundant description.

….driver.supervise' configuration parameter specification and default values are necessary.

HyukjinKwon · 2017-04-20T07:04:12Z

Not a big deal. Waht do you think about wrapping the command you ran with ``` ... ``` in the PR description?

curl -X POST http://10.43.183.120:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
  ...
}

I think this makes the PR description more readable.

guoxiaolongzte · 2017-04-20T07:20:54Z

@HyukjinKwon
But I wrote, no use ...
My computer is window7 system.
The browser is Chrome.

HyukjinKwon · 2017-04-20T07:24:25Z

I will probably test it by myself later. Thanks for taking my comment into account.

guoxiaolongzte · 2017-04-20T07:28:28Z

Please cut a picture, I see how you show on your computer?

HyukjinKwon · 2017-04-20T07:30:27Z

Yup, I will let you know when I make it. I think this does not block this PR BTW :).

SparkQA · 2017-04-20T07:35:49Z

Test build #3666 has finished for PR 17696 at commit 3e74d73.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2017-04-21T07:57:15Z

docs/configuration.md

+  <td><code>spark.driver.supervise</code></td>
+  <td>false</td>
+  <td>
+    If value set true, restarts the driver on failure, make sure that


This sentence needs to be simplified. "If true, restarts the driver automatically if it fails with a non-zero exit status."

HyukjinKwon · 2017-04-21T08:03:29Z

@guoxiaolongzte, I just tested the backquotes on Windows 7 with Chrome as below:

It is a backquote ` not a single-quote '. Could you try again? I see you made many PRs and I just would like them to be more readable.

HyukjinKwon · 2017-04-21T08:06:43Z

docs/configuration.md

+  <td>
+    If value set true, restarts the driver on failure, make sure that
+    the driver is automatically restarted if it fails with non-zero exit code.
+    Only in Spark standalone or Mesos with cluster deploy mode.


I am sorry for asking such a question @srowen. Is it the right sentence in English?

It'd be better to say "Only has effect in Spark standalone mode or Mesos cluster deploy mode."

OK,I will fix it.Thank you.

guoxiaolongzte · 2017-04-21T08:42:40Z

@HyukjinKwon
I will try to improve my ability to translate, sorry.

SparkQA · 2017-04-21T17:49:58Z

Test build #3671 has finished for PR 17696 at commit 16655db.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2017-04-21T19:08:56Z

Merged to master/2.2

…'spark.driver.supervise' configuration parameter specification and default values are necessary. ## What changes were proposed in this pull request? Use the REST interface submits the spark job. e.g. curl -X POST http://10.43.183.120:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data'{ "action": "CreateSubmissionRequest", "appArgs": [ "myAppArgument" ], "appResource": "/home/mr/gxl/test.jar", "clientSparkVersion": "2.2.0", "environmentVariables": { "SPARK_ENV_LOADED": "1" }, "mainClass": "cn.zte.HdfsTest", "sparkProperties": { "spark.jars": "/home/mr/gxl/test.jar", **"spark.driver.supervise": "true",** "spark.app.name": "HdfsTest", "spark.eventLog.enabled": "false", "spark.submit.deployMode": "cluster", "spark.master": "spark://10.43.183.120:6066" } }' **I hope that make sure that the driver is automatically restarted if it fails with non-zero exit code. But I can not find the 'spark.driver.supervise' configuration parameter specification and default values from the spark official document.** ## How was this patch tested? manual tests Please review http://spark.apache.org/contributing.html before opening a pull request. Author: 郭小龙 10207633 <[email protected]> Author: guoxiaolong <[email protected]> Author: guoxiaolongzte <[email protected]> Closes #17696 from guoxiaolongzte/SPARK-20401. (cherry picked from commit ad29040) Signed-off-by: Sean Owen <[email protected]>

…'spark.driver.supervise' configuration parameter specification and default values are necessary. ## What changes were proposed in this pull request? Use the REST interface submits the spark job. e.g. curl -X POST http://10.43.183.120:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data'{ "action": "CreateSubmissionRequest", "appArgs": [ "myAppArgument" ], "appResource": "/home/mr/gxl/test.jar", "clientSparkVersion": "2.2.0", "environmentVariables": { "SPARK_ENV_LOADED": "1" }, "mainClass": "cn.zte.HdfsTest", "sparkProperties": { "spark.jars": "/home/mr/gxl/test.jar", **"spark.driver.supervise": "true",** "spark.app.name": "HdfsTest", "spark.eventLog.enabled": "false", "spark.submit.deployMode": "cluster", "spark.master": "spark://10.43.183.120:6066" } }' **I hope that make sure that the driver is automatically restarted if it fails with non-zero exit code. But I can not find the 'spark.driver.supervise' configuration parameter specification and default values from the spark official document.** ## How was this patch tested? manual tests Please review http://spark.apache.org/contributing.html before opening a pull request. Author: 郭小龙 10207633 <[email protected]> Author: guoxiaolong <[email protected]> Author: guoxiaolongzte <[email protected]> Closes apache#17696 from guoxiaolongzte/SPARK-20401.

郭小龙 10207633 added 17 commits March 31, 2017 21:57

[SPARK-20177]Document about compression way has some little detail ch…

d383efb

…anges.

[SPARK-20177] event log add a space

3059013

'/applications/[app-id]/jobs' in rest api,status should be [running|s…

555cef8

…ucceeded|failed|unknown]

Merge branch 'master' of https://github.com/apache/spark into SPARK-2…

46bb1ad

…0177

[SPARK-20218]'/applications/[app-id]/stages' in REST API,add descript…

0efb0dd

…ion.

[SPARK-20218] '/applications/[app-id]/stages/[stage-id]' in REST API,…

0e37fde

…remove redundant description.

Merge branch 'SPARK-20218'

52641bb

Merge branch 'master' of https://github.com/apache/spark

d3977c9

Merge branch 'master' of https://github.com/apache/spark

137b90e

Merge branch 'SPARK-20190' of https://github.com/guoxiaolongzte/spark

0fe5865

Merge branch 'master' of https://github.com/apache/spark

cf6f42a

Merge branch 'master' of https://github.com/apache/spark

685cd6b

Merge branch 'master' of https://github.com/apache/spark

c716a92

Merge branch 'master' of https://github.com/apache/spark

679cec3

Merge branch 'master' of https://github.com/apache/spark

3c9387a

Merge branch 'master' of https://github.com/apache/spark

cb71f44

[SPARK-20401]In the spark official configuration document, the 'spark…

3e74d73

….driver.supervise' configuration parameter specification and default values are necessary.

srowen approved these changes Apr 20, 2017

View reviewed changes

srowen reviewed Apr 21, 2017

View reviewed changes

HyukjinKwon reviewed Apr 21, 2017

View reviewed changes

[SPARK-20401]This sentence has be simplified.

16655db

asfgit closed this in ad29040 Apr 21, 2017

guoxiaolongzte deleted the SPARK-20401 branch June 12, 2017 10:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-20401][DOC]In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary. #17696

[SPARK-20401][DOC]In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary. #17696

guoxiaolongzte commented Apr 20, 2017

HyukjinKwon commented Apr 20, 2017

guoxiaolongzte commented Apr 20, 2017

HyukjinKwon commented Apr 20, 2017 •

edited

Loading

guoxiaolongzte commented Apr 20, 2017

HyukjinKwon commented Apr 20, 2017

SparkQA commented Apr 20, 2017

srowen Apr 21, 2017

HyukjinKwon commented Apr 21, 2017

HyukjinKwon Apr 21, 2017

srowen Apr 21, 2017

guoxiaolongzte Apr 21, 2017 •

edited

Loading

guoxiaolongzte commented Apr 21, 2017

SparkQA commented Apr 21, 2017

srowen commented Apr 21, 2017

[SPARK-20401][DOC]In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary. #17696

[SPARK-20401][DOC]In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary. #17696

Conversation

guoxiaolongzte commented Apr 20, 2017

What changes were proposed in this pull request?

How was this patch tested?

HyukjinKwon commented Apr 20, 2017

guoxiaolongzte commented Apr 20, 2017

HyukjinKwon commented Apr 20, 2017 • edited Loading

guoxiaolongzte commented Apr 20, 2017

HyukjinKwon commented Apr 20, 2017

SparkQA commented Apr 20, 2017

srowen Apr 21, 2017

Choose a reason for hiding this comment

HyukjinKwon commented Apr 21, 2017

HyukjinKwon Apr 21, 2017

Choose a reason for hiding this comment

srowen Apr 21, 2017

Choose a reason for hiding this comment

guoxiaolongzte Apr 21, 2017 • edited Loading

Choose a reason for hiding this comment

guoxiaolongzte commented Apr 21, 2017

SparkQA commented Apr 21, 2017

srowen commented Apr 21, 2017

HyukjinKwon commented Apr 20, 2017 •

edited

Loading

guoxiaolongzte Apr 21, 2017 •

edited

Loading