[SPARK-6490][Core] Add spark.rpc.* and deprecate spark.akka.* #5595
Conversation
|
Test build #30590 has finished for PR 5595 at commit
|
|
|
||
| /** Returns the configured number of milliseconds to wait on each retry */ | ||
| def retryWaitMs(conf: SparkConf): Long = { | ||
| conf.getLong("spark.rpc.retry.wait", 3000) |
There was a problem hiding this comment.
Since we're here, could you use the new API:
conf.getTimeAsMs("spark.rpc.retry.wait", "3s")
Similar in other methods in this file.
|
I don't see these documented anywhere, so aside from needing an update to use the new conf API, this LGTM. |
|
|
||
| /** Returns the configured number of times to retry connecting */ | ||
| def numRetries(conf: SparkConf): Int = { | ||
| conf.getInt("spark.rpc.num.retries", 3) |
There was a problem hiding this comment.
we should do spark.rpc.numRetries instead (you might need to translate this to akka.num.retries if we pass this back to akka)
|
We should also update the documentation page, as @vanzin pointed out. That can be done as a follow-up pull request too. |
|
Test build #30626 has finished for PR 5595 at commit
|
|
Merging in master. Thanks. |
|
Can you create a PR to update documentation? |
Updated docs in #5607.
Just remind that the default network timeout isn't used now. |
Added docs for rpc configurations and also fixed two places that should have been fixed in #5595. Author: zsxwing <[email protected]> Closes #5607 from zsxwing/SPARK-6490-docs and squashes the following commits: 25a6736 [zsxwing] Increase the default timeout to 120s 6e37c30 [zsxwing] Update docs 5577540 [zsxwing] Use spark.network.timeout as the default timeout if it presents 4f07174 [zsxwing] Fix unit tests 1c2cf26 [zsxwing] Add docs for rpc configurations
Deprecated `spark.akka.num.retries`, `spark.akka.retry.wait`, `spark.akka.askTimeout`, `spark.akka.lookupTimeout`, and added `spark.rpc.num.retries`, `spark.rpc.retry.wait`, `spark.rpc.askTimeout`, `spark.rpc.lookupTimeout`. Author: zsxwing <[email protected]> Closes apache#5595 from zsxwing/SPARK-6490 and squashes the following commits: e0d80a9 [zsxwing] Use getTimeAsMs and getTimeAsSeconds and other minor fixes 31dbe69 [zsxwing] Add spark.rpc.* and deprecate spark.akka.*
Added docs for rpc configurations and also fixed two places that should have been fixed in apache#5595. Author: zsxwing <[email protected]> Closes apache#5607 from zsxwing/SPARK-6490-docs and squashes the following commits: 25a6736 [zsxwing] Increase the default timeout to 120s 6e37c30 [zsxwing] Update docs 5577540 [zsxwing] Use spark.network.timeout as the default timeout if it presents 4f07174 [zsxwing] Fix unit tests 1c2cf26 [zsxwing] Add docs for rpc configurations
Deprecated
spark.akka.num.retries,spark.akka.retry.wait,spark.akka.askTimeout,spark.akka.lookupTimeout, and addedspark.rpc.num.retries,spark.rpc.retry.wait,spark.rpc.askTimeout,spark.rpc.lookupTimeout.