[SPARK-29148][CORE][FOLLOW-UP] Don't dynamic allocation warning when it's disabled #27615
Conversation
|
@tgravescs mind taking a look please? |
|
Test build #118609 has finished for PR 27615 at commit
|
|
Jenkins, retest this please. |
|
Test build #118627 has finished for PR 27615 at commit
|
There was a problem hiding this comment.
+1, LGTM. Merged to master.
Since this is a warning, I verified this manually, too.
Thank you, @HyukjinKwon and @Ngone51 .
|
Thanks all! |
|
sorry was out on vacation, actually I found that this is affected by more then dynamic allocation which is why it was there all the time. The core scheduler code with barrier scheduling for instance relies on slots, there is other code sprinkled throughout that looks at "slots" that assumes cpus are limiting factor. So really I'm more for either doing more checking if custom resource specified or put the warning back with clarification. |
|
yeah so looked at it a bit more the check should only warn when custom resources were specified and it should be there all the time and remove the sentence about dynamic allocation. I'll put up a pr once jira comes back up |
|
Thanks @tgravescs. |
…it's disabled ### What changes were proposed in this pull request? Currently, after apache#27313, it shows the warning about dynamic allocation which is disabled by default. ```bash $ ./bin/spark-shell ``` ``` ... 20/02/18 11:04:56 WARN ResourceProfile: Please ensure that the number of slots available on your executors is limited by the number of cores to task cpus and not another custom resource. If cores is not the limiting resource then dynamic allocation will not work properly! ``` This PR brings back the configuration checking for this warning. Seems mistakenly removed at https://github.com/apache/spark/pull/27313/files#diff-364713d7776956cb8b0a771e9b62f82dL2841 ### Why are the changes needed? To remove false warning. ### Does this PR introduce any user-facing change? Yes, it will don't show the warning. It's master only change so no user-facing to end users. ### How was this patch tested? Manually tested. Closes apache#27615 from HyukjinKwon/SPARK-29148. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
What changes were proposed in this pull request?
Currently, after #27313, it shows the warning about dynamic allocation which is disabled by default.
This PR brings back the configuration checking for this warning. Seems mistakenly removed at https://github.com/apache/spark/pull/27313/files#diff-364713d7776956cb8b0a771e9b62f82dL2841
Why are the changes needed?
To remove false warning.
Does this PR introduce any user-facing change?
Yes, it will don't show the warning. It's master only change so no user-facing to end users.
How was this patch tested?
Manually tested.