Skip to content

Commit

Permalink
[SPARK-29148][CORE][FOLLOW-UP] Don't dynamic allocation warning when …
Browse files Browse the repository at this point in the history
…it's disabled

### What changes were proposed in this pull request?

Currently, after apache#27313, it shows the warning about dynamic allocation which is disabled by default.

```bash
$ ./bin/spark-shell
```

```
...
20/02/18 11:04:56 WARN ResourceProfile: Please ensure that the number of slots available on your executors is
limited by the number of cores to task cpus and not another custom resource. If cores is not the limiting resource
then dynamic allocation will not work properly!
```

This PR brings back the configuration checking for this warning. Seems mistakenly removed at https://github.com/apache/spark/pull/27313/files#diff-364713d7776956cb8b0a771e9b62f82dL2841

### Why are the changes needed?

To remove false warning.

### Does this PR introduce any user-facing change?

Yes, it will don't show the warning. It's master only change so no user-facing to end users.

### How was this patch tested?

Manually tested.

Closes apache#27615 from HyukjinKwon/SPARK-29148.

Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
HyukjinKwon authored and Seongjin Cho committed Apr 14, 2020
1 parent ed1d7e7 commit 2653a43
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ class ResourceProfile(
"no corresponding task resource request was specified.")
}
}
if(!shouldCheckExecCores) {
if(!shouldCheckExecCores && Utils.isDynamicAllocationEnabled(sparkConf)) {
// if we can't rely on the executor cores config throw a warning for user
logWarning("Please ensure that the number of slots available on your " +
"executors is limited by the number of cores to task cpus and not another " +
Expand Down

0 comments on commit 2653a43

Please sign in to comment.