Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-45670][CORE] SparkSubmit does not support --total-executor-cores when deploying on K8s #43536

Closed
wants to merge 1 commit into from
Closed

[SPARK-45670][CORE] SparkSubmit does not support --total-executor-cores when deploying on K8s #43536

wants to merge 1 commit into from

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Oct 26, 2023 •
edited
Loading

What changes were proposed in this pull request?

Remove Kubernetes from the support list of --total-executor-cores in SparkSubmit

Why are the changes needed?

--total-executor-cores does not take effect in Spark on K8s, the comments from original PR also proves that

Does this PR introduce any user-facing change?

The output of spark-submit --help changed

...
-  Spark standalone and Kubernetes only:
+  Spark standalone only:
    --total-executor-cores NUM  Total cores for all executors.
...

How was this patch tested?

Pass GA and review.

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the CORE label Oct 26, 2023
HyukjinKwon
HyukjinKwon reviewed Oct 26, 2023
View reviewed changes
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
@@ -566,7 +566,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
| --kill SUBMISSION_ID If given, kills the driver specified.
| --status SUBMISSION_ID If given, requests the status of the driver specified.
|
| Spark standalone and Kubernetes only:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was added from #23518 @LucaCanali FYI

Copy link
Member

@HyukjinKwon HyukjinKwon Oct 26, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@viirya and @dongjoon-hyun

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me. --total-executor-cores is assigned to spark.cores.max which I don't see it is used in K8s. Maybe @dongjoon-hyun can confirm it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are correct, @viirya . It seems that #23518 only checked the following code.

spark/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

Line 677 in 8c11804

OptionAssigner(args.totalExecutorCores, STANDALONE | KUBERNETES, ALL_DEPLOY_MODES,

dongjoon-hyun
dongjoon-hyun approved these changes Oct 26, 2023
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@dongjoon-hyun
Copy link
Member

Merged to master. Could you make backporting PRs, please, @pan3793 ?

pan3793 added a commit to pan3793/spark that referenced this pull request Oct 27, 2023
@pan3793
[SPARK-45670][CORE] SparkSubmit does not support `--total-executor-co…
4604018 
…res` when deploying on K8s

Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit

`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original PR](apache#19717 (comment)) also proves that

The output of `spark-submit --help` changed

```patch
...
-  Spark standalone, Mesos and Kubernetes only:
+  Spark standalone and Mesos only:
    --total-executor-cores NUM  Total cores for all executors.
...
```

Pass GA and review.

No

Closes apache#43536 from pan3793/tec.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
pan3793 added a commit to pan3793/spark that referenced this pull request Oct 27, 2023
@pan3793
[SPARK-45670][CORE] SparkSubmit does not support `--total-executor-co…
8b14629 
…res` when deploying on K8s

Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit

`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original PR](apache#19717 (comment)) also proves that

The output of `spark-submit --help` changed

```patch
...
-  Spark standalone, Mesos and Kubernetes only:
+  Spark standalone and Mesos only:
    --total-executor-cores NUM  Total cores for all executors.
...
```

Pass GA and review.

No

Closes apache#43536 from pan3793/tec.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
pan3793 added a commit to pan3793/spark that referenced this pull request Oct 27, 2023
@pan3793
[SPARK-45670][CORE] SparkSubmit does not support `--total-executor-co…
42874a5 
…res` when deploying on K8s

Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit

`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original PR](apache#19717 (comment)) also proves that

The output of `spark-submit --help` changed

```patch
...
-  Spark standalone, Mesos and Kubernetes only:
+  Spark standalone and Mesos only:
    --total-executor-cores NUM  Total cores for all executors.
...
```

Pass GA and review.

No

Closes apache#43536 from pan3793/tec.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
This was referenced Oct 27, 2023
Closed
Closed
Closed
@pan3793
Copy link
Member Author

pan3793 commented Oct 27, 2023

Thanks @dongjoon-hyun and all. I opened the following PRs for previous branches

HyukjinKwon pushed a commit that referenced this pull request Oct 27, 2023
@pan3793 @HyukjinKwon
[SPARK-45670][CORE][3.3] SparkSubmit does not support `--total-execut…
94a043d 
…or-cores` when deploying on K8s

This is the cherry-pick of #43536 for branch-3.3

### What changes were proposed in this pull request?

Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit

### Why are the changes needed?

`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original PR](#19717 (comment)) also proves that

### Does this PR introduce _any_ user-facing change?

The output of `spark-submit --help` changed

```patch
...
-  Spark standalone, Mesos and Kubernetes only:
+  Spark standalone and Mesos only:
    --total-executor-cores NUM  Total cores for all executors.
...
```
### How was this patch tested?

Pass GA and review.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #43548 from pan3793/SPARK-45670-3.3.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
HyukjinKwon pushed a commit that referenced this pull request Oct 27, 2023
@pan3793 @HyukjinKwon
[SPARK-45670][CORE][3.4] SparkSubmit does not support `--total-execut…
9c5d7c4 
…or-cores` when deploying on K8s

This is the cherry-pick of #43536 for branch-3.4

### What changes were proposed in this pull request?

Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit

### Why are the changes needed?

`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original PR](#19717 (comment)) also proves that

### Does this PR introduce _any_ user-facing change?

The output of `spark-submit --help` changed

```patch
...
-  Spark standalone, Mesos and Kubernetes only:
+  Spark standalone and Mesos only:
    --total-executor-cores NUM  Total cores for all executors.
...
```
### How was this patch tested?

Pass GA and review.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #43549 from pan3793/SPARK-45670-3.4.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
HyukjinKwon pushed a commit that referenced this pull request Oct 27, 2023
@pan3793 @HyukjinKwon
[SPARK-45670][CORE][3.5] SparkSubmit does not support `--total-execut…
e2b92b8 
…or-cores` when deploying on K8s

This is the cherry-pick of #43536 for branch-3.5

### What changes were proposed in this pull request?

Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit

### Why are the changes needed?

`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original PR](#19717 (comment)) also proves that

### Does this PR introduce _any_ user-facing change?

The output of `spark-submit --help` changed

```patch
...
-  Spark standalone, Mesos and Kubernetes only:
+  Spark standalone and Mesos only:
    --total-executor-cores NUM  Total cores for all executors.
...
```
### How was this patch tested?

Pass GA and review.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #43550 from pan3793/SPARK-45670-3.5.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Feb 7, 2024
@pan3793 @viirya
[SPARK-45670][CORE][3.4] SparkSubmit does not support `--total-execut…
ccd4e9d 
…or-cores` when deploying on K8s

This is the cherry-pick of apache#43536 for branch-3.4

### What changes were proposed in this pull request?

Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit

### Why are the changes needed?

`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original PR](apache#19717 (comment)) also proves that

### Does this PR introduce _any_ user-facing change?

The output of `spark-submit --help` changed

```patch
...
-  Spark standalone, Mesos and Kubernetes only:
+  Spark standalone and Mesos only:
    --total-executor-cores NUM  Total cores for all executors.
...
```
### How was this patch tested?

Pass GA and review.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes apache#43549 from pan3793/SPARK-45670-3.4.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
@pan3793 pan3793 deleted the tec branch March 1, 2024 05:13
This was referenced Sep 4, 2024
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments

@viirya viirya viirya left review comments

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

@yaooqinn yaooqinn yaooqinn approved these changes

Assignees
No one assigned
Labels
CORE
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@pan3793 @dongjoon-hyun @viirya @HyukjinKwon @yaooqinn