Upgrade to Spark 3.5.2 #2154

jacobsalway · 2024-09-09T03:16:25Z

Purpose

Upgrade from Spark 3.5.0 to Spark 3.5.2 in both the Dockerfile and examples.

Testing

The example JAR is present on the 3.5.2 images and examples/spark-pi-python.yaml ran on Kind.
There is nothing in particular I can see in the Spark 3.5.1 or 3.5.2 changelogs that would impact how the operator works (there is [SPARK-45670][CORE] SparkSubmit does not support --total-executor-cores when deploying on K8s apache/spark#43536 but this flag isn't used when constructing the spark-submit command internally)

Change Category

Indicate the type of change by marking the applicable boxes:

Bugfix (non-breaking change which fixes an issue)
Feature (non-breaking change which adds functionality)
Breaking change (fix or feature that could affect existing functionality)
Documentation update

Checklist

Before submitting your PR, please review the following:

I have conducted a self-review of my own code.
I have updated documentation accordingly.
I have added tests that prove my changes are effective or that my feature works.
Existing unit tests pass locally with my changes.

ChenYi015 · 2024-09-09T04:51:22Z

@jacobsalway The examples under config/samples also need to be updated.

Signed-off-by: Jacob Salway <[email protected]>

jacobsalway · 2024-09-09T05:01:41Z

@jacobsalway The examples under config/samples also need to be updated.

Done, didn't realise we had any sample applications outside examples/

ChenYi015 · 2024-09-09T05:05:58Z

/lgtm
/approve

google-oss-prow · 2024-09-09T05:06:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit e1b7a27)

Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit e1b7a27) Signed-off-by: Yi Chen <[email protected]>

* Support gang scheduling with Yunikorn (#2107) * Add Yunikorn scheduler and example Signed-off-by: Jacob Salway <[email protected]> * Add test cases Signed-off-by: Jacob Salway <[email protected]> * Add code comments Signed-off-by: Jacob Salway <[email protected]> * Add license comment Signed-off-by: Jacob Salway <[email protected]> * Inline mergeNodeSelector Signed-off-by: Jacob Salway <[email protected]> * Fix initial number implementation Signed-off-by: Jacob Salway <[email protected]> --------- Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 8fcda12) Signed-off-by: Yi Chen <[email protected]> * Update Makefile for building sparkctl (#2119) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 4bc6e89) Signed-off-by: Yi Chen <[email protected]> * fix: Add default values for namespaces to match usage descriptions (#2128) * fix: Add default values for namespaces to match usage descriptions Signed-off-by: pengfei4.li <[email protected]> * fix: remove incorrect cache settings Signed-off-by: pengfei4.li <[email protected]> --------- Signed-off-by: pengfei4.li <[email protected]> Co-authored-by: pengfei4.li <[email protected]> (cherry picked from commit 52f818d) Signed-off-by: Yi Chen <[email protected]> * Fix: Spark role binding did not render properly when setting spark service account name (#2135) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit a1a38ea) Signed-off-by: Yi Chen <[email protected]> * Reintroduce option webhook.enable (#2142) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 9e88049) Signed-off-by: Yi Chen <[email protected]> * Add default batch scheduler argument (#2143) * Add default batch scheduler argument Signed-off-by: Jacob Salway <[email protected]> * Add helm unit test Signed-off-by: Jacob Salway <[email protected]> --------- Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 9cc1c02) Signed-off-by: Yi Chen <[email protected]> * fix: unable to set controller/webhook replicas to zero (#2147) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 1afa72e) Signed-off-by: Yi Chen <[email protected]> * Adding support for setting spark job namespaces to all namespaces (#2123) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit c93b0ec) Signed-off-by: Yi Chen <[email protected]> * Support extended kube-scheduler as batch scheduler (#2136) * Support coscheduling with kube-scheduler plugins Signed-off-by: Yi Chen <[email protected]> * Add example for using kube-schulder coscheduling Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit e8d3de9) Signed-off-by: Yi Chen <[email protected]> * Run e2e tests on Kind (#2148) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit c810ece) Signed-off-by: Yi Chen <[email protected]> * Set schedulerName to Yunikorn (#2153) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 62b4ca6) Signed-off-by: Yi Chen <[email protected]> * Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (#2129) watching all namespaces Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 592b649) Signed-off-by: Yi Chen <[email protected]> * Fix: e2e test failes due to webhook not ready (#2149) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit dee91ba) Signed-off-by: Yi Chen <[email protected]> * Upgrade to Go 1.23.1 (#2155) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 10fcb8e) Signed-off-by: Yi Chen <[email protected]> * Upgrade to Spark 3.5.2 (#2154) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit e1b7a27) Signed-off-by: Yi Chen <[email protected]> * Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (#2159) Bumps [sigs.k8s.io/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) from 0.29.7 to 0.29.8. - [Release notes](https://github.com/kubernetes-sigs/scheduler-plugins/releases) - [Changelog](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/RELEASE.md) - [Commits](kubernetes-sigs/scheduler-plugins@v0.29.7...v0.29.8) --- updated-dependencies: - dependency-name: sigs.k8s.io/scheduler-plugins dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 95d202e) Signed-off-by: Yi Chen <[email protected]> * feat: support driver and executor pod use different priority (#2146) * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <[email protected]> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <[email protected]> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <[email protected]> * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> * feat: add adjust pointer if is nil Signed-off-by: Kevin.Wu <[email protected]> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <[email protected]> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <[email protected]> * feat: Optimize code to avoid null pointer exceptions Signed-off-by: Kevin.Wu <[email protected]> * fix: remove backup crd files Signed-off-by: Kevin.Wu <[email protected]> * fix: remove BatchSchedulerOptions.PriorityClassName test code Signed-off-by: Kevin Wu <[email protected]> * fix: add driver and executor pod priorityClassName test code Signed-off-by: Kevin Wu <[email protected]> --------- Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> Co-authored-by: Kevin Wu <[email protected]> (cherry picked from commit 6ae1b2f) Signed-off-by: Yi Chen <[email protected]> * Bump gocloud.dev from 0.37.0 to 0.39.0 (#2160) Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.37.0 to 0.39.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](google/go-cloud@v0.37.0...v0.39.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit e58023b) Signed-off-by: Yi Chen <[email protected]> * Update e2e tests (#2161) * Add sleep buffer to ensture the webhooks are ready before running the e2e tests Signed-off-by: Yi Chen <[email protected]> * Remove duplicate operator image build tasks Signed-off-by: Yi Chen <[email protected]> * Update e2e tests Signed-off-by: Yi Chen <[email protected]> * Update examples Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit e6a7805) Signed-off-by: Yi Chen <[email protected]> * fix: webhook not working when settings spark job namespaces to empty (#2163) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 7785107) Signed-off-by: Yi Chen <[email protected]> * fix: The logger had an odd number of arguments, making it panic (#2166) Signed-off-by: tcassaert <[email protected]> (cherry picked from commit eb48b34) Signed-off-by: Yi Chen <[email protected]> * Upgrade to Spark 3.5.2(#2012) (#2157) * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <[email protected]> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <[email protected]> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <[email protected]> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <[email protected]> --------- Signed-off-by: HyukSangCho <[email protected]> (cherry picked from commit 9f0c08a) Signed-off-by: Yi Chen <[email protected]> * Feature: Add pprof endpoint (#2164) * add pprof support to the operator Controller Manager Signed-off-by: ImpSy <[email protected]> * add pprof support to helm chart Signed-off-by: ImpSy <[email protected]> --------- Signed-off-by: ImpSy <[email protected]> (cherry picked from commit 75b9266) Signed-off-by: Yi Chen <[email protected]> * fix the make kind-delete-custer to avoid accidental kubeconfig deletion (#2172) Signed-off-by: ImpSy <[email protected]> (cherry picked from commit cbfefd5) Signed-off-by: Yi Chen <[email protected]> * Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (#2174) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.27 to 1.27.33. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.33) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit b818332) Signed-off-by: Yi Chen <[email protected]> * Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (#2173) Bumps [helm.sh/helm/v3](https://github.com/helm/helm) from 3.15.3 to 3.16.1. - [Release notes](https://github.com/helm/helm/releases) - [Commits](helm/helm@v3.15.3...v3.16.1) --- updated-dependencies: - dependency-name: helm.sh/helm/v3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit f3f80d4) Signed-off-by: Yi Chen <[email protected]> * Add specific error in log line when failed to create web UI service (#2170) * Add specific error in log line when failed to create web UI service Signed-off-by: tcassaert <[email protected]> * Update log to reflect correct resource that could not be created Co-authored-by: Yi Chen <[email protected]> Signed-off-by: tcassaert <[email protected]> --------- Signed-off-by: tcassaert <[email protected]> Signed-off-by: tcassaert <[email protected]> Co-authored-by: Yi Chen <[email protected]> (cherry picked from commit ed3226e) Signed-off-by: Yi Chen <[email protected]> * Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (#2178) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit a2f71c6) Signed-off-by: Yi Chen <[email protected]> * Fix: spark application does not respect time to live seconds (#2165) * Add time to live seconds example spark application Signed-off-by: Yi Chen <[email protected]> * fix: spark application does not respect time to live seconds Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit c855ee4) Signed-off-by: Yi Chen <[email protected]> * Update release workflow and docs (#2121) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit bca6aa8) Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Jacob Salway <[email protected]> Signed-off-by: Yi Chen <[email protected]> Signed-off-by: pengfei4.li <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> Signed-off-by: tcassaert <[email protected]> Signed-off-by: HyukSangCho <[email protected]> Signed-off-by: ImpSy <[email protected]> Signed-off-by: tcassaert <[email protected]> Co-authored-by: Jacob Salway <[email protected]> Co-authored-by: Neo <[email protected]> Co-authored-by: pengfei4.li <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevinz <[email protected]> Co-authored-by: Kevin Wu <[email protected]> Co-authored-by: tcassaert <[email protected]> Co-authored-by: ha2hi <[email protected]> Co-authored-by: Sébastien Maintrot <[email protected]>

* Support gang scheduling with Yunikorn (kubeflow#2107) * Add Yunikorn scheduler and example Signed-off-by: Jacob Salway <[email protected]> * Add test cases Signed-off-by: Jacob Salway <[email protected]> * Add code comments Signed-off-by: Jacob Salway <[email protected]> * Add license comment Signed-off-by: Jacob Salway <[email protected]> * Inline mergeNodeSelector Signed-off-by: Jacob Salway <[email protected]> * Fix initial number implementation Signed-off-by: Jacob Salway <[email protected]> --------- Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 8fcda12) Signed-off-by: Yi Chen <[email protected]> * Update Makefile for building sparkctl (kubeflow#2119) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 4bc6e89) Signed-off-by: Yi Chen <[email protected]> * fix: Add default values for namespaces to match usage descriptions (kubeflow#2128) * fix: Add default values for namespaces to match usage descriptions Signed-off-by: pengfei4.li <[email protected]> * fix: remove incorrect cache settings Signed-off-by: pengfei4.li <[email protected]> --------- Signed-off-by: pengfei4.li <[email protected]> Co-authored-by: pengfei4.li <[email protected]> (cherry picked from commit 52f818d) Signed-off-by: Yi Chen <[email protected]> * Fix: Spark role binding did not render properly when setting spark service account name (kubeflow#2135) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit a1a38ea) Signed-off-by: Yi Chen <[email protected]> * Reintroduce option webhook.enable (kubeflow#2142) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 9e88049) Signed-off-by: Yi Chen <[email protected]> * Add default batch scheduler argument (kubeflow#2143) * Add default batch scheduler argument Signed-off-by: Jacob Salway <[email protected]> * Add helm unit test Signed-off-by: Jacob Salway <[email protected]> --------- Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 9cc1c02) Signed-off-by: Yi Chen <[email protected]> * fix: unable to set controller/webhook replicas to zero (kubeflow#2147) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 1afa72e) Signed-off-by: Yi Chen <[email protected]> * Adding support for setting spark job namespaces to all namespaces (kubeflow#2123) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit c93b0ec) Signed-off-by: Yi Chen <[email protected]> * Support extended kube-scheduler as batch scheduler (kubeflow#2136) * Support coscheduling with kube-scheduler plugins Signed-off-by: Yi Chen <[email protected]> * Add example for using kube-schulder coscheduling Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit e8d3de9) Signed-off-by: Yi Chen <[email protected]> * Run e2e tests on Kind (kubeflow#2148) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit c810ece) Signed-off-by: Yi Chen <[email protected]> * Set schedulerName to Yunikorn (kubeflow#2153) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 62b4ca6) Signed-off-by: Yi Chen <[email protected]> * Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (kubeflow#2129) watching all namespaces Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 592b649) Signed-off-by: Yi Chen <[email protected]> * Fix: e2e test failes due to webhook not ready (kubeflow#2149) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit dee91ba) Signed-off-by: Yi Chen <[email protected]> * Upgrade to Go 1.23.1 (kubeflow#2155) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 10fcb8e) Signed-off-by: Yi Chen <[email protected]> * Upgrade to Spark 3.5.2 (kubeflow#2154) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit e1b7a27) Signed-off-by: Yi Chen <[email protected]> * Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (kubeflow#2159) Bumps [sigs.k8s.io/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) from 0.29.7 to 0.29.8. - [Release notes](https://github.com/kubernetes-sigs/scheduler-plugins/releases) - [Changelog](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/RELEASE.md) - [Commits](kubernetes-sigs/scheduler-plugins@v0.29.7...v0.29.8) --- updated-dependencies: - dependency-name: sigs.k8s.io/scheduler-plugins dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 95d202e) Signed-off-by: Yi Chen <[email protected]> * feat: support driver and executor pod use different priority (kubeflow#2146) * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <[email protected]> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <[email protected]> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <[email protected]> * feat: support driver and executor pod use different priority Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> * feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> * feat: merge the logic of setPodPriorityClassName into addPriorityClassName Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> * feat: add adjust pointer if is nil Signed-off-by: Kevin.Wu <[email protected]> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <[email protected]> * feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass Signed-off-by: Kevin Wu <[email protected]> * feat: Optimize code to avoid null pointer exceptions Signed-off-by: Kevin.Wu <[email protected]> * fix: remove backup crd files Signed-off-by: Kevin.Wu <[email protected]> * fix: remove BatchSchedulerOptions.PriorityClassName test code Signed-off-by: Kevin Wu <[email protected]> * fix: add driver and executor pod priorityClassName test code Signed-off-by: Kevin Wu <[email protected]> --------- Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> Co-authored-by: Kevin Wu <[email protected]> (cherry picked from commit 6ae1b2f) Signed-off-by: Yi Chen <[email protected]> * Bump gocloud.dev from 0.37.0 to 0.39.0 (kubeflow#2160) Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.37.0 to 0.39.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](google/go-cloud@v0.37.0...v0.39.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit e58023b) Signed-off-by: Yi Chen <[email protected]> * Update e2e tests (kubeflow#2161) * Add sleep buffer to ensture the webhooks are ready before running the e2e tests Signed-off-by: Yi Chen <[email protected]> * Remove duplicate operator image build tasks Signed-off-by: Yi Chen <[email protected]> * Update e2e tests Signed-off-by: Yi Chen <[email protected]> * Update examples Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit e6a7805) Signed-off-by: Yi Chen <[email protected]> * fix: webhook not working when settings spark job namespaces to empty (kubeflow#2163) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 7785107) Signed-off-by: Yi Chen <[email protected]> * fix: The logger had an odd number of arguments, making it panic (kubeflow#2166) Signed-off-by: tcassaert <[email protected]> (cherry picked from commit eb48b34) Signed-off-by: Yi Chen <[email protected]> * Upgrade to Spark 3.5.2(kubeflow#2012) (kubeflow#2157) * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <[email protected]> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <[email protected]> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <[email protected]> * Upgrade to Spark 3.5.2 Signed-off-by: HyukSangCho <[email protected]> --------- Signed-off-by: HyukSangCho <[email protected]> (cherry picked from commit 9f0c08a) Signed-off-by: Yi Chen <[email protected]> * Feature: Add pprof endpoint (kubeflow#2164) * add pprof support to the operator Controller Manager Signed-off-by: ImpSy <[email protected]> * add pprof support to helm chart Signed-off-by: ImpSy <[email protected]> --------- Signed-off-by: ImpSy <[email protected]> (cherry picked from commit 75b9266) Signed-off-by: Yi Chen <[email protected]> * fix the make kind-delete-custer to avoid accidental kubeconfig deletion (kubeflow#2172) Signed-off-by: ImpSy <[email protected]> (cherry picked from commit cbfefd5) Signed-off-by: Yi Chen <[email protected]> * Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (kubeflow#2174) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.27 to 1.27.33. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.33) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit b818332) Signed-off-by: Yi Chen <[email protected]> * Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (kubeflow#2173) Bumps [helm.sh/helm/v3](https://github.com/helm/helm) from 3.15.3 to 3.16.1. - [Release notes](https://github.com/helm/helm/releases) - [Commits](helm/helm@v3.15.3...v3.16.1) --- updated-dependencies: - dependency-name: helm.sh/helm/v3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit f3f80d4) Signed-off-by: Yi Chen <[email protected]> * Add specific error in log line when failed to create web UI service (kubeflow#2170) * Add specific error in log line when failed to create web UI service Signed-off-by: tcassaert <[email protected]> * Update log to reflect correct resource that could not be created Co-authored-by: Yi Chen <[email protected]> Signed-off-by: tcassaert <[email protected]> --------- Signed-off-by: tcassaert <[email protected]> Signed-off-by: tcassaert <[email protected]> Co-authored-by: Yi Chen <[email protected]> (cherry picked from commit ed3226e) Signed-off-by: Yi Chen <[email protected]> * Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (kubeflow#2178) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit a2f71c6) Signed-off-by: Yi Chen <[email protected]> * Fix: spark application does not respect time to live seconds (kubeflow#2165) * Add time to live seconds example spark application Signed-off-by: Yi Chen <[email protected]> * fix: spark application does not respect time to live seconds Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit c855ee4) Signed-off-by: Yi Chen <[email protected]> * Update release workflow and docs (kubeflow#2121) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit bca6aa8) Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Jacob Salway <[email protected]> Signed-off-by: Yi Chen <[email protected]> Signed-off-by: pengfei4.li <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Kevin Wu <[email protected]> Signed-off-by: Kevin.Wu <[email protected]> Signed-off-by: tcassaert <[email protected]> Signed-off-by: HyukSangCho <[email protected]> Signed-off-by: ImpSy <[email protected]> Signed-off-by: tcassaert <[email protected]> Co-authored-by: Jacob Salway <[email protected]> Co-authored-by: Neo <[email protected]> Co-authored-by: pengfei4.li <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevinz <[email protected]> Co-authored-by: Kevin Wu <[email protected]> Co-authored-by: tcassaert <[email protected]> Co-authored-by: ha2hi <[email protected]> Co-authored-by: Sébastien Maintrot <[email protected]>

google-oss-prow bot added the do-not-merge/work-in-progress label Sep 9, 2024

google-oss-prow bot requested review from mwielgus and vara-bonthu September 9, 2024 03:16

google-oss-prow bot added the size/M label Sep 9, 2024

jacobsalway force-pushed the deps/spark-3.5.2 branch from 9d23351 to 60dc8b6 Compare September 9, 2024 03:18

jacobsalway mentioned this pull request Sep 9, 2024

Update Golang to 1.23 and Spark to 3.5.2 #2150

Closed

8 tasks

jacobsalway force-pushed the deps/spark-3.5.2 branch from 60dc8b6 to ec0ec0c Compare September 9, 2024 03:24

jacobsalway marked this pull request as ready for review September 9, 2024 04:01

google-oss-prow bot removed the do-not-merge/work-in-progress label Sep 9, 2024

google-oss-prow bot requested a review from andreyvelich September 9, 2024 04:01

Upgrade to Spark 3.5.2

ca1f050

Signed-off-by: Jacob Salway <[email protected]>

jacobsalway force-pushed the deps/spark-3.5.2 branch from ec0ec0c to ca1f050 Compare September 9, 2024 04:57

google-oss-prow bot added size/L and removed size/M labels Sep 9, 2024

google-oss-prow bot assigned ChenYi015 Sep 9, 2024

google-oss-prow bot added the lgtm label Sep 9, 2024

google-oss-prow bot added the approved label Sep 9, 2024

google-oss-prow bot merged commit e1b7a27 into kubeflow:master Sep 9, 2024
7 checks passed

jacobsalway deleted the deps/spark-3.5.2 branch September 9, 2024 05:07

ChenYi015 pushed a commit to ChenYi015/spark-operator that referenced this pull request Sep 9, 2024

Upgrade to Spark 3.5.2 (kubeflow#2154)

a241127

Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit e1b7a27)

ChenYi015 mentioned this pull request Sep 9, 2024

Cherry pick commits for releasing v2.0.0 #2156

Merged

4 tasks

ChenYi015 pushed a commit to ChenYi015/spark-operator that referenced this pull request Sep 9, 2024

Upgrade to Spark 3.5.2 (kubeflow#2154)

8b3b279

Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit e1b7a27) Signed-off-by: Yi Chen <[email protected]>

jacobsalway mentioned this pull request Sep 10, 2024

(#2020) Bump to Spark 3.5.1 #2023

Closed

8 tasks

ChenYi015 mentioned this pull request Sep 18, 2024

[FEATURE] Upgrade default image to Spark 3.5.1 #2020

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to Spark 3.5.2 #2154

Upgrade to Spark 3.5.2 #2154

jacobsalway commented Sep 9, 2024 •

edited

Loading

ChenYi015 commented Sep 9, 2024

jacobsalway commented Sep 9, 2024 •

edited

Loading

ChenYi015 commented Sep 9, 2024

google-oss-prow bot commented Sep 9, 2024

Upgrade to Spark 3.5.2 #2154

Upgrade to Spark 3.5.2 #2154

Conversation

jacobsalway commented Sep 9, 2024 • edited Loading

Purpose

Testing

Change Category

Checklist

ChenYi015 commented Sep 9, 2024

jacobsalway commented Sep 9, 2024 • edited Loading

ChenYi015 commented Sep 9, 2024

google-oss-prow bot commented Sep 9, 2024

jacobsalway commented Sep 9, 2024 •

edited

Loading

jacobsalway commented Sep 9, 2024 •

edited

Loading