Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update e2e tests #2161

Merged
merged 4 commits into from
Sep 13, 2024
Merged

Conversation

ChenYi015
Copy link
Contributor

@ChenYi015 ChenYi015 commented Sep 10, 2024

Purpose of this PR

Proposed changes:

  • Remove duplicate operator image build tasks in integration test
  • Add time sleep buffer to ensure that webhooks are ready before running the e2e tests
  • Check whether volumes are mounted in e2e tests
  • Check whether resource requests and limits are set in e2e tests
  • Remove the coreLimit field from all example spark applications except for spark-pi-custom-resource.yaml. For that, added coreRequest and coreLimit fields to it.

Change Category

Indicate the type of change by marking the applicable boxes:

  • Bugfix (non-breaking change which fixes an issue)
  • Feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that could affect existing functionality)
  • Documentation update

Checklist

Before submitting your PR, please review the following:

  • I have conducted a self-review of my own code.
  • I have updated documentation accordingly.
  • I have added tests that prove my changes are effective or that my feature works.
  • Existing unit tests pass locally with my changes.

Additional Notes

The e2e tests of the recent PRs still failed due to the webhook not ready, even after checking the CA bundle and webhook service enpoints.

@ChenYi015
Copy link
Contributor Author

/assign @yuchaoran2011 @vara-bonthu @jacobsalway

@google-oss-prow google-oss-prow bot added size/M and removed size/S labels Sep 10, 2024
@ChenYi015 ChenYi015 changed the title Add time sleep buffer to ensure that webhooks are ready before running the e2e tests Update e2e tests Sep 10, 2024
@ChenYi015 ChenYi015 force-pushed the workflow/integration-test branch from 14b97d4 to ed1d525 Compare September 10, 2024 09:57
Signed-off-by: Yi Chen <[email protected]>
@ChenYi015 ChenYi015 force-pushed the workflow/integration-test branch from ed1d525 to bbf1e88 Compare September 10, 2024 10:47
@@ -149,6 +149,8 @@ var _ = BeforeSuite(func() {
validatingWebhookKey := types.NamespacedName{Name: ValidatingWebhookName}
Expect(waitForMutatingWebhookReady(context.Background(), mutatingWebhookKey)).NotTo(HaveOccurred())
Expect(waitForValidatingWebhookReady(context.Background(), validatingWebhookKey)).NotTo(HaveOccurred())
// TODO: Remove this when there is a better way to ensure the webhooks are ready before running the e2e tests.
time.Sleep(10 * time.Second)
Copy link
Member

@jacobsalway jacobsalway Sep 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine as a temporary workaround. Two ways we can solve this in the future:

  • Remove the dependency on the webhook altogether by constructing pod template files on the fly from the existing SparkApplication spec. Maybe via a ConfigMap into the driver pod?
  • If the webhook is enabled, the controller should only start reconciling if the webhook is ready to serve mutation requests.

}
Expect(container.Resources.Requests.Memory).NotTo(BeNil())
Expect(container.Resources.Limits.Memory).NotTo(BeNil())
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious: was there any specific reason to add this assertion? The request and limit fields on the pods are set via spark-submit args and aren't patched by the webhook issue that I believe you raised this PR as part of

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the cpu/memory requests/limits are not patched by the webhook. I added these assertions just to ensure that spark-pi-custom-resouce.yaml works as expected in setting resource requests/limits correctly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about removing the coreLimit field from the existing example applications and creating a new application that specifies both coreRequest and coreLimit, creating a new end-to-end test for that, then moving this assertion logic to that test? I feel like that captures the intention better but open to your opinion too obviously.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jacobsalway Sounds good. I have removed the coreLimit field from all the example applications except for spark-pi-custom-resource.yaml. For that, I added the coreRequest and coreLimit fields to it.

Signed-off-by: Yi Chen <[email protected]>
@google-oss-prow google-oss-prow bot added size/L and removed size/M labels Sep 12, 2024
@jacobsalway
Copy link
Member

/lgtm

@google-oss-prow google-oss-prow bot added the lgtm label Sep 12, 2024
@ChenYi015
Copy link
Contributor Author

/approve

Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit e6a7805 into kubeflow:master Sep 13, 2024
7 checks passed
@ChenYi015 ChenYi015 deleted the workflow/integration-test branch September 13, 2024 04:34
ChenYi015 added a commit to ChenYi015/spark-operator that referenced this pull request Sep 19, 2024
* Add sleep buffer to ensture the webhooks are ready before running the e2e tests

Signed-off-by: Yi Chen <[email protected]>

* Remove duplicate operator image build tasks

Signed-off-by: Yi Chen <[email protected]>

* Update e2e tests

Signed-off-by: Yi Chen <[email protected]>

* Update examples

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit e6a7805)
Signed-off-by: Yi Chen <[email protected]>
google-oss-prow bot pushed a commit that referenced this pull request Sep 23, 2024
* Support gang scheduling with Yunikorn (#2107)

* Add Yunikorn scheduler and example

Signed-off-by: Jacob Salway <[email protected]>

* Add test cases

Signed-off-by: Jacob Salway <[email protected]>

* Add code comments

Signed-off-by: Jacob Salway <[email protected]>

* Add license comment

Signed-off-by: Jacob Salway <[email protected]>

* Inline mergeNodeSelector

Signed-off-by: Jacob Salway <[email protected]>

* Fix initial number implementation

Signed-off-by: Jacob Salway <[email protected]>

---------

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 8fcda12)
Signed-off-by: Yi Chen <[email protected]>

* Update Makefile for building sparkctl (#2119)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 4bc6e89)
Signed-off-by: Yi Chen <[email protected]>

* fix: Add default values for namespaces to match usage descriptions  (#2128)

* fix: Add default values for namespaces to match usage descriptions

Signed-off-by: pengfei4.li <[email protected]>

* fix: remove incorrect cache settings

Signed-off-by: pengfei4.li <[email protected]>

---------

Signed-off-by: pengfei4.li <[email protected]>
Co-authored-by: pengfei4.li <[email protected]>
(cherry picked from commit 52f818d)
Signed-off-by: Yi Chen <[email protected]>

* Fix: Spark role binding did not render properly when setting spark service account name (#2135)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit a1a38ea)
Signed-off-by: Yi Chen <[email protected]>

* Reintroduce option webhook.enable (#2142)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 9e88049)
Signed-off-by: Yi Chen <[email protected]>

* Add default batch scheduler argument (#2143)

* Add default batch scheduler argument

Signed-off-by: Jacob Salway <[email protected]>

* Add helm unit test

Signed-off-by: Jacob Salway <[email protected]>

---------

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 9cc1c02)
Signed-off-by: Yi Chen <[email protected]>

* fix: unable to set controller/webhook replicas to zero (#2147)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 1afa72e)
Signed-off-by: Yi Chen <[email protected]>

* Adding support for setting spark job namespaces to all namespaces (#2123)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit c93b0ec)
Signed-off-by: Yi Chen <[email protected]>

* Support extended kube-scheduler as batch scheduler (#2136)

* Support coscheduling with kube-scheduler plugins

Signed-off-by: Yi Chen <[email protected]>

* Add example for using kube-schulder coscheduling

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit e8d3de9)
Signed-off-by: Yi Chen <[email protected]>

* Run e2e tests on Kind (#2148)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit c810ece)
Signed-off-by: Yi Chen <[email protected]>

* Set schedulerName to Yunikorn (#2153)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 62b4ca6)
Signed-off-by: Yi Chen <[email protected]>

* Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (#2129)

watching all namespaces

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 592b649)
Signed-off-by: Yi Chen <[email protected]>

* Fix: e2e test failes due to webhook not ready (#2149)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit dee91ba)
Signed-off-by: Yi Chen <[email protected]>

* Upgrade to Go 1.23.1 (#2155)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 10fcb8e)
Signed-off-by: Yi Chen <[email protected]>

* Upgrade to Spark 3.5.2 (#2154)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit e1b7a27)
Signed-off-by: Yi Chen <[email protected]>

* Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (#2159)

Bumps [sigs.k8s.io/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) from 0.29.7 to 0.29.8.
- [Release notes](https://github.com/kubernetes-sigs/scheduler-plugins/releases)
- [Changelog](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/RELEASE.md)
- [Commits](kubernetes-sigs/scheduler-plugins@v0.29.7...v0.29.8)

---
updated-dependencies:
- dependency-name: sigs.k8s.io/scheduler-plugins
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 95d202e)
Signed-off-by: Yi Chen <[email protected]>

* feat: support driver and executor pod use different priority (#2146)

* feat: support driver and executor pod use different priority

Signed-off-by: Kevin Wu <[email protected]>

* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName

Signed-off-by: Kevin Wu <[email protected]>

* feat: merge the logic of setPodPriorityClassName into addPriorityClassName

Signed-off-by: Kevin Wu <[email protected]>

* feat: support driver and executor pod use different priority

Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>

* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName

Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>

* feat: merge the logic of setPodPriorityClassName into addPriorityClassName

Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>

* feat: add adjust pointer if is nil

Signed-off-by: Kevin.Wu <[email protected]>

* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass

Signed-off-by: Kevin Wu <[email protected]>

* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass

Signed-off-by: Kevin Wu <[email protected]>

* feat: Optimize code to avoid null pointer exceptions

Signed-off-by: Kevin.Wu <[email protected]>

* fix: remove backup crd files

Signed-off-by: Kevin.Wu <[email protected]>

* fix: remove BatchSchedulerOptions.PriorityClassName test code

Signed-off-by: Kevin Wu <[email protected]>

* fix: add driver and executor pod priorityClassName test code

Signed-off-by: Kevin Wu <[email protected]>

---------

Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>
Co-authored-by: Kevin Wu <[email protected]>
(cherry picked from commit 6ae1b2f)
Signed-off-by: Yi Chen <[email protected]>

* Bump gocloud.dev from 0.37.0 to 0.39.0 (#2160)

Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.37.0 to 0.39.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](google/go-cloud@v0.37.0...v0.39.0)

---
updated-dependencies:
- dependency-name: gocloud.dev
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit e58023b)
Signed-off-by: Yi Chen <[email protected]>

* Update e2e tests (#2161)

* Add sleep buffer to ensture the webhooks are ready before running the e2e tests

Signed-off-by: Yi Chen <[email protected]>

* Remove duplicate operator image build tasks

Signed-off-by: Yi Chen <[email protected]>

* Update e2e tests

Signed-off-by: Yi Chen <[email protected]>

* Update examples

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit e6a7805)
Signed-off-by: Yi Chen <[email protected]>

* fix: webhook not working when settings spark job namespaces to empty (#2163)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 7785107)
Signed-off-by: Yi Chen <[email protected]>

* fix: The logger had an odd number of arguments, making it panic (#2166)

Signed-off-by: tcassaert <[email protected]>
(cherry picked from commit eb48b34)
Signed-off-by: Yi Chen <[email protected]>

* Upgrade to Spark 3.5.2(#2012) (#2157)

* Upgrade to Spark 3.5.2

Signed-off-by: HyukSangCho <[email protected]>

* Upgrade to Spark 3.5.2

Signed-off-by: HyukSangCho <[email protected]>

* Upgrade to Spark 3.5.2

Signed-off-by: HyukSangCho <[email protected]>

* Upgrade to Spark 3.5.2

Signed-off-by: HyukSangCho <[email protected]>

---------

Signed-off-by: HyukSangCho <[email protected]>
(cherry picked from commit 9f0c08a)
Signed-off-by: Yi Chen <[email protected]>

* Feature: Add pprof endpoint (#2164)

* add pprof support to the operator Controller Manager

Signed-off-by: ImpSy <[email protected]>

* add pprof support to helm chart

Signed-off-by: ImpSy <[email protected]>

---------

Signed-off-by: ImpSy <[email protected]>
(cherry picked from commit 75b9266)
Signed-off-by: Yi Chen <[email protected]>

* fix the make kind-delete-custer to avoid accidental kubeconfig deletion (#2172)

Signed-off-by: ImpSy <[email protected]>
(cherry picked from commit cbfefd5)
Signed-off-by: Yi Chen <[email protected]>

* Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (#2174)

Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.27 to 1.27.33.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.33)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit b818332)
Signed-off-by: Yi Chen <[email protected]>

* Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (#2173)

Bumps [helm.sh/helm/v3](https://github.com/helm/helm) from 3.15.3 to 3.16.1.
- [Release notes](https://github.com/helm/helm/releases)
- [Commits](helm/helm@v3.15.3...v3.16.1)

---
updated-dependencies:
- dependency-name: helm.sh/helm/v3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit f3f80d4)
Signed-off-by: Yi Chen <[email protected]>

* Add specific error in log line when failed to create web UI service (#2170)

* Add specific error in log line when failed to create web UI service

Signed-off-by: tcassaert <[email protected]>

* Update log to reflect correct resource that could not be created

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: tcassaert <[email protected]>

---------

Signed-off-by: tcassaert <[email protected]>
Signed-off-by: tcassaert <[email protected]>
Co-authored-by: Yi Chen <[email protected]>
(cherry picked from commit ed3226e)
Signed-off-by: Yi Chen <[email protected]>

* Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (#2178)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit a2f71c6)
Signed-off-by: Yi Chen <[email protected]>

* Fix: spark application does not respect time to live seconds (#2165)

* Add time to live seconds example spark application

Signed-off-by: Yi Chen <[email protected]>

* fix: spark application does not respect time to live seconds

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit c855ee4)
Signed-off-by: Yi Chen <[email protected]>

* Update release workflow and docs (#2121)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit bca6aa8)
Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Jacob Salway <[email protected]>
Signed-off-by: Yi Chen <[email protected]>
Signed-off-by: pengfei4.li <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>
Signed-off-by: tcassaert <[email protected]>
Signed-off-by: HyukSangCho <[email protected]>
Signed-off-by: ImpSy <[email protected]>
Signed-off-by: tcassaert <[email protected]>
Co-authored-by: Jacob Salway <[email protected]>
Co-authored-by: Neo <[email protected]>
Co-authored-by: pengfei4.li <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kevinz <[email protected]>
Co-authored-by: Kevin Wu <[email protected]>
Co-authored-by: tcassaert <[email protected]>
Co-authored-by: ha2hi <[email protected]>
Co-authored-by: Sébastien Maintrot <[email protected]>
jbhalodia-slack pushed a commit to jbhalodia-slack/spark-operator that referenced this pull request Oct 4, 2024
* Support gang scheduling with Yunikorn (kubeflow#2107)

* Add Yunikorn scheduler and example

Signed-off-by: Jacob Salway <[email protected]>

* Add test cases

Signed-off-by: Jacob Salway <[email protected]>

* Add code comments

Signed-off-by: Jacob Salway <[email protected]>

* Add license comment

Signed-off-by: Jacob Salway <[email protected]>

* Inline mergeNodeSelector

Signed-off-by: Jacob Salway <[email protected]>

* Fix initial number implementation

Signed-off-by: Jacob Salway <[email protected]>

---------

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 8fcda12)
Signed-off-by: Yi Chen <[email protected]>

* Update Makefile for building sparkctl (kubeflow#2119)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 4bc6e89)
Signed-off-by: Yi Chen <[email protected]>

* fix: Add default values for namespaces to match usage descriptions  (kubeflow#2128)

* fix: Add default values for namespaces to match usage descriptions

Signed-off-by: pengfei4.li <[email protected]>

* fix: remove incorrect cache settings

Signed-off-by: pengfei4.li <[email protected]>

---------

Signed-off-by: pengfei4.li <[email protected]>
Co-authored-by: pengfei4.li <[email protected]>
(cherry picked from commit 52f818d)
Signed-off-by: Yi Chen <[email protected]>

* Fix: Spark role binding did not render properly when setting spark service account name (kubeflow#2135)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit a1a38ea)
Signed-off-by: Yi Chen <[email protected]>

* Reintroduce option webhook.enable (kubeflow#2142)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 9e88049)
Signed-off-by: Yi Chen <[email protected]>

* Add default batch scheduler argument (kubeflow#2143)

* Add default batch scheduler argument

Signed-off-by: Jacob Salway <[email protected]>

* Add helm unit test

Signed-off-by: Jacob Salway <[email protected]>

---------

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 9cc1c02)
Signed-off-by: Yi Chen <[email protected]>

* fix: unable to set controller/webhook replicas to zero (kubeflow#2147)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 1afa72e)
Signed-off-by: Yi Chen <[email protected]>

* Adding support for setting spark job namespaces to all namespaces (kubeflow#2123)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit c93b0ec)
Signed-off-by: Yi Chen <[email protected]>

* Support extended kube-scheduler as batch scheduler (kubeflow#2136)

* Support coscheduling with kube-scheduler plugins

Signed-off-by: Yi Chen <[email protected]>

* Add example for using kube-schulder coscheduling

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit e8d3de9)
Signed-off-by: Yi Chen <[email protected]>

* Run e2e tests on Kind (kubeflow#2148)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit c810ece)
Signed-off-by: Yi Chen <[email protected]>

* Set schedulerName to Yunikorn (kubeflow#2153)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 62b4ca6)
Signed-off-by: Yi Chen <[email protected]>

* Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces (kubeflow#2129)

watching all namespaces

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 592b649)
Signed-off-by: Yi Chen <[email protected]>

* Fix: e2e test failes due to webhook not ready (kubeflow#2149)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit dee91ba)
Signed-off-by: Yi Chen <[email protected]>

* Upgrade to Go 1.23.1 (kubeflow#2155)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 10fcb8e)
Signed-off-by: Yi Chen <[email protected]>

* Upgrade to Spark 3.5.2 (kubeflow#2154)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit e1b7a27)
Signed-off-by: Yi Chen <[email protected]>

* Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (kubeflow#2159)

Bumps [sigs.k8s.io/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) from 0.29.7 to 0.29.8.
- [Release notes](https://github.com/kubernetes-sigs/scheduler-plugins/releases)
- [Changelog](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/RELEASE.md)
- [Commits](kubernetes-sigs/scheduler-plugins@v0.29.7...v0.29.8)

---
updated-dependencies:
- dependency-name: sigs.k8s.io/scheduler-plugins
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 95d202e)
Signed-off-by: Yi Chen <[email protected]>

* feat: support driver and executor pod use different priority (kubeflow#2146)

* feat: support driver and executor pod use different priority

Signed-off-by: Kevin Wu <[email protected]>

* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName

Signed-off-by: Kevin Wu <[email protected]>

* feat: merge the logic of setPodPriorityClassName into addPriorityClassName

Signed-off-by: Kevin Wu <[email protected]>

* feat: support driver and executor pod use different priority

Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>

* feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.PriorityClassName specifically defined, then can precedence over spec.batchSchedulerOptions.priorityClassName

Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>

* feat: merge the logic of setPodPriorityClassName into addPriorityClassName

Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>

* feat: add adjust pointer if is nil

Signed-off-by: Kevin.Wu <[email protected]>

* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass

Signed-off-by: Kevin Wu <[email protected]>

* feat: remove spec.batchSchedulerOptions.priorityClassName define , split driver and executor pod priorityClass

Signed-off-by: Kevin Wu <[email protected]>

* feat: Optimize code to avoid null pointer exceptions

Signed-off-by: Kevin.Wu <[email protected]>

* fix: remove backup crd files

Signed-off-by: Kevin.Wu <[email protected]>

* fix: remove BatchSchedulerOptions.PriorityClassName test code

Signed-off-by: Kevin Wu <[email protected]>

* fix: add driver and executor pod priorityClassName test code

Signed-off-by: Kevin Wu <[email protected]>

---------

Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>
Co-authored-by: Kevin Wu <[email protected]>
(cherry picked from commit 6ae1b2f)
Signed-off-by: Yi Chen <[email protected]>

* Bump gocloud.dev from 0.37.0 to 0.39.0 (kubeflow#2160)

Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.37.0 to 0.39.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](google/go-cloud@v0.37.0...v0.39.0)

---
updated-dependencies:
- dependency-name: gocloud.dev
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit e58023b)
Signed-off-by: Yi Chen <[email protected]>

* Update e2e tests (kubeflow#2161)

* Add sleep buffer to ensture the webhooks are ready before running the e2e tests

Signed-off-by: Yi Chen <[email protected]>

* Remove duplicate operator image build tasks

Signed-off-by: Yi Chen <[email protected]>

* Update e2e tests

Signed-off-by: Yi Chen <[email protected]>

* Update examples

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit e6a7805)
Signed-off-by: Yi Chen <[email protected]>

* fix: webhook not working when settings spark job namespaces to empty (kubeflow#2163)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 7785107)
Signed-off-by: Yi Chen <[email protected]>

* fix: The logger had an odd number of arguments, making it panic (kubeflow#2166)

Signed-off-by: tcassaert <[email protected]>
(cherry picked from commit eb48b34)
Signed-off-by: Yi Chen <[email protected]>

* Upgrade to Spark 3.5.2(kubeflow#2012) (kubeflow#2157)

* Upgrade to Spark 3.5.2

Signed-off-by: HyukSangCho <[email protected]>

* Upgrade to Spark 3.5.2

Signed-off-by: HyukSangCho <[email protected]>

* Upgrade to Spark 3.5.2

Signed-off-by: HyukSangCho <[email protected]>

* Upgrade to Spark 3.5.2

Signed-off-by: HyukSangCho <[email protected]>

---------

Signed-off-by: HyukSangCho <[email protected]>
(cherry picked from commit 9f0c08a)
Signed-off-by: Yi Chen <[email protected]>

* Feature: Add pprof endpoint (kubeflow#2164)

* add pprof support to the operator Controller Manager

Signed-off-by: ImpSy <[email protected]>

* add pprof support to helm chart

Signed-off-by: ImpSy <[email protected]>

---------

Signed-off-by: ImpSy <[email protected]>
(cherry picked from commit 75b9266)
Signed-off-by: Yi Chen <[email protected]>

* fix the make kind-delete-custer to avoid accidental kubeconfig deletion (kubeflow#2172)

Signed-off-by: ImpSy <[email protected]>
(cherry picked from commit cbfefd5)
Signed-off-by: Yi Chen <[email protected]>

* Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (kubeflow#2174)

Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.27 to 1.27.33.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.33)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit b818332)
Signed-off-by: Yi Chen <[email protected]>

* Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (kubeflow#2173)

Bumps [helm.sh/helm/v3](https://github.com/helm/helm) from 3.15.3 to 3.16.1.
- [Release notes](https://github.com/helm/helm/releases)
- [Commits](helm/helm@v3.15.3...v3.16.1)

---
updated-dependencies:
- dependency-name: helm.sh/helm/v3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit f3f80d4)
Signed-off-by: Yi Chen <[email protected]>

* Add specific error in log line when failed to create web UI service (kubeflow#2170)

* Add specific error in log line when failed to create web UI service

Signed-off-by: tcassaert <[email protected]>

* Update log to reflect correct resource that could not be created

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: tcassaert <[email protected]>

---------

Signed-off-by: tcassaert <[email protected]>
Signed-off-by: tcassaert <[email protected]>
Co-authored-by: Yi Chen <[email protected]>
(cherry picked from commit ed3226e)
Signed-off-by: Yi Chen <[email protected]>

* Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (kubeflow#2178)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit a2f71c6)
Signed-off-by: Yi Chen <[email protected]>

* Fix: spark application does not respect time to live seconds (kubeflow#2165)

* Add time to live seconds example spark application

Signed-off-by: Yi Chen <[email protected]>

* fix: spark application does not respect time to live seconds

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit c855ee4)
Signed-off-by: Yi Chen <[email protected]>

* Update release workflow and docs (kubeflow#2121)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit bca6aa8)
Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Jacob Salway <[email protected]>
Signed-off-by: Yi Chen <[email protected]>
Signed-off-by: pengfei4.li <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Kevin Wu <[email protected]>
Signed-off-by: Kevin.Wu <[email protected]>
Signed-off-by: tcassaert <[email protected]>
Signed-off-by: HyukSangCho <[email protected]>
Signed-off-by: ImpSy <[email protected]>
Signed-off-by: tcassaert <[email protected]>
Co-authored-by: Jacob Salway <[email protected]>
Co-authored-by: Neo <[email protected]>
Co-authored-by: pengfei4.li <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kevinz <[email protected]>
Co-authored-by: Kevin Wu <[email protected]>
Co-authored-by: tcassaert <[email protected]>
Co-authored-by: ha2hi <[email protected]>
Co-authored-by: Sébastien Maintrot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants