Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Spark Operator to v1beta2 #573

Closed
2 of 13 tasks
akhurana001 opened this issue Oct 27, 2020 · 1 comment
Closed
2 of 13 tasks

Update Spark Operator to v1beta2 #573

akhurana001 opened this issue Oct 27, 2020 · 1 comment
Assignees
@akhurana001
Labels
enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers
Milestone
0.10.0

Comments

@akhurana001
Copy link
Contributor

akhurana001 commented Oct 27, 2020 •
edited
Loading

Motivation: Why do you think this is important?
Flyte currently uses v1beta1 version of SparkOperator. Update this to v1beta2 which has support for resource-quota enforcement and support for batch scheduler among other new features/changes.

This needs two changes (WIP):

If the cluster already has Spark v1beta1 CRD & SparkOperator deployed, these will need to be manually deleted before the new CRD/operator can be deployed. This will cause any existing spark tasks/jobs running on the cluster to fail.

If there are multiple clusters, they can be drained one by one via cluster-level weights in flyteadmin and then rolled over the cluster to the newer version of Spark CRD/Operator and FlytePropeller.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

[Optional] Propose: Link/Inline
If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document.

Additional context
Add any other context or screenshots about the feature request here.

Is this a blocker for you to adopt Flyte
Please let us know if this makes it impossible to adopt Flyte
@akhurana001 akhurana001 added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Oct 27, 2020
This was referenced Oct 27, 2020
Merged
Merged
Merged
@akhurana001 akhurana001 added this to the 0.10.0 milestone Nov 4, 2020
@akhurana001
Copy link
Contributor Author

Completed, resolving.

eapolinario pushed a commit to eapolinario/flyte that referenced this issue Jul 24, 2023
@EngHabu
Add logs and stats to sync loop (flyteorg#573)
949f5a2 
* Add logs and stats to sync loop

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Add per namespace log as well

Signed-off-by: Haytham Abuelfutuh <[email protected]>

---------

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Jul 24, 2023
@EngHabu
Use GetExecutionData instead (flyteorg#573)
89dfd5b 
* Use GetExecutionData instead

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* bump

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* fix unit tests

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* more fixes

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* fix more

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Regenerate

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Only call GetExecutionData when the workflow succeeds

Signed-off-by: Haytham Abuelfutuh <[email protected]>

---------

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@EngHabu
Use GetExecutionData instead (flyteorg#573)
10efce7 
* Use GetExecutionData instead

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* bump

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* fix unit tests

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* more fixes

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* fix more

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Regenerate

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Only call GetExecutionData when the workflow succeeds

Signed-off-by: Haytham Abuelfutuh <[email protected]>

---------

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@EngHabu
Add logs and stats to sync loop (flyteorg#573)
738c21e 
* Add logs and stats to sync loop

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Add per namespace log as well

Signed-off-by: Haytham Abuelfutuh <[email protected]>

---------

Signed-off-by: Haytham Abuelfutuh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@akhurana001 akhurana001

Labels
enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers
Projects
None yet
Milestone
0.10.0
Development

No branches or pull requests
1 participant
@akhurana001