Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark3 Support: Add support for all with Pyspark 3.x #482

Closed
13 tasks
akhurana001 opened this issue Aug 25, 2020 · 1 comment
Closed
13 tasks

Spark3 Support: Add support for all with Pyspark 3.x #482

akhurana001 opened this issue Aug 25, 2020 · 1 comment
Assignees
@akhurana001
Labels
enhancement New feature or request
Milestone
0.7.0

Comments

@akhurana001
Copy link
Contributor

Motivation: Why do you think this is important?
Currently all directive in flytekit installs Spark 2.4 and other libraries including pandas/pytorch etc. There is no corresponding all directive with support for Spark 3.0. This makes it difficult for users to migrate from Spark 2.4 to Spark 3.0.

Goal: What should the final outcome look like, ideally?
Add support for all3 directive. At some point, we should move all to default to Spark 3.x.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

[Optional] Propose: Link/Inline
If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document.

Additional context
Add any other context or screenshots about the feature request here.

Is this a blocker for you to adopt Flyte
Please let us know if this makes it impossible to adopt Flyte
@akhurana001 akhurana001 added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Aug 25, 2020
@akhurana001 akhurana001 self-assigned this Aug 25, 2020
This was referenced Aug 25, 2020
Closed
Merged
@wild-endeavor wild-endeavor removed the untriaged This issues has not yet been looked at by the Maintainers label Aug 25, 2020
@wild-endeavor wild-endeavor added this to the 0.7.0 milestone Aug 25, 2020
@akhurana001
Copy link
Contributor Author

akhurana001 commented Aug 26, 2020 •
edited
Loading

Resolved, released in flytekit 0.12.0

eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@hamersaw
Make k8s secrets optional (flyteorg#482)
66eaf12 
* make secrets optional - to skip mounting missing values

Signed-off-by: Daniel Rammer <[email protected]>

* add env var secrets to optional values

Signed-off-by: Daniel Rammer <[email protected]>

* fixed lint issue

Signed-off-by: Daniel Rammer <[email protected]>

* fixed unit tests

Signed-off-by: Daniel Rammer <[email protected]>

* fixed lint issue

Signed-off-by: Daniel Rammer <[email protected]>

Signed-off-by: Daniel Rammer <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
Skipping of cached task outputs via execution config (flyteorg#482)
4cc2381 
* Implemented SkipCache handling for execution config

Signed-off-by: Nick Müller <[email protected]>

* Added cache skip override to RelaunchExecution endpoint

Signed-off-by: Nick Müller <[email protected]>

* Updated to latest version of flytepropeller

Signed-off-by: Nick Müller <[email protected]>

* Renamed SkipCache flag to OverwriteCache
Updated to latest released versions of flyteidl, flyteplugins and flytestdlib
Updated to latest unmerged version of flytepropeller

Signed-off-by: Nick Müller <[email protected]>

* Updated flyteidl, flytepropeller and flytestdlib to latest released versions

Signed-off-by: Nick Müller <[email protected]>

* Reworded comment for clarity

Signed-off-by: Nick Müller <[email protected]>

Signed-off-by: Nick Müller <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@samhita-alla
Fix Feast's Jupyter notebook code (flyteorg#482)
6874282 
* fix Feast's Jupyter notebook code

Signed-off-by: Samhita Alla <[email protected]>

* remove cell outputs

Signed-off-by: Samhita Alla <[email protected]>

* working

Signed-off-by: Samhita Alla <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@hamersaw
Make k8s secrets optional (flyteorg#482)
506cee1 
* make secrets optional - to skip mounting missing values

Signed-off-by: Daniel Rammer <[email protected]>

* add env var secrets to optional values

Signed-off-by: Daniel Rammer <[email protected]>

* fixed lint issue

Signed-off-by: Daniel Rammer <[email protected]>

* fixed unit tests

Signed-off-by: Daniel Rammer <[email protected]>

* fixed lint issue

Signed-off-by: Daniel Rammer <[email protected]>

Signed-off-by: Daniel Rammer <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
Skipping of cached task outputs via execution config (flyteorg#482)
bc87220 
* Implemented SkipCache handling for execution config

Signed-off-by: Nick Müller <[email protected]>

* Added cache skip override to RelaunchExecution endpoint

Signed-off-by: Nick Müller <[email protected]>

* Updated to latest version of flytepropeller

Signed-off-by: Nick Müller <[email protected]>

* Renamed SkipCache flag to OverwriteCache
Updated to latest released versions of flyteidl, flyteplugins and flytestdlib
Updated to latest unmerged version of flytepropeller

Signed-off-by: Nick Müller <[email protected]>

* Updated flyteidl, flytepropeller and flytestdlib to latest released versions

Signed-off-by: Nick Müller <[email protected]>

* Reworded comment for clarity

Signed-off-by: Nick Müller <[email protected]>

Signed-off-by: Nick Müller <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@akhurana001 akhurana001

Labels
enhancement New feature or request
Projects
None yet
Milestone
0.7.0
Development

No branches or pull requests
2 participants
@wild-endeavor @akhurana001