Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Default flytekit[all] to Pyspark 3.x #483

Closed
13 tasks
akhurana001 opened this issue Aug 25, 2020 · 1 comment
Closed
13 tasks

Default flytekit[all] to Pyspark 3.x #483

akhurana001 opened this issue Aug 25, 2020 · 1 comment
Assignees
@akhurana001
Labels
enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers
Milestone
0.11.0

Comments

@akhurana001
Copy link
Contributor

akhurana001 commented Aug 25, 2020 •
edited
Loading

Motivation: Why do you think this is important?
Currently we support all which defaults to Spark 2.4 and all3 which defaults to Spark 3.x . We should deprecate support for Spark 2.4 and move all to Spark 3.x.

Ref: #482

Goal: What should the final outcome look like, ideally?
all directive in flytekit defaults to Pyspark 3.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

[Optional] Propose: Link/Inline
If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document.

Additional context
Add any other context or screenshots about the feature request here.

Is this a blocker for you to adopt Flyte
Please let us know if this makes it impossible to adopt Flyte
@akhurana001 akhurana001 added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Aug 25, 2020
@akhurana001 akhurana001 mentioned this issue Aug 25, 2020
Merged
8 tasks
@wild-endeavor wild-endeavor added this to the 0.11.0 milestone Aug 25, 2020
@akhurana001 akhurana001 mentioned this issue Oct 4, 2020
Merged
8 tasks
@akhurana001 akhurana001 mentioned this issue Oct 14, 2020
Closed
8 tasks
This was referenced Jan 12, 2021
Merged
Merged
Closed
@akhurana001 akhurana001 self-assigned this Jan 14, 2021
@akhurana001
Copy link
Contributor Author

Done

@james-union james-union mentioned this issue May 23, 2022
Merged
8 tasks
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@pingsutw
Skip the structured dataset format check when the expected format is …
69158ea 
…empty (flyteorg#483)

Signed-off-by: Kevin Su <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@EngHabu
Add gsutil to all Dockerfiles (flyteorg#483)
e4495fc 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Jul 24, 2023
@hamersaw
Populate TaskNodeMetadata on CreateNodeExecutionModel to correctly di…
eb695b1 
…splay fast cache (flyteorg#483)

* adding cache information on first node event

Signed-off-by: Daniel Rammer <[email protected]>

* fixed unit tests

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@pingsutw
Skip the structured dataset format check when the expected format is …
0d961dc 
…empty (flyteorg#483)

Signed-off-by: Kevin Su <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@hamersaw
Populate TaskNodeMetadata on CreateNodeExecutionModel to correctly di…
9a03ca1 
…splay fast cache (flyteorg#483)

* adding cache information on first node event

Signed-off-by: Daniel Rammer <[email protected]>

* fixed unit tests

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@akhurana001 akhurana001

Labels
enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers
Projects
None yet
Milestone
0.11.0
Development

No branches or pull requests
2 participants
@wild-endeavor @akhurana001