Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DOC] Document how to write backend plugins, the different plugin types and examples #681

Closed
1 of 13 tasks
EngHabu opened this issue Jan 29, 2021 · 1 comment
Closed
1 of 13 tasks

[DOC] Document how to write backend plugins, the different plugin types and examples #681

EngHabu opened this issue Jan 29, 2021 · 1 comment
Labels
documentation Improvements or additions to documentation plugins Plugins related labels (backend or frontend)

Comments

@EngHabu
Copy link
Contributor

EngHabu commented Jan 29, 2021

Describe the bug
Document how to develop new backend plugins, the supported plugin types, the iteration, validation and testing story. How to expose metrics and write logs. How to maintain the statemachine and best practices to avoid a corrupt or an inconsistent state.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other
@EngHabu EngHabu added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers labels Jan 29, 2021
@kumare3 kumare3 added documentation Improvements or additions to documentation and removed untriaged This issues has not yet been looked at by the Maintainers bug Something isn't working labels Mar 18, 2021
@kumare3 kumare3 mentioned this issue Oct 1, 2021
Merged
@SandraGH5 SandraGH5 added the plugins Plugins related labels (backend or frontend) label Nov 18, 2021
palchicz pushed a commit to palchicz/flyte that referenced this issue Dec 23, 2021
@eapolinario @pingsutw
@wild-endeavor @cosmicBboy
Add official support for python 3.9 (flyteorg#687)
ae15c4e 
* Run 3.9 in CI

Signed-off-by: Eduardo Apolinario <[email protected]>

* Also run 3.9 in plugins tests.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* Account for the different exception type raised in case of wrong types

Signed-off-by: Eduardo Apolinario <[email protected]>

* Exclude spark2

Signed-off-by: Eduardo Apolinario <[email protected]>

* Failed to load json_data to dataclass (flyteorg#684)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove sync singledispatch, add option for top-level only sync (flyteorg#681)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Failed to transform path string to Literal (flyteorg#689)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Don't node sync on remote wait (flyteorg#690)

* no node sync

Signed-off-by: Yee Hing Tong <[email protected]>

* add param to wait

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix plugin regressions (flyteorg#688)

* fix pandera regression

Signed-off-by: Niels Bantilan <[email protected]>

* install plugin with pip

Signed-off-by: Niels Bantilan <[email protected]>

* fix pandera plugin tests

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* add spark flytekit plugin to papermill test_requires

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* add sqlalchemy to great expectations plugin

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* wip

Signed-off-by: Niels Bantilan <[email protected]>

* plugins plugins plugins!

Signed-off-by: Niels Bantilan <[email protected]>

* lint

Signed-off-by: Niels Bantilan <[email protected]>

* Exclude does not understand lists

Signed-off-by: Eduardo Apolinario <[email protected]>

* Invert python version check

Signed-off-by: Eduardo Apolinario <[email protected]>

* Invert python version check for real this time

Signed-off-by: Eduardo Apolinario <[email protected]>

* Enable 3.10 just for kicks

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add quotes around python versions

Yes, this is needed, please see https://dev.to/hugovk/the-python-3-1-problem-85g.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Add quotes around python versions"

This reverts commit 4d619d5c9e829aa1fdc18d2d87e1dbe4dbf42372.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Enable 3.10 just for kicks"

This reverts commit bd6d69462cb722afc5e2eb09ce77e35da512d288.

Signed-off-by: Eduardo Apolinario <[email protected]>

* wip - restricted types

Signed-off-by: Eduardo Apolinario <[email protected]>

* wip - restricted types

Signed-off-by: Eduardo Apolinario <[email protected]>

* Comment use of restricted types in get_transformer

Signed-off-by: Eduardo Apolinario <[email protected]>

* Publish 3.9 image

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add python 3.9 to the list of supported languages

Signed-off-by: Eduardo Apolinario <[email protected]>

* Comment RestrictedTypeTransformer and add one test case.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Review feedback

Signed-off-by: Eduardo Apolinario <[email protected]>

* Comment RestrictedTypeTransformer

Signed-off-by: Eduardo Apolinario <[email protected]>

* Handle the TypeError

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove breakpoint

Signed-off-by: Eduardo Apolinario <[email protected]>

Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
@cosmicBboy
Copy link
Contributor

this issue is covered in #1541

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation Improvements or additions to documentation plugins Plugins related labels (backend or frontend)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@EngHabu @cosmicBboy @kumare3 @SandraGH5