[Core feature] External WebAPI plugins: write plugins with FastAPI server without having to compile

[cosmicBboy]

Motivation: Why do you think this is important?

Currently it is hard to implement backend plugins, especially for data-scientists & MLE’s who do not have working knowledge of Golang. Also, performance requirements, maintenance and development is cumbersome. The document here proposes a path to make it possible to write plugins rapidly, while decoupling them from the core flytepropeller engine.

Plugins should be easy to author - no need of code generation, using tools that MLEs and Data Scientists are not accustomed to using.

Goal: What should the final outcome look like, ideally?

• Most important plugins for Flyte today are plugins that communicate with external services.
• It should be possible to test these plugins independently and also deploy them privately.
• It should be possible for users to use backend plugins for local development, especially in flytekit and unionML
• It should be possible to author plugins in any language.
• Plugins should be scalable
• Plugins should have a very simple API
• Plugins should show up in the UI (extra details)

Describe alternatives you've considered

NA

Propose: Link/Inline OR Additional context

https://unionai.atlassian.net/wiki/spaces/SE/pages/143425541/Managing+Cluster+Config#Get-cluster-data

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes