[Feature] Permit enabling plugins for certain projects + domains

[katrogan]

Motivation: Why do you think this is important?
Currently an enabled plugin is applied for all matching task types. This poses several problems. As part of the process for deploying and testing new plugins, there is no means to gradually introduce one plugin implementation as a replacement for a given task type. Furthermore, sometimes individual users are interested in one implementation vs another (e.g. k8s array tasks vs batch for having access to credentials).

Goal: What should the final outcome look like, ideally?
Alternate plugin implementations for a task type should be configurable via a project+domain whitelist. e.g.

tasks:
  task-plugins:
    enabled-plugins:
    - container
    - aws_array
    configurable-plugins:
    - k8s-array:
      - production:
        - my_project
        - my_other_project

In this example, all projects+domains will always have the container plugin enabled. However, the container_array implementation will default to the aws batch plugin. In the case of executions running in the production domain for my_project or my_other_project the container_array plugin used will be k8s-array since it is registered for the same task type.

Describe alternatives you've considered
Another means of permitting split testing for a replacement plugin implementation could include a percentage-based traffic split. However this is non-deterministic which leads to confusing behavior for customer teams.

Flyte component

• Overall
• Flyte Setup and Installation scripts
• Flyte Documentation
• Flyte communication (slack/email etc)
• FlytePropeller
• FlyteIDL (Flyte specification language)
• Flytekit (Python SDK)
• FlyteAdmin (Control Plane service)
• FlytePlugins
• DataCatalog
• FlyteStdlib (common libraries)
• FlyteConsole (UI)
• Other

[Optional] Propose: Link/Inline
This change will require updating WranglePluginsAndGenerateFinalList and plugin selection in ResolvePlugin

Additional context
This is to facilitate testing out improvements to the k8s array plugin.

Is this a blocker for you to adopt Flyte
N/A

[katrogan]

@EngHabu @kumare3 mind reviewing?
cc @migueltol22

[kumare3]

@katrogan thank you for filing this issue, this makes sense to me. The intention of WranglePluginsAndGenerateFinalList was to allow much complex configuration, and eventually also supporting optional roll-outs

[EngHabu]

I'm 💯 behind the proposal you outlined here instead of the approach outlined above.
The basic principle is: system-level configs live next to the component they configure, project/wf/user specific config live in Admin DB (manipulatable by privileged users/admins)

[katrogan]

@EngHabu thanks! since the other issue encapsulates this one, I'll close this out and use the linked issue for tracking work items