[Core feature] Add default execution cluster option for a multi-cluster setup

[Antaxify]

Motivation: Why do you think this is important?

In our company, we run an specialized on-premise cluster and want to join it as an execution cluster to our Flyte installation in the cloud. This way we can easily interchange results between on on-premise / cloud and write launch plans using both clusters using launch plans. However, we only want specific workflows/tasks to run on the on-premise cluster.

In a Flyte multi-cluster setup, we can control which cluster is used by setting "execution cluster labels" on projects/workflows. However, if an execution does not match a any label, a cluster is randomly chosen from all enabled clusters. This is not ideal if one of the clusters is in any way specialized and may only execute certain workloads.

To achieve this, we would have to configure an execution cluster label on each and every project (we have quite a few) and tell people to not forget configuring it when a new project is created.

Goal: What should the final outcome look like, ideally?

The idea is to extend the clusters config by adding a defaultExecutionLabel as shown in this example below. If no execution label matches for an execution, the configured defaultExecutionLabel is used. This way, we can easily configure one or more default execution clusters and even apply weights on them.

This feature would be purely opt-in: If defaultExecutionLabel is not set, the original behavior of selecting a cluster at random is preserved.

clusters:
  defaultExecutionLabel: defaultclusters
  labelClusterMap:
    defaultclusters:
      - id: defaultcluster1
        weight: 0.5
      - id: defaultcluster2
        weight: 0.5
    specializedcluster:
      - id: specializedcluster
        weight: 1
  ...

Describe alternatives you've considered

We could set an execution cluster label on every project to enforce using the "default cluster". However we found this cumbersome and "yet another thing to think-of/remember"

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[Antaxify]

We already have a working solution for this and would like to contribute it, if the outlined solution is acceptable.