Scale out with propeller manager and workflow sharding

[hamersaw]

TL;DR

Adding a FlytePropeller Manager component which is responsible for configuring and ensuring liveness over a collection of FlytePropeller instances. FlyteWorkflow CRDs are effectively sharded over these instances, so that there is a deterministic one-to-one relationship. this enables horizontally scaling FlytePropeller.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

Functionality has been implemented in accordance to the accepted RFC.

Tracking Issue

flyteorg/flyte#125

Follow-up issue

NA

[codecov]

Codecov Report

Merging #351 (f794181) into master (990dc8e) will decrease coverage by 0.09%.
The diff coverage is 39.76%.

[kumare3]

Suggested change

	labelSelectorRequirement := v1.LabelSelectorRequirement{"shardKey", v1.LabelSelectorOpNotIn, cfg.IncludeShardKey}
	labelSelectorRequirement := v1.LabelSelectorRequirement{"shardKey", v1.LabelSelectorOpNotIn, cfg.ExcludeShardKey}

[kumare3]

how will we use exclude? isnt this harder?

[hamersaw]

exclude is being used for the "enableUncoveredReplica" option. mainly for the project and domain shard strategies, it's only included on the shard key option for completeness - but it's easy to remove.

[kumare3]

@hamersaw I was talking with @EngHabu and he mentioned one important thing. You should create all the pods, with an owner-reference. The owner reference should be the same as - the flytepropeller deployment owner reference that is populated on the manager.
Thus the manager should pass down its own owner-reference to the propeller pods.

This will enable that when you delete the deployment it will automatically delete all the pods that were created

[hamersaw]

@hamersaw I was talking with @EngHabu and he mentioned one important thing. You should create all the pods, with an owner-reference. The owner reference should be the same as - the flytepropeller deployment owner reference that is populated on the manager. Thus the manager should pass down its own owner-reference to the propeller pods.

This will enable that when you delete the deployment it will automatically delete all the pods that were created

This is completed, to track this we added the owner references (if they exist) from the pod which the manager is started in to the manager pods.

This approach differs from the previous shutdown hook which deleted all managed pods when the manager exited. Meaning a failure in the manager would delete them as well as a successful shutdown. In our new approach, failures in the manager pod do not affect managed pods.

[EngHabu]

A couple of nits

[EngHabu]

Feel free to ignore...