Support reference model for workloads

[resouer]

Summary

What change needs making?

Consider change the embedded mode to reference mode, so we could have more flexibility on workload side.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-bluegreen
spec:
  workloadRef: 
    apiVersion: apps/v1
    kind: Deployment
    name: demo1
  strategy:
    blueGreen: 
      activeService: rollout-bluegreen-active
      previewService: rollout-bluegreen-preview
      autoPromotionEnabled: false

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-bluegreen
spec:
  workloadRef: 
    apiVersion: apps/v1
    kind: ReplicaSet
    name: demo2
  strategy:
    blueGreen: 
      activeService: rollout-bluegreen-active
      previewService: rollout-bluegreen-preview
      autoPromotionEnabled: false

Use Cases

To support we use Deployment or our own stateless workload (like CloneSet).

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[jessesuen]

The challenge of doing something like this:

spec:
  workloadRef: 
    apiVersion: apps/v1
    kind: Deployment
    name: demo1

is that the rollout-controller has no control over how the underlying ReplicaSet/Pods are created, and thus it cannot control things such as:

1. control the individual scale of the canary vs. stable.
2. perform last-mile injection of pod spec adjustments (e.g. inter-revision anti-affinity, or ephemeral canary labels)

FWIW, Flagger does take the reference model approach. Its Canary CRD references a Deployment object. However, in order for Flagger to control scale of the canary vs. stable, it needs to make a "shadow copy" of the Deployment object and related objects (e.g. HPA), so that it can adjust the scale and labels appropriately. In flagger, the referenced Deployment object is mostly used as a pod template definition, since the shadow copy is the eventual deployment that ultimately serves traffic to users at a steady state. The referenced object is used during updates, but it is eventually scaled to 0 at a steady state.

If there was a way to use a reference model without resorting to creating shadow copies of the original object, I would support this model.

[resouer]

@jessesuen The motivation is indeed to we are trying to enable other system to manage workload by itself, for example create and manage workload revisions.

In CloneSet, it generates revision labels to group old/new pods during upgrade. Another example is ContainerizedWorkload in OAM, it generates deployment-v1, deployment-v2... for different revisions.

So the solution in my mind could be introduce a standard contract between external workload controller and rollout controller.

[jessesuen]

We're investigating a way to do a reference model due to the following motivations:

• CRDs (e.g. Rollouts) are not supported well in kustomize, and strategic merge patches simply don't work as expected with a Rollout because lists will be replaced and not merged. By referencing a native Kubernetes kind, kustomize would work expectedly against the k8s native referenced object, which is the portion of the spec that users typically want to customize overlays against.
• During a migration from a Deployment to a Rollout, it has been inconvenient for users to duplicate the entire Deployment spec to a Rollout, and keeping them always in sync during the transition. By referencing the definition, we would be able to eliminate the possibility of pod template spec duplication.

However, the way we are thinking this will work, probably won't work the way you expect based on the previous comment. Our thought is to reference an object containing a PodTemplateSpec (e.g. a Deployment, PodTemplate), and monitor the referenced object as simply a configuration object holding the definition of the pod template. However, a Rollout will still create and own the ReplicaSets (like it does currently)

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-bluegreen
spec:
  workloadRef: 
    apiVersion: apps/v1
    kind: Deployment
    name: demo
  replicas: 5
  strategy:
    blueGreen: 
      activeService: rollout-bluegreen-active
      previewService: rollout-bluegreen-preview
      autoPromotionEnabled: false

In the above example, the demo Deployment would be only used as a reference to contain the pod template. It could still be left scaled up, but the only reason you would want to leave it scaled up is if you were transitioning from a Deployment to a Rollout. Eventually, you would either want to scale the demo deployment to 0, or delete the deployment and inline the pod template spec in the Rollout.

[resouer]

Yeah, I can get the motivation of your new proposal (i.e. treating user's Deployment as a "UI"). And yes, it can not solve the problem that we need to rollout the custom workloads like CloneSet etc. I'd assume it could address some folks' concerns (i.e. they can not use Deployment.