[feature request] Support ordered execution of the application containers during in-place updates in CloneSet

[shuyingliang]

What would you like to be added:

We would like CloneSet to support ordered execution of the application containers during in-place updates.

Specifically,

• user can specify the priority for each application container in a pod.
• When in-place updates happen (e.g version change) in the multiple applications containers, the application containers stop and start based on the priority/ordering container by container.
• in-place update to a container (not multiple containers) with specified ordering just have effect to that container, and no impact to the other containers even though they are specified with orders.
• For containers that are not specified with the orders/priority, no guarantees of the execution order is fine.
• In the case of containers with specified orders fail at kubelet probes and gets restarted by kubelet, the ordering might not apply anymore. But it will be a good-to-have to ensure ordering restarting as well.

Why is this needed:

We believe this is a critical feature that can help us to leverage the CloneSet in-place updates successfully.

We have business logic have to be lined up for execution. While we can (and are) leveraging init container to ensure the execution ordering for some business logic. However, updates to the init containers will cause the whole pod to be recreated.
Even though we can do in-place update in the application container, the nature of the init container updates defeat the whole purpose of in-place updates.

While we can separate the business logic in init container per pod lifecycle to avoid updates in init container, but in practice, there are business logic is not per pod lifecycle but per container lifecycle and we have to line them up for ordered execution. With the feature support, we can put the business logic into long running service as in application container and achieve ordered in-place updates

[FillZpp]

@wgitg Thanks for the engaging discussion to make Kruise better :)

• user can specify the priority for each application container in a pod.
• When in-place updates happen (e.g version change) in the multiple applications containers, the application containers stop and start based on the priority/ordering container by container.
• in-place update to a container (not multiple containers) with specified ordering just have effect to that container, and no impact to the other containers even though they are specified with orders.

These looks good.

• For containers that are not specified with the orders/priority, no guarantees of the execution order is fine.

• For pods without container orders, no guarantees of the execution order during in-place update multiple containers.
• For pods with container orders:
  • keep execution order during in-place update multiple containers with different priorities.
  • no guarantees of the execution order during in-place update multiple containers with the same priority (you can see the two ways to configure the priorities).

• In the case of containers with specified orders fail at kubelet probes and gets restarted by kubelet, the ordering might not apply anymore. But it will be a good-to-have to ensure ordering restarting as well.

We might not care about the probe failure. When in-place update containers by order, we should just update the high-priority container first, and then the next if the first one has been started and ready (maybe for more than minReadySeconds).

What do you think?

[shuyingliang]

@FillZpp Thanks for the excellent support! Yes. Your comments looks good. 👍

[shuyingliang]

@FillZpp Thanks for the awesome support! I just would like to circle back on the stop ordering, are there ways to control he stop ordering for the containers when the pod dies?

[FillZpp]

@FillZpp Thanks for the awesome support! I just would like to circle back on the stop ordering, are there ways to control he stop ordering for the containers when the pod dies?

Yeah, we are planning to support stop ordering. Please create a new issue for it :) cc @zmberg