Incremental conditions patching leads to reconciliation errors

[gab-satchi]

While adding conditions to AWSMachine and other CAPA resources, we wanted to show incremental updates using conditions for long running processes that do a busy wait. For ex: InstanceProvisionStarted

In the same reconciliation loop, the same condition can be updated further down the line for other reasons. When the reconcile finishes, patch is called which attempts to patch conditions but sees a conflict as the condition has already been updated to show the incremental progress. A subsequent reconcile will be successful.

Some half baked ideas:

• Allow bypassing the comparison with latest altogether. Will lead to the newest 'Apply' becoming the source of truth but since subsequent reconciles will succeed anyway this might already be the case.
• Convert each of the incremental updates to be Conditions themselves and not Reasons. With dedicated conditions, they won't conflict with others but may lead to a lot of noise in the Status.
• Have conditions patching consider the severity. An Error can override a Warning which can override an Info even if it differs from the before object’s conditions

/kind design
cc @fabriziopandini

[fabriziopandini]

Condition patching was originally designed to handle concurrency between two controllers, and in that case each controller was expected to control different conditions. As a consequence, the conflict resolution logic was implemented "at best effort" mimicking the three-way merge.

This is actually a new use case because the same controller is patching the same conditions both incrementally and in the deferred patch, with the latest expected to have priority in conflict resolution.

My first reaction is that we should declare when we are in this case, eventually specifying the list of conditions owned by the controller. But let me dig a little bit more

[fabriziopandini]

/assign
/lifecycle active