Introduce Progressive status

[darkowlzz]

Depends on fluxcd/pkg#422

This change introduces progressive status across all the reconcilers. It uses the helpers and checks proposed in fluxcd/pkg#422 to implement and verify the result of the change.

Mid-reconciliation status

Before this change, the reconcilers patched the target object of reconciliation only once, at the end of the reconciliation to report the final result of the reconciliation. The change in the status that took place during reconciliation were not reflected on the object. As described in #936, this resulted in stale status reported on the object. This also resulted in not reporting the progressive status of the reconciliation, which made it hard to understand if an object is undergoing reconciliation or not, refer #935 .

In order to perform multiple status updates on the object during reconciliation, a serial patcher was introduced in fluxcd/pkg#379 . This patcher is a drop-in replacement for the previous patch helper. It keeps track of the previously patched object and constructs subsequent patch based on the previously patched object.

To perform mid-reconciliation patching, all the sub-reconcilers need an instance of the patcher. This change extends the sub-reconciler function signature to allow passing the patcher.

When reconciliation of an object begins, the object is immediately marked with Reconciling=True. If it's a new object generation or reconciliation is manually requested using the reconcile annotation value, the reconciling status is patched and reported on the object.

In the reconcileStorage() phase, if there's no artifact in the storage or an existing artifact disappeared from the storage, it is considered as a drift and is immediately reported on the object by patching the status.

In the reconcileSource() phase, if a drift is detected in the artifact or the included artifact, it is again considered as a drift and is immediately reported on the object by patching the status.

All the sub-reconciler tests have been updated accordingly with in-progress status condition check.

Reconciled status

In order to indicate that a reconciliation has resulted in a failure and reconciliation will be retried, the Reconciling=True condition is updated with ProgressingWithRetry reason. Kustomize-controller already does this.

After a full reconciliation, if the reconciliation didn't succeed and Reconciling is still True, the ComputeReconcileResult() helper is updated to add the Reconciling=True reason with ProgressingWithRetry.
Subsequent reconciliation updates the Reconciling=True reason with Progressing to indicate that a reconciliation is in progress.

SummarizeAndPatch() has been updated to replace the previous patch helper with the serial patcher.

Fixes #936, #935

---

Snippets showing the result of this change with status output:

• GitRepository object status https://gist.github.com/darkowlzz/6912540b359d199175101eaeaa72c359
• HelmRepository OCI object status https://gist.github.com/darkowlzz/2961a21ac3519be960327bdc7020da1e

Except for HelmRepository OCI reconciler, all the other reconcilers have similar implementations. Their status results should be similar.

[hiddeco]

This all looks sensible to me, thank you @darkowlzz 🥇