[feat] Enable regular reconciliation of promise workflows

[abangser]

With Kratix being a Kubernetes controller, it is expected that it could reconcile at any point. The most common triggers are:

1. Changes
2. Restarts of the controller

It is also common to have a regular interval for that reconciliation to enable drift detection and re-alignment in between the active triggers mentioned above.

Kratix should have a sensible default for regular reconciliations and also allow user override of this number within a reasonable range.

Assumptions

• The forced-reconciliation should not happen very often by default
  • The re-execution of the pipeline is an expensive operation
  • In the future, we will have healthchecks that would tell Kratix when something has drifted
• User will want to configure the interval
• Re-reconciling on a Promise should not re-reconcile all of its resources
  • we may want to add support for this in the future

Scenarios

Feature: Default reconciliation time
  Scenario: Promise workflow is run at least every 10 hours
    Given the Kratix manager has not restarted in the last 10 hours
    When 10 hours have passed
    Then the Promise workflows should be triggered 

    Given the Promise workflow has run recently
    When the planned 10 hour reconciliation time occurs
    Then the Promise workflow does not run unnecessarily

Feature: documentation
  Given a user wanting to know more about Kratix reconciliation
  When they go to the docs
  Then they can find information on the default values

Note: While valuable there are a few outstanding aspects that will make this more complete. Referencing these in the docs will be helpful. These will all be covered in additional pieces of work:

1. This will not affect the statestores unless the workflow output documents have changed. I.e. if someone has deleted a file from git, but the expected files have not changed, this will not rewrite the file to git.
2. This will not rerun resource pipelines as those will be done in a different story

Engineering notes

• We should not try to "detect" drift, but rather just re-execute the pipelines every X hours.

TODO List

• Validate the writer will rewrite if we update a file in the state store
  • We tested this and found that files were not rewritten so this needs to be actively ensured as part of a different story
• Validate behaviour when pipeline is in failure state (we are checking on the PipelineCompleted condition)
  • The pipeline will not get rerun. This was deemed the right call since if the pipeline is in crashloopbackoff or another failure mode it most likely should not be rerun without a human intervention.
• Validate if our requeue logic will be noisy on manager restarts
  • no, in anything it is not effective enough! If the kratix controller restarts, there may be less reconciliation loops scheduled as the 10 hours is recalculated on controller boot. Therefore, in a worst case scenario, if the controller is restarted every 9h and 59m we will not have this job ever automatically run.

[abangser]

This should be configurable by the operator. 👈 this will be covered in #228 which is where configurability is introduced.

[ChunyiLyu]

This can be achieved by customizing manager cache options. See: https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/cache/cache.go#L170

[SaphMB]

Documentation

• FAQs - ❌ out of place on reflection
• Workflows (Promise)
• Workshop RE Idempotency - ❌ the workshop only dives into resource configure workflows