Improve Error Handling + Code refactor

[hossainemruz]

Fixes: #881

Error and Metrics Handling:

• BackupConfiguration Controller:
  • Failed to trigger workload controller:
    • Write event to BackupConfiguration
    • Log error
  • Failed to ensure cronjob:
    • Write event to BackupConfiguration
    • Log error
• BackupSession Controller:
  • Failed to ensure backup job:
    • Write event to BackupSession
    • Log error
    • Fail BackupSeesion
  • Backup session failed:
    • Set BackupSession phase to "Failed"
    • Write event to BackupSession
    • Send backup session specific metrics
  • BackupSession succeeded:
    • Set BackupSession phase to "Succeeded"
    • Write event to BackupSession
    • Send backup session specific metrics
• Auto Backup:
  • Failed/Succeeded to create/delete auto backup resources
    • Write event to the respective resource
    • Write log
• Workload Controller:
  • Failed/Succeeded to inject/delete stash sidecar/init-container
    • Write event to workload
    • Write log
• RestoreSession Controller:
  • Failed to create restore job
    • Write event to RestoreSession
    • Write log
    • Fail RestoreSession
  • Restore session failed:
    • Set RestoreSession phase to "Failed"
    • Write event to RestoreSession
    • Send restore session specific metrics
  • RestoreSession succeeded:
    • Set RestoreSession phase to "Succeeded"
    • Write event to RestoreSession
    • Send restore session specific metrics
• Sidecar:
  • Failed to start BackupSession controller
    • Write event to BackupConfiguration
    • Write log
    • Fail the container so it restart
  • Succeeded to start BackupSession controller
    • Write event to BackupConfiguration
    • Write log
  • Failed/Succeeded to take backup
    • Update status of the BackupSession
    • Write log
• Init-container/Backup Job:
  • Failed/Succeeded to restore
    • Update status of the RestoreSession
    • Write log
    • Fail the container

[codecov-io]

Codecov Report

Merging #891 into master will decrease coverage by 0.17%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #891      +/-   ##
==========================================
- Coverage   32.21%   32.03%   -0.18%     
==========================================
  Files          13       13              
  Lines        1279     1286       +7     
==========================================
  Hits          412      412              
- Misses        798      805       +7     
  Partials       69       69

Impacted Files	Coverage Δ
pkg/restic/metrics.go	0% <0%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 88c8427...fcfbecc. Read the comment docs.