[ENH] - Scheduled Nebari Backups

[pt247]

Feature description

We want to expand the existing configuration to allow it to specify backup parameters.
Backing up Nebari includes the following:

1. KeyCloak user details.
2. User data
3. Conda environments

Interface

backup
  location: <BACKUP LOCATION>>
  schedule: <CRON EXPRESSION>
  retention_period: <NUMBER OF DAYS>

Implementation

There are several ways this can be achieved:

1. Kubernetes CronJob
   1. Pros:
      1. Native to K8
      2. Good for simple disconnected tasks
   2. Cons:
      2. Will need to custom build and maintain status, monitoring, and alerting.
2. Argo Workflows
   1. Pros:
      1. Fully featured UI
      2. Native artifact support
      3. Workflow archive
      4. Rest API
   2. Cons:
      1. Argo becomes a core component in Nebari.
      2. Most of our workflows might not be DAGs; it's possible that all the tasks can be independently executed.

Value and/or benefit

1. It will save the state from accidental deletes.

Anything else?

No response

Tasks

Preview Give feedback

1. Add the pros and cons of each of the available options.
2. Discuss available options

[Adam-D-Lewis]

BTW, I think you would use k8s Jobs instead of k8S Cronjobs. Also, one option to run things sequentially with k8s Jobs - https://stackoverflow.com/questions/40713573/how-to-run-containers-sequentially-as-a-kubernetes-job (haven't tried it myself, but worth a try)

[pt247]

Closing in favor of #2518