[BUG] [BACKUP/RECOVERY] rsync command fails trying to copy artifacts

[przemyslavic]

Describe the bug
Backup command fails on rsync task with the following error: rsync error: syntax or usage error (code 1) at main.c(1344) [sender=3.1.2].

2022-02-02T19:54:15.2675029Z[38;21m19:54:15 INFO cli.engine.ansible.AnsibleCommand - TASK [backup : Use rsync to copy all artifacts] ********************************
2022-02-02T19:54:16.0768848Z[31;21m19:54:16 ERROR cli.engine.ansible.AnsibleCommand - failed: [ci-recazurcentcanal-kubernetes-master-vm-0 -> ci-recazurcentcanal-repository-vm-0(1.1.1.1)] (item=/epibackup/k8s_snapshot_20220202-195348.tar.gz) => {"ansible_loop_var": "item", "changed": false, "cmd": "/bin/rsync --delay-updates -F --compress --checksum --archive --rsync-path='sudo -u root rsync' --rsh=/usr/bin/ssh -S none -i /root/.ssh/ansible.Qk7_tI.tmp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --out-format='<<CHANGED>>%i %n%L' [email protected]:/epibackup/k8s_snapshot_20220202-195348.tar.gz /epibackup/mounted", "item": "/epibackup/k8s_snapshot_20220202-195348.tar.gz", "msg": "Unexpected remote arg: [email protected]:/epibackup/k8s_snapshot_20220202-195348.tar.gz\nrsync error: syntax or usage error (code 1) at main.c(1344) [sender=3.1.2]\n", "rc": 1}
2022-02-02T19:54:16.4286627Z[31;21m19:54:16 ERROR cli.engine.ansible.AnsibleCommand - failed: [ci-recazurcentcanal-kubernetes-master-vm-0 -> ci-recazurcentcanal-repository-vm-0(1.1.1.1)] (item=/epibackup/k8s_snapshot_20220202-195348.tar.gz.sha1) => {"ansible_loop_var": "item", "changed": false, "cmd": "/bin/rsync --delay-updates -F --compress --checksum --archive --rsync-path='sudo -u root rsync' --rsh=/usr/bin/ssh -S none -i /root/.ssh/ansible.Qk7_tI.tmp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --out-format='<<CHANGED>>%i %n%L' [email protected]:/epibackup/k8s_snapshot_20220202-195348.tar.gz.sha1 /epibackup/mounted", "item": "/epibackup/k8s_snapshot_20220202-195348.tar.gz.sha1", "msg": "Unexpected remote arg: [email protected]:/epibackup/k8s_snapshot_20220202-195348.tar.gz.sha1\nrsync error: syntax or usage error (code 1) at main.c(1344) [sender=3.1.2]\n", "rc": 1}

https://github.com/epiphany-platform/epiphany/blob/develop/ansible/playbooks/roles/backup/tasks/common/download_via_rsync.yml#L72

https://github.com/epiphany-platform/epiphany/blob/develop/ansible/playbooks/roles/recovery/tasks/common/upload_via_rsync.yml#L67

This is related to Ansible/Python update #2933.

Worth checking the remote shell we use: /usr/bin/ssh -S none -i {{ private_key_file.path }} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
https://github.com/epiphany-platform/epiphany/blob/develop/ansible/playbooks/roles/backup/tasks/common/download_via_rsync.yml#L84.
Maybe it's no longer needed or it should be within quotation marks

How to reproduce
Steps to reproduce the behavior:

1. execute epicli apply to deploy a cluster with e.g. kubernetes master component enabled
2. execute epicli backup -b /build/directory -f /backup/config.yml

Expected behavior
Backup should be successful

Config files
Backup config

---
kind: configuration/backup
title: Backup Config
provider: aws
name: default
specification:
  components:
    load_balancer:
      enabled: true
    logging:
      enabled: true
    monitoring:
      enabled: true
    postgresql:
      enabled: true
    rabbitmq:
      enabled: true
    kubernetes:
      enabled: true

Environment

• Cloud provider: found on Azure but probably provider doesn't matter
• OS: [RHEL, CentOS, Ubuntu]

epicli version: 2.0.0dev

Additional context

---

DoD checklist

• Changelog
  • updated
  • not needed
• COMPONENTS.md
  • updated
  • not needed
• Schema
  • updated
  • not needed
• Backport tasks
  • created
  • not needed
• Documentation
  • added
  • updated
  • not needed
• Feature has automated tests
• Automated tests passed (QA pipelines)
  • apply
  • upgrade
  • backup/restore
• Idempotency tested
• All conversations in PR resolved

[przemyslavic]

From what I've already tested this should do the trick:
https://github.com/epiphany-platform/epiphany/blob/c687e8eca0347fda6dd7a0b092deafeccdb53fb6/ansible/playbooks/roles/backup/tasks/common/download_via_rsync.yml#L79

          - --rsh="{{ rsh }}"

https://github.com/epiphany-platform/epiphany/blob/c687e8eca0347fda6dd7a0b092deafeccdb53fb6/ansible/playbooks/roles/recovery/tasks/common/upload_via_rsync.yml#L74

          - --rsh="{{ rsh }}"

But it might be worth refactoring this task:

        # this fixes / replaces incorrect path to the private key file that synchronize provides
        # (setting private_key parameter has no effect whatsoever, looks like a bug tbh)

[przemyslavic]

Fixed rsync command.
Improved validation for backup/recovery.
Configuration for backup/recovery can be placed in a separate file (recommended) or attached to the cluster configuration (or any yaml file).