[FEATURE REQUEST] Optimize downloading requirements introducing file checksums, use Python instead of Bash

[to-bar]

Is your feature request related to a problem? Please describe.

1. Downloading all requirements takes about 12 min (Azure RHEL), sometimes longer.
2. Each time Epiphany downloads all requirements, there is no option to download only required ones - e.g. if there is no Ignite component enabled, its heavy binary is downloaded unnecessarily.
3. requirements.txt file doesn't contain files' checksums so we are not able to check if existing file was fully downloaded or it's corrupted. For this reason, files are re-downloaded each time (for RHEL) or may be corrupted (for Ubuntu).
4. requirements.txt file in current format doesn't allow to have flexible (structured) configuration (for checksums or dependencies).
5. We have redundant code so if you want to make a change in download-requirements.sh you may need to do it for every OS directory.

Describe the solution you'd like

1. Download any file only once. If exists, compare chechsums. If checksums differs, then re-download.
2. Switch from txt files with custom format to YAML.
3. Support 2 download modes: full (current solution) and adaptive (download only files required by enabled components).
4. Support multiple repositories for the same package.
5. Use the same code for all OS distribution when possible.
6. Support retries for network related tasks which may succeed on another try.
7. Ansible seems to meet the above requirements.

---

DoD checklist

• Changelog
  • updated
  • not needed
• COMPONENTS.md
  • updated
  • not needed
• Schema
  • updated
  • not needed
• Backport tasks
  • created
  • not needed
• Documentation
  • added
  • updated
  • not needed
• Feature has automated tests
• Automated tests passed (QA pipelines)
  • apply
  • upgrade
  • backup/restore
• Idempotency tested
• All conversations in PR resolved

[rafzei]

I've check how messed this role is during #127...

We should keep consistency of the repository role for supported OS's.

Proposed structure of files directory in repository role:

repository role
- files (dir)
  - client (dir)
    - common (dir contain common files with functions)
      - common_functions.sh (example file with functions for all OSs)
      - ...
    - debian.sh (this file should source common functions)
    - redhat.sh (same)
  - download-requirements (dir)
    - common (dir contain common files with functions)
      - common_functions.sh (example file with functions for all OSs)
    - centos-7
      - download-requirements.sh (this file should source common functions)
      - requirements.<arch>.txt
    - redhad-7
      - ...
    - ubuntu-18.04
      - ...
  - server
    - ... (same as above)

[rafzei]

#2378 , #804 Can be done parallel

[rafzei]

I was discussing with @to-bar the possibilities of refactoring this role. We agreed that if possible, download-requirements should be rewritten to python. It will give us the possibility to keep requirements.txt in a more structured file type like JSON. Another thing we should consider is adding md5sum to this file, so we will be able to verify if file was downloaded successfully and skip re-downloading in case of re-apply.

[romsok24]

#2778 is related.