[BUG] [Ubuntu] System repositories are not restored on epicli re-run

[przemyslavic]

Describe the bug
Assuming the build fails, the system repositories should be restored on re-run to be able to download the requirements and proceed with the installation. However, the script fails as it cannot locate packages to download:

2022-01-18T13:31:43.6671351Z[38;21m13:31:43 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Run download-requirements script, this can take a long time
2022-01-18T13:31:43.6674861Z[38;21m13:31:43 INFO cli.engine.ansible.AnsibleCommand - You can check progress on repository host with: journalctl -f -t download-requirements.sh] ***
2022-01-18T13:31:54.7584710Z[31;21m13:31:54 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ec2-1-1-1-1.eu-west-1.compute.amazonaws.com]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /var/tmp/epi-download-requirements/download-requirements.sh /var/www/html/epirepo --no-logfile |& tee >(systemd-cat --identifier=download-requirements.sh)", "delta": "0:00:10.231158", "end": "2022-01-18 13:31:54.655743", "msg": "non-zero return code", "rc": 100, "start": "2022-01-18 13:31:44.424585", "stderr": "", "stderr_lines": [], "stdout": "Detected arch: x86_64
Docker platform: linux/amd64
Testing apt connection:
- /etc/apt/sources.list...
Connection successful.
Dependency list: /var/tmp/epi-download-requirements/.dependencies
Command used to download packages: run_cmd_with_retries 3 apt-get download
Destination directory for packages: /var/www/html/epirepo/packages
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb http://apt.kubernetes.io/ kubernetes-xenial main
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb https://packages.erlang-solutions.com/ubuntu focal contrib
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu bionic main
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb https://d3g5vo6xdbdb9a.cloudfront.net/apt stable main
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb http://apt.postgresql.org/pub/repos/apt focal-pgdg main
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb http://apt-archive.postgresql.org/pub/repos/apt focal-pgdg-archive main
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
deb https://dl.2ndquadrant.com/default/release/apt focal-2ndquadrant main
Testing apt connection:
- 2ndquadrant-dl-default-release.list...
Connection successful.

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Ign:2 https://packages.erlang-solutions.com/ubuntu focal InRelease
Hit:3 https://packages.erlang-solutions.com/ubuntu focal Release
Hit:4 http://apt.postgresql.org/pub/repos/apt focal-pgdg InRelease
Hit:5 https://download.docker.com/linux/ubuntu focal InRelease
Hit:6 https://d3g5vo6xdbdb9a.cloudfront.net/apt stable InRelease
Hit:1 https://apt-archive.postgresql.org/pub/repos/apt focal-pgdg-archive InRelease
Hit:9 https://dl.2ndquadrant.com/default/release/apt focal-2ndquadrant InRelease
Hit:10 https://artifacts.elastic.co/packages/oss-6.x/apt stable InRelease
Hit:11 https://artifacts.elastic.co/packages/oss-7.x/apt stable InRelease
Hit:7 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Hit:12 https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu bionic InRelease
Reading package lists...
Building dependency tree...
Reading state information...
5 packages can be upgraded. Run 'apt list --upgradable' to see them.

Crane binary already exists

Testing crane connection:
- haproxy...
Connection successful.
Resolving dependencies to download. This might take a while and will be cached in /var/tmp/epi-download-requirements/.dependencies
Package read from requirements file: adduser
Package to download: adduser
Package read from requirements file: apt-transport-https
Package to download: apt-transport-https
Package read from requirements file: auditd
Package to download: auditd
Package read from requirements file: bash-completion
Package to download: bash-completion
Package read from requirements file: ca-certificates
Package to download: ca-certificates
Package read from requirements file: cifs-utils
Package to download: cifs-utils
E: No packages found", "stdout_lines": ["Detected arch: x86_64", "Docker platform: linux/amd64", "Testing apt connection:", "- /etc/apt/sources.list...", "Connection successful.", "Dependency list: /var/tmp/epi-download-requirements/.dependencies", "Command used to download packages: run_cmd_with_retries 3 apt-get download", "Destination directory for packages: /var/www/html/epirepo/packages", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb http://apt.kubernetes.io/ kubernetes-xenial main", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb https://packages.erlang-solutions.com/ubuntu focal contrib", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu bionic main", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb https://d3g5vo6xdbdb9a.cloudfront.net/apt stable main", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb http://apt.postgresql.org/pub/repos/apt focal-pgdg main", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb http://apt-archive.postgresql.org/pub/repos/apt focal-pgdg-archive main", "Warning: apt-key output should not be parsed (stdout is not a terminal)", "OK", "deb https://dl.2ndquadrant.com/default/release/apt focal-2ndquadrant main", "Testing apt connection:", "- 2ndquadrant-dl-default-release.list...", "Connection successful.", "", "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.", "", "Ign:2 https://packages.erlang-solutions.com/ubuntu focal InRelease", "Hit:3 https://packages.erlang-solutions.com/ubuntu focal Release", "Hit:4 http://apt.postgresql.org/pub/repos/apt focal-pgdg InRelease", "Hit:5 https://download.docker.com/linux/ubuntu focal InRelease", "Hit:6 https://d3g5vo6xdbdb9a.cloudfront.net/apt stable InRelease", "Hit:1 https://apt-archive.postgresql.org/pub/repos/apt focal-pgdg-archive InRelease", "Hit:9 https://dl.2ndquadrant.com/default/release/apt focal-2ndquadrant InRelease", "Hit:10 https://artifacts.elastic.co/packages/oss-6.x/apt stable InRelease", "Hit:11 https://artifacts.elastic.co/packages/oss-7.x/apt stable InRelease", "Hit:7 https://packages.cloud.google.com/apt kubernetes-xenial InRelease", "Hit:12 https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu bionic InRelease", "Reading package lists...", "Building dependency tree...", "Reading state information...", "5 packages can be upgraded. Run 'apt list --upgradable' to see them.", "", "Crane binary already exists", "", "Testing crane connection:", "- haproxy...", "Connection successful.", "Resolving dependencies to download. This might take a while and will be cached in /var/tmp/epi-download-requirements/.dependencies", "Package read from requirements file: adduser", "Package to download: adduser", "Package read from requirements file: apt-transport-https", "Package to download: apt-transport-https", "Package read from requirements file: auditd", "Package to download: auditd", "Package read from requirements file: bash-completion", "Package to download: bash-completion", "Package read from requirements file: ca-certificates", "Package to download: ca-certificates", "Package read from requirements file: cifs-utils", "Package to download: cifs-utils", "E: No packages found"]}

The reason is that the condition to restore system repos is if [[ ! -f /etc/apt/sources.list ]]; then
https://github.com/epiphany-platform/epiphany/blob/develop/ansible/playbooks/roles/repository/files/download-requirements/ubuntu-20.04/download-requirements.sh#L60
However, there is a case that this file actually exists (that's why the condition is not met) but it's empty.
Probably it gets created by some system job/apt tool.

How to reproduce
Steps to reproduce the behavior:

1. execute epicli apply
2. cancel the build after configuring the repository
3. execute epicli apply again

Expected behavior
The build should be successful on re-run.

Config files
If applicable, add config files to help explain your problem.

Environment

• Cloud provider: [all]
• OS: [e.g. Ubuntu 20.04.3 LTS]

epicli version: v1.3.0, develop

Additional context
Add any other context about the problem here.

---

DoD checklist

• Changelog updated (if affected version was released)
• COMPONENTS.md updated / doesn't need to be updated
• Automated tests passed (QA pipelines)
  • apply
  • upgrade
• Case covered by automated test (if possible)
• Idempotency tested
• Documentation updated / doesn't need to be updated
• All conversations in PR resolved
• Backport tasks created / doesn't need to be backported

[to-bar]

Analysis:

# mv /etc/apt/sources.list /etc/apt/sources.list.bak
# systemctl start apt-daily.service  # triggered by apt-daily.timer
# ls -s /etc/apt/sources.list
0 /etc/apt/sources.list

[przemyslavic]

✔️ Tested multiple times apply under various conditions and now it works fine.