[FEATURE REQUEST] Add Internet connection test to download-requirements.sh

[to-bar]

Is your feature request related to a problem? Please describe.
Misleading error messages when download-requirements.sh is run on host without Internet access.

Describe the solution you'd like
Before running commands that require Internet access, test whether external URL is accessible.

Basic solution:
Test external (e.g. GitHub) URL with curl or wget.

Comprehensive solution:
Test Internet access for all tools (apt, yum, curl, wget, crane) since behind a proxy they can require dedicated configuration.

Describe alternatives you've considered
n/a

Additional context
n/a

---

DoD checklist

• Changelog updated
• COMPONENTS.md updated / doesn't need to be updated
• Feature has automated tests
• Automated tests passed (QA pipelines)
  • apply
  • upgrade
• Idempotency tested
• Documentation added / updated / doesn't need to be updated
• All conversations in PR resolved
• Solution meets requirements and is done according to design doc
• Usage compliant with license

[przemyslavic]

@sbbroot @to-bar

Network connectivity tests have been added for the following tools:

• apt
• yum
• crane
• curl
• wget

Tested automatically in the pipelines and locally in Ubuntu and CentOS containers and it works fine if it has internet access.

I also performed tests with no internet access in the container.
In the case of CentOS, the script immediately fails on testing the yum connection (which is desirable):

[root@fb0103547eca /]# /shared/build/prepare_scripts_centos/download-requirements.sh /shared/build/epicli-centos/
Detected arch: x86_64
Docker platform: linux/amd64
Process with pid: 77 not found
Removing file: /var/run/download-requirements.sh.pid
PID is: 142, creating file: /var/run/download-requirements.sh.pid
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"
Testing yum connection:
- base...
- extras...
- updates...
Connection failure, reason: [Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates&infra=container error was 14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error" One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid> or subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: updates/7/x86_64]

but when it comes to Ubuntu it fails trying to install prerequisities (wget, gpg, curl and tar packages) without testing the connection first:

root@9085b550b3ba:/# /shared/build/prepare_scripts_ubu/download-requirements.sh /shared/build/epicli-ubu/
Detected arch: x86_64
Docker platform: linux/amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ca-certificates libpsl5 libssl1.1 openssl publicsuffix
The following NEW packages will be installed:
  ca-certificates libpsl5 libssl1.1 openssl publicsuffix wget
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 2517 kB of archives.
After this operation, 6991 kB of additional disk space will be used.
Ign:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl1.1 amd64 1.1.1-1ubuntu2.1~18.04.13
Ign:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 openssl amd64 1.1.1-1ubuntu2.1~18.04.13
Ign:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ca-certificates all 20210119~18.04.1
Err:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpsl5 amd64 0.19.1-5build1
  Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 publicsuffix all 20180223.1310-1
  Temporary failure resolving 'archive.ubuntu.com'
Ign:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 wget amd64 1.19.4-1ubuntu2.2
Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 libssl1.1 amd64 1.1.1-1ubuntu2.1~18.04.13
  Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 openssl amd64 1.1.1-1ubuntu2.1~18.04.13
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 ca-certificates all 20210119~18.04.1
  Temporary failure resolving 'archive.ubuntu.com'
Err:6 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 wget amd64 1.19.4-1ubuntu2.2
  Temporary failure resolving 'archive.ubuntu.com'
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1-1ubuntu2.1~18.04.13_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.1-1ubuntu2.1~18.04.13_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/c/ca-certificates/ca-certificates_20210119~18.04.1_all.deb  Temporary failure resolving 'archive.ubuntu.com'
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libp/libpsl/libpsl5_0.19.1-5build1_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/publicsuffix/publicsuffix_20180223.1310-1_all.deb  Temporary failure resolving 'archive.ubuntu.com'
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/w/wget/wget_1.19.4-1ubuntu2.2_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Assuming these packages are already installed on the system, then the script fails on adding repositories (importing gpg keys), still not having the connection test executed:

root@b06c161d15a9:/#  /shared/build/prepare_scripts_ubu/download-requirements.sh /shared/build/epicli-ubu/
Detected arch: x86_64
Docker platform: linux/amd64
Dependency list: /shared/build/prepare_scripts_ubu/.dependencies
Command used to download packages: run_cmd_with_retries 3 apt-get download
Destination directory for packages: /shared/build/epicli-ubu/packages
gpg: no valid OpenPGP data found.

In my opinion, for Ubuntu these tests should be run earlier, or we should have some global test as the first step to check the internet connection before proceeding with the script.
To be discussed.

[przemyslavic]

✔️ Fixed in #2560. Added additional test before installing prerequisites.
Tested automatically in the pipelines and locally in Ubuntu and CentOS containers with and without internet access.

[plirglo]

Please create backport to 1.0, 1.1 tasks.