[BUG] :distribution:docker:buildDockerImage failures downloading from http://vault.centos.org

[andrross]

Gradle task :distribution:docker:buildDockerImage fails occasionally with a timeout from http://vault.centos.org

| Errors during downloading metadata for repository 'baseos':
|   - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8/BaseOS/x86_64/os/repodata/repomd.xml [Connection timed out after 30001 milliseconds]
|   - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8/BaseOS/x86_64/os/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
|   - Curl error (28): Timeout was reached for https://vault.centos.org/centos/8/BaseOS/x86_64/os/repodata/repomd.xml [Operation timed out after 30000 milliseconds with 0 out of 0 bytes received]
| Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Example: #2147 (comment)

[reta]

@andrross I think the best we can do is switch to another distribution (question - which one? amazonlinux:2? ubuntu? debian?), @dblock do you have thoughts? thanks!

[andrross]

My questions: Is that CentOS endpoint really that flakey? Are we getting throttled or something? Is something else misconfigured on our side?

[reta]

CentOS is dead, for better or worst, taking into account we use vault for ~1 week only and it was down already - it looks flaky at this moment. The problem was affecting everyone (not only us) - all my local builds were failing, same for others.

[reta]

@andrross down again: #2244 (comment)

[CEHENKLE]

Hmmm...if we're using amazonlinux for our distribution, it makes sense to me change it to that. @bbarani any concerns if we make that switch? (also curious if engineering effectiveness would make the switch or the core team, but he who is less important that what)

[peterzhuamazon]

In my experience centos is that flaky on endpoint.
Previously I just manually force it to use official mirror instead of other vault.

We cannot change to another distro at this point due to dependencies from knn.

[owaiskazi19]

Can we use vault.epel.cloud for more reliability here as a mirror? Switching to a new distribution is a better option though.

[bbarani]

In my experience centos is that flaky on endpoint. Previously I just manually force it to use official mirror instead of other vault.

We cannot change to another distro at this point due to dependencies from knn.

More info on plugin dependencies with CENTOS: opensearch-project/opensearch-build#635

FYI.. KNN dependency might not be an issue starting 2.0 as they are not going to be built in the distribution (rather built as standalone plugin) opensearch-project/opensearch-build#1670

[gaiksaya]

In my experience centos is that flaky on endpoint. Previously I just manually force it to use official mirror instead of other vault.

We cannot change to another distro at this point due to dependencies from knn.

If this is related only to OpenSearch tests (example gradle check) is switching distribution still an issue? (Not the distribution just the min)

[reta]

If this is related only to OpenSearch tests (example gradle check) is switching distribution still an issue? (Not the distribution just the min)

@gaiksaya it is related to OpenSearch Docker distribution (and consequently, tests).

[owaiskazi19]

Closing this issue for now as the mirror url has been changed. Let's reopen if we face anymore downtime issue and have a plan to change the distribution.