Infrastructure Flake on Jenkins when building openshift/node image: [Errno -1] repomd.xml does not match metalink for epel #8571

stevekuznetsov · 2016-04-20T13:31:50Z

When building the openshift/node docker image:

--- openshift/node ---
Sending build context to Docker daemon 18.94 kB
Sending build context to Docker daemon 18.94 kB

Step 1 : FROM openshift/origin
 ---> 6e39b07a70f9
Step 2 : MAINTAINER Devan Goodwin <[email protected]>
 ---> Running in 73d875dde0c7
 ---> 4837e7ea6aff
Removing intermediate container 73d875dde0c7
Step 3 : ADD https://copr.fedoraproject.org/coprs/maxamillion/origin-next/repo/epel-7/maxamillion-origin-next-epel-7.repo /etc/yum.repos.d/

 ---> 14cab38052e9
Removing intermediate container 9eb5cda1db50
Step 4 : RUN INSTALL_PKGS="libmnl libnetfilter_conntrack openvswitch     libnfnetlink iptables iproute bridge-utils procps-ng ethtool socat openssl     binutils xz kmod-libs kmod sysvinit-tools device-mapper-libs dbus     ceph-common iscsi-initiator-utils" &&     yum install -y $INSTALL_PKGS &&     rpm -V $INSTALL_PKGS &&     yum clean all
 ---> Running in 5dfb4fa924e4
Loaded plugins: fastestmirror, ovl
http://mirror.cc.columbia.edu/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
...
...


 One of the configured repositories failed (Extra Packages for Enterprise Linux 7 - x86_64),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable epel

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=epel.skip_if_unavailable=true

failure: repodata/repomd.xml from epel: [Errno 256] No more mirrors to try.

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/208/console

The text was updated successfully, but these errors were encountered:

cgwalters · 2016-05-09T15:23:58Z

See also https://lists.centos.org/pipermail/ci-users/2016-April/thread.html

stevekuznetsov · 2016-06-20T20:33:00Z

This has gotten worse ...

http://s3-mirror-us-east-1.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.es.its.nyu.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://download-i2.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.cc.columbia.edu/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.cs.pitt.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://lug.mtu.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.oss.ou.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://ftp.osuosl.org/pub/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://linux.mirrors.es.net/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://kdeforge2.unl.edu/mirrors/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.cs.princeton.edu/pub/mirrors/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.unl.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirrors.rit.edu/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirror.umd.edu/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://archive.linux.duke.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.math.princeton.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.sfo12.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://fedora-epel.mirror.lstn.net/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirrors.kernel.org/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.datto.com/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirrors.xmission.com/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.cogentco.com/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirrors.syringanetworks.net/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.nexcess.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.metrocast.net/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://reflector.westga.edu/repos/Fedora-EPEL/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.symnds.com/distributions/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirrors.tummy.com/pub/fedora.redhat.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://epel.wallawalla.edu/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://mirror.prgmr.com/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://muug.ca/mirror/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
http://fedora.westmancom.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
https://mirror.cpsc.ucalgary.ca/mirror/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.


 One of the configured repositories failed (Extra Packages for Enterprise Linux 7 - x86_64),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable epel

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=epel.skip_if_unavailable=true

failure: repodata/repomd.xml from epel: [Errno 256] No more mirrors to try.
http://s3-mirror-us-east-1.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.es.its.nyu.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://download-i2.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.cc.columbia.edu/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.cs.pitt.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://lug.mtu.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.oss.ou.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://ftp.osuosl.org/pub/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://linux.mirrors.es.net/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://kdeforge2.unl.edu/mirrors/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.cs.princeton.edu/pub/mirrors/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.unl.edu/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirrors.rit.edu/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirror.umd.edu/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://archive.linux.duke.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.math.princeton.edu/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.sfo12.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://fedora-epel.mirror.lstn.net/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirrors.kernel.org/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.datto.com/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirrors.xmission.com/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.cogentco.com/pub/linux/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirrors.syringanetworks.net/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.nexcess.net/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.metrocast.net/fedora/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://reflector.westga.edu/repos/Fedora-EPEL/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.symnds.com/distributions/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirrors.tummy.com/pub/fedora.redhat.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://epel.wallawalla.edu/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://mirror.prgmr.com/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://muug.ca/mirror/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
http://fedora.westmancom.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
https://mirror.cpsc.ucalgary.ca/mirror/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel

@tdawson did you say this was just EPEL mirrors being out of sync with the source? Can we mitigate failures like this in CI?

stevekuznetsov · 2016-06-21T00:20:35Z

Upgrading this since we're seeing quite a lot of it.

@danmcp @tdawson is there any possible way for us to latch onto a stable EPEL mirror from RH so our CI doesn't suffer needlessly? Can we cache the RPMs we're actually using somewhere?

danmcp · 2016-06-21T01:15:25Z

Mirrors are certainly possible. @tdawson Do you have any particular recommendations or advice on the best approach?

sdodson · 2016-06-21T17:30:06Z

@stevekuznetsov Can you try adding a yum clean all prior to attempting to use the repos?

cgwalters · 2016-06-21T17:32:32Z

It has nothing to do with the yum caches - due to the way the repo is managed updates will always be racy. It would be a lot less racy if 2 copies of the RPM data were cached and the RPMs were sync'd before the repodata.

tdawson · 2016-06-21T21:21:36Z

I'd say use openshifts mirrors, they have epel mirrored and are only updated once a day, so you won't get a weird update halfway through your build.

Instead of installing epel-release, do the following

RUN wget -O /etc/yum.repos.d/epel7.repo https://mirror.openshift.com/mirror/epel/epel7.repo

stevekuznetsov · 2016-06-22T15:34:25Z

We can address this by switching to the mirror.openshfit.com EPEL mirror for AMI builds, etc, but we will need oc ex dockerbuild capability to inject secrets (sslclientcert/key) for the mirror into our Docker builds, where we see the majority of these flakes.

This will be blocked on @smarterclayton implementing that secrets feature.

tdawson · 2016-06-22T15:40:49Z

You do not need the certs/key for that area https://mirror.openshift.com/mirror/epel/

Since there is nothing private in that repo, you do not need a certificate to access it.

If you are getting failures when you are accessing them, please let me know.

stevekuznetsov · 2016-06-22T17:08:07Z

@tdawson ah, incredible. I assumed I would need them due to some other repos on the domain needing them.

@smarterclayton since our internal mirror doesn't need the secrets, can you see anything wrong with simply disabling the upstream EPEL repo in the RHEL base image that I think we use for all of our builds?

soltysh · 2017-05-02T15:50:24Z

Slightly different, but still this:

Failure talking to yum: failure: repodata/repomd.xml from centos-paas-sig-openshift-origin13-rpms: [Errno 256] No more mirrors to try.
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin13/repodata/repomd.xml: [Errno 14] curl#52 - \"Empty reply from server\""

in https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_install/1590/console

stevekuznetsov added priority/P3 area/tests kind/test-flake Categorizes issue or PR as related to test flakes. priority/P2 and removed priority/P3 labels Apr 20, 2016

stevekuznetsov mentioned this issue Apr 20, 2016

implemented glog wrapper for use in delegated commands #8024

Merged

0xmichalis mentioned this issue May 9, 2016

Generation and separate status updates for DCs #7149

Merged

This was referenced May 9, 2016

add Bash trap framework basics, refactored stack trace #8527

Merged

Fixing test flake for docker builds #8846

Merged

danmcp assigned stevekuznetsov May 12, 2016

stevekuznetsov added component/test-infrastructure area/infrastructure and removed component/test-infrastructure labels Jun 7, 2016

stevekuznetsov mentioned this issue Jun 20, 2016

Migrate utility scripts into `hack/lib' #9350

Merged

stevekuznetsov added priority/P1 and removed priority/P2 labels Jun 21, 2016

stevekuznetsov mentioned this issue Jun 21, 2016

Refactored test/cmd/authentication.sh to use proper strings and literals #9360

Merged

0xmichalis mentioned this issue Jun 22, 2016

Use shared caches in the deploymentconfig controller #9002

Merged

stevekuznetsov added the blocked label Jun 22, 2016

stevekuznetsov removed the blocked label Jun 22, 2016

fabianofranz mentioned this issue Mar 13, 2017

don't use base cmd name when printing client version #12781

Merged

sosiouxme mentioned this issue Apr 26, 2017

add preflight checks integration tests openshift/openshift-ansible#3816

Merged

soltysh mentioned this issue May 2, 2017

Revert "disable image-references" #13990

Merged

stevekuznetsov added priority/P1 and removed priority/P2 labels May 2, 2017

soltysh mentioned this issue May 17, 2017

Prefer secure connection during image pruning #14114

Merged

stevekuznetsov mentioned this issue Jul 5, 2017

TASK [Check latest available OpenShift RPM version] Could not parse metalink #15040

Closed

tbielawa mentioned this issue Jul 17, 2017

Fix the PROXY variables, once and for all openshift/openshift-ansible#4762

Merged

kwoodson mentioned this issue Aug 1, 2017

Normalize logging entry point to include std_include. openshift/openshift-ansible#4850

Merged

This was referenced Aug 7, 2017

reconcile cluster roles instead of overwriting #15654

Merged

start a command for running the template service broker #15655

Merged

miminar mentioned this issue Aug 8, 2017

Image pruner: Determine protocol just once #14914

Merged

This was referenced Aug 8, 2017

openshift_checks: refactor check results openshift/openshift-ansible#4913

Merged

openshift_checks: enable variable conversion openshift/openshift-ansible#4922

Merged

This was referenced Aug 9, 2017

Remove extensions.jobs #15581

Merged

Run the TSB in a separate pod #15672

Merged

This was referenced Aug 9, 2017

openshift-checks: have playbooks invoke std_include openshift/openshift-ansible#5026

Merged

openshift_checks: refactor find_ansible_mount openshift/openshift-ansible#4944

Merged

dmage mentioned this issue Aug 10, 2017

rebase to Docker Distribution v2.6.2 #15694

Merged

kwoodson mentioned this issue Aug 10, 2017

More complete discovery of entry point playbooks openshift/openshift-ansible#5028

Merged

juanvallejo mentioned this issue Aug 10, 2017

Enable CLI plugins in oc #15717

Merged

juanvallejo mentioned this issue Aug 11, 2017

Adding additonal checks upgrade path openshift/openshift-ansible#5014

Merged

kwoodson mentioned this issue Aug 15, 2017

adding check to a yaml dump to work properly with new ruamel lib openshift/openshift-ansible#5006

Merged

stevekuznetsov closed this as completed Aug 31, 2017

juanvallejo mentioned this issue Sep 13, 2017

Merge inventory-generator with origin-ansible image openshift/openshift-ansible#5043

Merged

coreydaley mentioned this issue Jan 4, 2018

Ability to specify default tolerations via the buildconfig defaulter #17955

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infrastructure Flake on Jenkins when building openshift/node image: [Errno -1] repomd.xml does not match metalink for epel #8571

Infrastructure Flake on Jenkins when building openshift/node image: [Errno -1] repomd.xml does not match metalink for epel #8571

stevekuznetsov commented Apr 20, 2016

cgwalters commented May 9, 2016

stevekuznetsov commented Jun 20, 2016

stevekuznetsov commented Jun 21, 2016

danmcp commented Jun 21, 2016

sdodson commented Jun 21, 2016

cgwalters commented Jun 21, 2016

tdawson commented Jun 21, 2016

stevekuznetsov commented Jun 22, 2016

tdawson commented Jun 22, 2016 •

edited

Loading

stevekuznetsov commented Jun 22, 2016

soltysh commented May 2, 2017

Infrastructure Flake on Jenkins when building openshift/node image: [Errno -1] repomd.xml does not match metalink for epel #8571

Infrastructure Flake on Jenkins when building openshift/node image: [Errno -1] repomd.xml does not match metalink for epel #8571

Comments

stevekuznetsov commented Apr 20, 2016

cgwalters commented May 9, 2016

stevekuznetsov commented Jun 20, 2016

stevekuznetsov commented Jun 21, 2016

danmcp commented Jun 21, 2016

sdodson commented Jun 21, 2016

cgwalters commented Jun 21, 2016

tdawson commented Jun 21, 2016

stevekuznetsov commented Jun 22, 2016

tdawson commented Jun 22, 2016 • edited Loading

stevekuznetsov commented Jun 22, 2016

soltysh commented May 2, 2017

tdawson commented Jun 22, 2016 •

edited

Loading