Update rhcos to 42.80.20190827.1

[stbenjam]

We'll need this when openshift/installer#2264 lands.

Eventually we can avoid this if and when openshift/installer#2092 lands, otherwise there's not a convenient way to get the data out of the installer.

[hardys]

I've been testing this locally, and the bootstrap VM doesn't come up correctly - still trying to figure out why, but it's also been replicated by @karmab and a few other folks - reverting openshift/installer to 0454021 appears to resolve the issue so I suspect some badness in the new qemu image

[metal3ci]

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1099/

[stbenjam]

On the latest rhcos the code that keeps retrying waiting for etcd cluster to come up isn't working...

Aug 27 18:22:46 localhost bootkube.sh[1669]: https://etcd-0.ostest.test.metalkube.org:2379 is unhealthy: f
ailed to connect: dial tcp: lookup etcd-0.ostest.test.metalkube.org on 192.168.111.1:53: no such host
Aug 27 18:22:46 localhost bootkube.sh[1669]: Error: unhealthy cluster
Aug 27 18:22:46 localhost bootkube.sh[1669]: etcd cluster up. Killing etcd certificate signer...

This should retry:

https://github.com/openshift/installer/blob/master/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template#L272-L288

but it doesn't.

[stbenjam]

I can verify it's exiting 0, so we never retry:

# bootkube_podman_run \
>                 --rm \
>                 --name etcdctl \
>                 --env ETCDCTL_API=3 \
>                 --volume /opt/openshift/tls:/opt/openshift/tls:ro,z \
>                 --entrypoint etcdctl \
>                 6f3f175b53ca \
>                 --dial-timeout=5s \
>                 --cacert=/opt/openshift/tls/etcd-ca-bundle.crt \
>                 --cert=/opt/openshift/tls/etcd-client.crt \
>                 --key=/opt/openshift/tls/etcd-client.key \
>                 --endpoints=https://nonsense.example:2379 \
>                 endpoint health
https://nonsense.example:2379 is unhealthy: failed to connect: dial tcp: lookup nonsense.example on 127.0.0.1:53: no such host
Error: unhealthy cluster
[root@localhost openshift]# echo $?
0

Here's the full debug logs:

https://gist.github.com/stbenjam/24bf10326664ddee968df407c716e20b

podman version:

[root@localhost openshift]# podman --version
podman version 1.4.2-stable1

I don't notice any changes to etcd, it's last commit is from July.

[stbenjam]

Interactively in the container, it does return 1:

[root@localhost /]# etcdctl --dial-timeout=5s --cacert=/opt/openshift/tls/etcd-ca-bundle.crt --cert=/opt/openshift/tls/etcd-client.crt --key=/opt/openshift/tls/etcd-client.key --endpoints=https://nonsense.example:2379  endpoint health
https://nonsense.example:2379 is unhealthy: failed to connect: dial tcp: lookup nonsense.example on 127.0.0.1:53: no such host
Error: unhealthy cluster
[root@localhost /]# echo $?
1

[stbenjam]

Summary:

• podman in 42.80.20190823.0 is busted, we need https://bugzilla.redhat.com/show_bug.cgi?id=1741157 backported
• Backport fixes for v1.4.2 branch containers/podman#3895 will backport the required changes to podman, which will eventually end up in RHCOS
• In the meanwhile, the older, working version of podman is going to be pinned, and we'll get a new RHCOS build shortly. Once we have that we'll update openshift/installer + this PR.

[stbenjam]

42.80.20190827.1 is out with the fixes, got a good local install.

openshift/installer#2277 bumps the installer rhcos.

[metal3ci]

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1103/

[metal3ci]

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1104/

[metal3ci]

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1105/

[hardys]

Ok with openshift/installer#2277 and rebased to include #775 this is working for me locally 👍

@stbenjam thanks for all the effort yesterday debugging the podman issues! :)

[metal3ci]

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1107/

[metal3ci]

Build SUCCESS, see build http://10.8.144.11:8080/job/dev-tools/1108/

[hardys]

Proven locally and CI so should be good to go when the installer PR lands and we remove the temporary stuff here to test it 👍

[stbenjam]

Great thanks! openshift PR landed. We should get a nightly with that fix in about an hour.

[hardys]

Looks like the nightly picked up the new image so I removed the changes to run the installer from source and force-pushed - I then saw #778 which looks good too, we can discuss later which approach you'd prefer to go with.

[metal3ci]

Build SUCCESS, see build http://10.8.144.11:8080/job/dev-tools/1111/

[russellb]

Since CI passed here, let’s merge this. #778 can go in on top of this later