CI: Setup Jenkins CI on IBM Power Systems

[nitkon]

We would like to get an IBM Power 8 system added to the current Jenkins CI setup. We have a Ubuntu 16.04.5 LTS bare metal server with 10cores @ 3.6Ghz and 60GB memory with a public IP. Nested virtualization is not supported.

[jodh-intel]

/cc @chavafg, @grahamwhaley.

[grahamwhaley]

Cool @nitkon .
I'm presuming this is to be added as a Jenkins slave node. That should be relatively easy.
First steps are to create a jenkins user on the system, set them up an SSH key and provide the details to us (@chavafg and myself to start with, and we will then lodge the details with the OSF/arch committee as well). We can then add an appropriate slave node to the CI master, give it the correct labels so we know it is an IBM Power and Ubuntu 16.04. Then we set up a 'job' in Jenkins (we normally use the proxy repo as it is the quietest one), and fire a build to see what happens.

My only concern is going to be around running on a bare metal 'deployed-once' machine. There is always the possibility that a PR somehow corrupts the machine, and makes builds unstable. This is more of a concern for the metrics CI than the QA CI, but it can still cause stability issues. Some time ago I documented some of that here There are a number of potential mitigations:

• some cleanup scripts were added when the ARM node was added. If you go bare metal then you'll want to tie those into your arch as well. Example: https://github.com/kata-containers/tests/tree/master/.ci/x86_64
• You can hack it so Jenkins launches the build/run inside a VM. I did that for the metrics for a while, but it was too 'noisy' for the metrics results. It should be OK for a QA CI though. This would depend on your machine being able to support nested VMs.
• Similarly (and I've not tried this), if nested VMs are supported, you could probably theoretically set up a nodepool on the machine and have that serve up one-shot instances to use.

We can always start with the bare metal and cleanup scripts and go from there...

[sboeuf]

@nitkon This is great news :)

@grahamwhaley I agree with your concern of running things directly on the baremetal machine. The easiest way to solve this would be to define this machine as Jenkins worker that would spawn one or more VMs as you suggested.

[grahamwhaley]

Sadly, @nitkon informs me that the hardware does not support nested VMs, so we probably can't do that.
It if comes to it, we could ultimately use the skanky fix I have in place for the metrics bare metal right now, which is to have a chained dependent jenkins job that tries to reboot the machine after each build.... which helps 'clean out cruft', but is still not a 100% guarantee.

[sboeuf]

well it's better than letting the machine run, but I'm worried we won't have stable tests because the machine could have been compromised by any PR...

[grahamwhaley]

And update - the slave node is created and 'connected' (http://jenkins.katacontainers.io/computer/Power8_slave01/log).
Now we need to make a test job (proxy repo is our normal target), and test a build run.
Let me create that, and we'll see how it goes..

[grahamwhaley]

I fired a build over on kata-containers/proxy#97
Results look like you need to set up the passwordless sudo for the jenkins user on the slave @nitkon

sudo chgrp -R $(id -ng) ${WORKSPACE}
[kata-containers-proxy-Power8-16.04-PR] $ /bin/bash /tmp/jenkins8052303251752720049.sh
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified

I've also just fixed job so you can now see the fail status reported over on that test build.

[nitkon]

@grahamwhaley: Thank you. Fixed that issue. I installed docker-ce and libostree-dev for ppc64le as well on the P8 jenkins slave after looking at the failure logs.

However, now looking at the logs, I see kernel build failures and also realize two things.

1. We cannot have clearlinux based rootfs as clearlinux would not work on Power. I guess arm CI is using rootfs image based on Fedora.
2. As of today on Power systems, we can only have initrd based images since nvdimm support in qemu on Power is yet to get upstream. So we will have to base our CI on initrd image based on Fedora only.

[grahamwhaley]

Right @nitkon progress :-)
I believe what you need to do is create and PR an arch specific CI setup (and teardown) dir with the relevant scripts/config in it just like the aarch64 has at https://github.com/kata-containers/tests/tree/master/.ci/aarch64
And in there you can config what the CI builds and installs and tests against :-)
And, that will include adding some info to the runtime versions.yaml file for the arch, similar to: https://github.com/kata-containers/runtime/blob/master/versions.yaml#L123

[grahamwhaley]

Hey @nitkon - Power CI systems are up and happy now, yes? Can we close this Issue?

[nitkon]

Sure.