ARM: get the ARM CIs up

[grahamwhaley]

Provide details and help to get the ARM CI's up for kata.

The basic information for setting up Jenkins should be available in the https://github.com/kata-containers/ci/tree/master/jenkins folder, but there will no doubt be differences (such as the use of Azure vs packet.net vs bare metal servers and agents), and the information is somewhat coded into the Jenkins xml configs, so unless you are a Jenkins expert, that may not be the most helpful ;-)

We may wish/need/want to extract some of the core information from those xml files and write a short .md file describing the basic requirements and setup.

/cc @chavafg for knowledge of existing x86 Azure Jenkins setup
/cc @gnawux @Pennyzct @Weichen81 for input on the ARM side.

[jodh-intel]

/cc @nitkon wrt PPC64el (see kata-containers/runtime#286).

[Weichen81]

Thanks for opening such a thread!
This would be very helpful for us to set up CI : )

[jongwu]

@jodh-intel how to use those configure files under https://github.com/kata-containers/ci/tree/master/jenkins. I start a jenkins serve in my local machine and put all of those configure file into $JENKINS_HOME, after restart jenkins serve, a lot of errors occur. I am a jenkins user not so skillful with jenkins, I think that is not the way to configure jenkins, So what to do with all of those configure files to build kata-container ci system at a arm server?

[jodh-intel]

Hi @jongwu - we're in the process of documenting the setup.

Please see: #32 and feel free to comment as that PR is still being reviewed.

[jongwu]

@jodh-intel the document is helpful, I know what to do now! thank you.

[grahamwhaley]

:-) - I will be working on the feedback on that PR later today - one thing I think maybe is not made clear is that we are applying the scripts to 'freestyle' job/projects under Jenkins (as opposed to say Pipeline projects). That 'nugget' may help you locate the right dialog boxes ;-)
Oh, also, the other missing step in the document is 'install the necessary plugins'. I will have to try and define the list of the minimum required plugins (like the GHPRB plugin at least).

[vielmetti]

Happy to help from the packet.net side in any way that I can.

[vielmetti]

Related to #39 - it is possible to spin up a completely new machine at Packet through an API, and I believe that through Jenkins this can be done with the Jenkins Jclouds provider at https://github.com/apache/jclouds/tree/master/providers/packet . The spin-up cycle is about 10 minutes. That would assuredly get you a "clean environment"; however there might be faster ways to accomplish the same task.

[grahamwhaley]

We may just have to live with the 10 minute bringup time :-) For reference, our CI runs have two phases:

• build and install
• test

On my local machines I'm seeing the build and install phase take something in the order of 12m30s (some of it is package downloads, so it does vary according to network bandwidth and remote server availability). On the Azure QA CI's I'm seeing full run times on the 42m-1h8m scale.
So, we can probably live with the 10 minute startup time if that is the best option we have right now.

[vielmetti]

I am looking at http://jenkins.katacontainers.io/ @chavafg - the original issue for this task was to get the Arm CI up - but I don't see any of them.

Perhaps it was premature to close this issue re Arm just because the doc re Jenkins has been improved?

[chavafg]

Hi @vielmetti, sorry for that, I merged #32 and this issue got closed.
Reopening.

@grahamwhaley maybe #32 should've solved another issue?

[grahamwhaley]

Yeah, mea culpa, that should probably have had a sub-Issue rather than close this one... it's merged now, so I think we can just move on...

[vielmetti]

Hi @grahamwhaley @chavafg - has there been any change or progress on this? I notice a tracking issue for related stuff at #42 (Zuul CI) and am not sure that I understand how things are progressing on arm64 support.

[grahamwhaley]

Hi @vielmetti - ARM Jenkins stuff is being looked at by @Pennyzct over at kata-containers/tests#472 - is that what you are looking for?

[Pennyzct]

Hi~@vielmetti I'm working on ARM Jenkins from Arm, and it is still on going😊, few issues left to fixed.

[grahamwhaley]

Hey there ARM and CI folks (@Pennyzct @chavafg @jongwu @Weichen81 ) ... afaik, the ARM CI is up and running, but still only enabled for the proxy repo. We should discuss if it is stable enough to enable for the other repos?
Note, enabling the other repos (in particular the test and runtime repos), will result in the CI being run pretty much once an hour (on average) - if it is flaky it will likely be noticed very quickly, which is:

• good, as we get testing!
• add some pressure, as we will need a fairly quick response from the ARM CI maintainers if we see issues ... or after probably 48h we will disable it from the Jenkins jobs again :-)

So, discuss and let me know - are we happy to start enabling other repos for ARM CI?

/cc @alicefr @nitkon just for interest for IBM CIs that will soon get to a similar point...

[chavafg]

We actually have a job for the tests repo and it seems to be stable:
http://jenkins.katacontainers.io/job/kata-containers-tests-ARM-18.04-PR/

I agree we can enable the job for the other repos :)

[Pennyzct]

Thanks!!!!!! @grahamwhaley @chavafg that's what I always hope to do, bringing ARM CI running on more repositories. agent, osbuilder, runtime, step by step, (≧∀≦)ゞ.
I, and my folks, @jongwu @hejianet will definitely try our best to respond asap :) .

[grahamwhaley]

Great. @chavafg , you OK enabling for the runtime repo next the, as that is the next obvious candidate :-)

[chavafg]

sure, I've just enabled it, lets keep an eye on it and see how it behaves.
http://jenkins.katacontainers.io/job/kata-containers-runtime-ARM-18.04-PR/

[vielmetti]

I see this error repeatedly in the logs above:

12:10:20 go: disabling cache (/home/jenkins/.cache/go-build) due to initialization failure: open /home/jenkins/.cache/go-build/log.txt: permission denied

Perhaps the Jenkins setup needs to be tweaked?

[grahamwhaley]

Thanks @vielmetti - my best guess is that maybe when the jenkins user got set up it was tested using possibly a sudo shell, that might have created a local user dir that was then not accessible by the user.
@Pennyzct - can you log into the slave and check that out? I may only have a small effect on the performance (maybe it is slowing down golang code builds), but will be an annoyance in the logs ;-)

I should probably check that that is not a warning we get anyway in the other CIs... let me go peek...

[grahamwhaley]

Nope, had a search in a current x86 build (http://jenkins.katacontainers.io/job/kata-containers-runtime-ubuntu-16-04-PR/1443/consoleFull), and didn't find that line.

[Pennyzct]

@grahamwhaley thanks!! you're right. /home/jenkins/.cache/go-build was previously owned by root.
drwxr-xr-x 258 root root 4096 Jul 30 2018 go-build
and I have already changed its ownership to jenkins
drwxr-xr-x 258 jenkins jenkins 4096 Jul 30 2018 go-build.
let's see how it goes now. ;) @vielmetti

[chavafg]

I see some linter issues when running checks on the runtime repo:

13:18:09 cli/kata-check_arm64.go:129:9:warning: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary) (golint)
13:18:11 virtcontainers/qemu_arm64_test.go:43:23:warning: undeclared name: virt (unused)
13:18:12 cli/kata-check_test.go:141:113:warning: unused variable or constant undeclared name: TestDataa (varcheck)
13:18:12 virtcontainers/qemu_arm64_test.go:43:23:warning: unused variable or constant undeclared name: virt (varcheck)
13:18:12 virtcontainers/qemu_arm64_test.go:50:11:warning: unused variable or constant undeclared name: defaultMemSlots (varcheck)
13:18:12 virtcontainers/qemu_arm64_test.go:54:40:warning: unused variable or constant too few arguments in call to arm64.memoryTopology (varcheck)
13:18:12 virtcontainers/qemu_arm64_test.go:29:23:warning: unused variable or constant undeclared name: virt (varcheck)
13:18:12 virtcontainers/qemu_arm64_test.go:43:23:warning: unused struct field undeclared name: virt (structcheck)
13:18:12 virtcontainers/qemu_arm64_test.go:50:11:warning: unused struct field undeclared name: defaultMemSlots (structcheck)
13:18:12 virtcontainers/qemu_arm64_test.go:54:40:warning: unused struct field too few arguments in call to arm64.memoryTopology (structcheck)
13:18:13 virtcontainers/qemu_arm64_test.go:29:23:warning: unused struct field undeclared name: virt (structcheck)
13:18:13 cli/kata-check_test.go:141:113:warning: unused struct field undeclared name: TestDataa (structcheck)
13:18:13 cli/kata-check_test.go:141:113:warning: undeclared name: TestDataa (staticcheck)

From: http://jenkins.katacontainers.io/job/kata-containers-runtime-ARM-18.04-PR/52/console

@Pennyzct can you help us check those?
Thanks.

[Pennyzct]

Hi~ @chavafg sorry for the delay. I have pulled request #1202 under runtime/ repo to fix this issue. ptal. ;)

[GabyCT]

Closing this issue as we already have an ARM CI running in kata repositories