nemu: Enable static nemu for ci #1595

ganeshmaharaj · 2019-05-18T06:20:19Z

Build NEMU statically for every CI run to make sure we test a setup
similar to the one we are shipping with Kata.

Fixes: #1536
Signed-off-by: Ganesh Maharaj Mahalingam [email protected]

virtio-fs is now available in 1.7 release and needs hugepages enabled. NEMU's machine_type `virt` has issues getting network access when hugepages is enabled. Temporarily changing type to `pc` to fix this issue. Accompanying changes to runtime and tests are underway. Fixes: kata-containers#1709 Depends-on: kata-containers/tests#1595 Depends-on: kata-containers/packaging#522 Signed-off-by: Ganesh Maharaj Mahalingam <[email protected]>

virtio-fs is now available in 1.7 release and needs hugepages enabled. NEMU's machine_type `virt` has issues getting network access when hugepages is enabled. Temporarily changing type to `pc` to fix this issue. Accompanying changes to runtime and tests are underway. Fixes: kata-containers#1709 Depends-on: github.com/kata-containers/tests#1595 Depends-on: github.com/kata-containers/packaging#522 Signed-off-by: Ganesh Maharaj Mahalingam <[email protected]>

.ci/install_nemu.sh

jodh-intel · 2019-05-20T07:50:35Z

.ci/install_nemu.sh

 	case "$arch" in
 	x86_64)
-		local nemu_bin="qemu-system-${arch}_virt"
-		;;
-	aarch64)


Why is aarch64 being dropped? Even if we don't provide the static binaries today, NEMU works on arm64 too so can't we leave it as we should endeavor to add test support for that arch at some future point too?

We can add aarch64 here back, but it does come with some limitations. 1. It is not statically built, so we will need to add more scaffolding to the install scripts to get all the dependent libraries for it to work. It will also only work with xenial and probably fail on other distros. In the current form, it is not really consumable. As stage 2, we can start shipping aarch64 static binaries also along with x86_64 and then re-enable it. I can file a bug for that, if that approach would work.

@Pennyzct @justin-he Is this something that you could help with? We really would like to see support for building nemu on arm.
@ganeshmaharaj Can you open a separate issue for this, and list out what issues you are running into there?

.ci/install_runtime.sh

virtio-fs is now available in 1.7 release and needs hugepages enabled. NEMU's machine_type `virt` has issues getting network access when hugepages is enabled. Temporarily changing type to `pc` to fix this issue. Accompanying changes to runtime and tests are underway. Fixes: kata-containers#1709 Depends-on: github.com/kata-containers/tests#1595 Depends-on: github.com/kata-containers/packaging#522 Signed-off-by: Ganesh Maharaj Mahalingam <[email protected]>

Pennyzct · 2019-05-21T01:45:57Z

Hi~ @jongwu we definitely want to lend a hand here. We will firstly try to build static binary on our local machine and make it work. :)

amshinde · 2019-05-21T17:44:53Z

Thanks @Pennyzct !

jongwu · 2019-05-22T01:13:04Z

@Pennyzct ok, I will take it.

virtio-fs is now available in 1.7 release and needs hugepages enabled. Updating version of NEMU that ships with kata by default which contains the fixes for hugepages, machine_type=virt and network access. Fixes: kata-containers#1709 Depends-on: github.com/kata-containers/tests#1595 Depends-on: github.com/kata-containers/packaging#522 Signed-off-by: Ganesh Maharaj Mahalingam <[email protected]>

jongwu · 2019-05-22T09:49:54Z

@ganeshmaharaj @amshinde @jodh-intel @Pennyzct now I have enabled nemu on arm64 after some modification work. what's next?

jodh-intel · 2019-05-22T10:02:43Z

@jongwu - I think the next step is for @ganeshmaharaj to undo the aarch64 removal changes on this PR.

ganeshmaharaj · 2019-05-22T17:25:20Z

@jongwu is there a way we can enable the static build of arm64 similar to this one? https://github.com/kata-containers/packaging/pull/522/files#diff-62470642d2b1b882e092c3185db7f88b that would help us generate both the sets of binaries for CI and ship the same with kata for all future releases.

You could use this issue to make the follow on changes? #1620

chavafg · 2019-05-22T18:01:27Z

I think we can re-add arm once the static build for that arch is ready.

jongwu · 2019-05-23T09:59:56Z

@ganeshmaharaj after disable avx2 and modify the source code, nemu can be built as the way in https://github.com/kata-containers/packaging/pull/522/files#diff-62470642d2b1b882e092c3185db7f88b .
@jodh-intel @chavafg how to transfer my patch to nemu? as we can't touch nemu project for the legal issue

virtio-fs is now available in 1.7 release and needs hugepages enabled. Updating version of NEMU that ships with kata by default which contains the fixes for hugepages, machine_type=virt and network access. Fixes: kata-containers#1709 Depends-on: github.com/kata-containers/tests#1595 Depends-on: github.com/kata-containers/packaging#522 Signed-off-by: Ganesh Maharaj Mahalingam <[email protected]>

ganeshmaharaj · 2019-05-23T14:31:55Z

@jongwu if we can build NEMU for arm64, with the current scripts, can you kindly expand the build_nemu_static.sh script to make builds for both x86 and arm64 and make them both part of the giant tarball? we don't need to modify NEMU sources, just our way of configuring them to build. ohh just out of curiosity, are you able to cross compile it on an x86 machine?

ganeshmaharaj · 2019-05-23T21:14:36Z

/test

Build NEMU statically for every CI run to make sure we test a setup similar to the one we are shipping with Kata. Fixes: kata-containers#1536 Depends-on: github.com/kata-containers/packaging#522 Signed-off-by: Ganesh Maharaj Mahalingam <[email protected]>

ganeshmaharaj · 2019-05-23T22:31:16Z

/test-nemu

ganeshmaharaj · 2019-05-24T03:42:19Z

Not sure if i should trigger all the CI given this only touches NEMU. WIll refrain from it unless folks feel otherwise.

chavafg · 2019-05-24T12:45:58Z

lets run them as ubuntu job is required.
/test

ganeshmaharaj · 2019-05-24T17:29:06Z

@chavafg i am guessing the failed jobs are all expected and we can proceed merging this change?

chavafg · 2019-05-24T17:33:34Z

yeah, but we need another ack to be able to merge.
@kata-containers/tests, ptal

jongwu · 2019-05-27T07:36:18Z

@ganeshmaharaj for now, I can't build nemu on arm64 only modify build_nemu_static.sh. hid_dev should be enabled which must be done when make, as CONFIG_HID_DEV can not be configured by configure script. so I need change Dockerfile.
All these work are on arm64 machine, so no cross compile on x86.

ganeshmaharaj · 2019-05-28T15:20:03Z

@jongwu Aah. I believe it would be easy if we can enable cross compile within a container, this way we can do the build both for CI and for releases on any backend machine. Is there a way we can help to get this enabled from your side?

jongwu · 2019-05-29T11:07:34Z

@ganeshmaharaj I try to cross compile nemu on x86, but building cross compile tool chain blocks me. I find it not easy for me to get all dependency library in. I am appreciate of you that you can involved in and help me to do that.
but another concern is I can't build nemu on arm64 machine by just modify build_nemu_static.sh, I need change Dockerfile, which maybe not make sense from your part.

jongwu · 2019-05-31T11:29:52Z

@ganeshmaharaj I have successfully cross compiled nemu for arm64 on x86 in container.

virtio-fs is now available in 1.7 release and needs hugepages enabled. Updating version of NEMU that ships with kata by default which contains the fixes for hugepages, machine_type=virt and network access. Fixes: kata-containers#1709 Depends-on: github.com/kata-containers/tests#1595 Depends-on: github.com/kata-containers/packaging#522 Signed-off-by: Ganesh Maharaj Mahalingam <[email protected]> (cherry picked from commit 722ac5a) Signed-off-by: Ganesh Maharaj Mahalingam <[email protected]>

ganeshmaharaj force-pushed the nemu-ci branch 2 times, most recently from da2e4ea to e9a4cd8 Compare May 18, 2019 23:57

jodh-intel reviewed May 20, 2019

View reviewed changes

ganeshmaharaj force-pushed the nemu-ci branch 2 times, most recently from bc7bdb2 to 8ae23b7 Compare May 20, 2019 21:30

ganeshmaharaj force-pushed the nemu-ci branch 2 times, most recently from 642a7de to 9d683c6 Compare May 22, 2019 06:48

ganeshmaharaj force-pushed the nemu-ci branch from 9d683c6 to 8af0c6b Compare May 23, 2019 14:17

chavafg approved these changes May 23, 2019

View reviewed changes

ganeshmaharaj force-pushed the nemu-ci branch from 8af0c6b to ab6a9a3 Compare May 23, 2019 22:30

jcvenegas approved these changes May 24, 2019

View reviewed changes

chavafg merged commit 34fdb3e into kata-containers:master May 24, 2019

ganeshmaharaj mentioned this pull request May 28, 2019

ci: Add virtio-fs support #1537

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nemu: Enable static nemu for ci #1595

nemu: Enable static nemu for ci #1595

ganeshmaharaj commented May 18, 2019

jodh-intel May 20, 2019

ganeshmaharaj May 20, 2019

amshinde May 20, 2019

ganeshmaharaj May 22, 2019

Pennyzct commented May 21, 2019

amshinde commented May 21, 2019

jongwu commented May 22, 2019

jongwu commented May 22, 2019

jodh-intel commented May 22, 2019

ganeshmaharaj commented May 22, 2019 •

edited

Loading

chavafg commented May 22, 2019

jongwu commented May 23, 2019

ganeshmaharaj commented May 23, 2019

ganeshmaharaj commented May 23, 2019

ganeshmaharaj commented May 23, 2019

ganeshmaharaj commented May 24, 2019

chavafg commented May 24, 2019

ganeshmaharaj commented May 24, 2019

chavafg commented May 24, 2019

jongwu commented May 27, 2019

ganeshmaharaj commented May 28, 2019

jongwu commented May 29, 2019

jongwu commented May 31, 2019

nemu: Enable static nemu for ci #1595

nemu: Enable static nemu for ci #1595

Conversation

ganeshmaharaj commented May 18, 2019

jodh-intel May 20, 2019

Choose a reason for hiding this comment

ganeshmaharaj May 20, 2019

Choose a reason for hiding this comment

amshinde May 20, 2019

Choose a reason for hiding this comment

ganeshmaharaj May 22, 2019

Choose a reason for hiding this comment

Pennyzct commented May 21, 2019

amshinde commented May 21, 2019

jongwu commented May 22, 2019

jongwu commented May 22, 2019

jodh-intel commented May 22, 2019

ganeshmaharaj commented May 22, 2019 • edited Loading

chavafg commented May 22, 2019

jongwu commented May 23, 2019

ganeshmaharaj commented May 23, 2019

ganeshmaharaj commented May 23, 2019

ganeshmaharaj commented May 23, 2019

ganeshmaharaj commented May 24, 2019

chavafg commented May 24, 2019

ganeshmaharaj commented May 24, 2019

chavafg commented May 24, 2019

jongwu commented May 27, 2019

ganeshmaharaj commented May 28, 2019

jongwu commented May 29, 2019

jongwu commented May 31, 2019

ganeshmaharaj commented May 22, 2019 •

edited

Loading