port the nabla to aarch64 #21

xiaobo55x · 2019-01-09T02:10:03Z

Hi,

Do you have the plan to support the Nabla on aarch64 platform?

Regards,

Haibo

lumjjb · 2019-01-09T13:52:43Z

I believe there is tangent where @ananos is looking into ARM amd64 support.

add: @djwillia @ricarkol

xiaobo55x · 2019-01-14T10:04:00Z

@lumjjb
Thanks!

@djwillia @ricarkol
Is it possible to replace the rumprun with the unikraft?

ricarkol · 2019-01-14T15:43:28Z

Hi @xiaobo55x , don't know about unikraft to be honest. But what we would actually like to try is to replace netbsd with Linux. Something like this: https://www.mail-archive.com/[email protected]/msg01773.html

xiaobo55x · 2019-01-15T01:36:34Z

@ricarkol
Thanks for your comments!
To be short, Unikraft is aimed to provide a "Unikernel Core" which is able to build unikernels targeted at specific applications without requiring the time-consuming, expert work that building such a unikernel requires today. You can refer to the link below for more info.
Unikraft

xiaobo55x · 2019-01-15T06:33:58Z

Hi @lumjjb @ricarkol,
We are investigating the possibility to port the Nabla to arm64 platform.

For the solo5 project, we already have added the support for arm64 in the master branch. Could you please let me know when will you upgrade the solo5 src in this repository to its master branch?

Thanks,

Haibo

ricarkol · 2019-01-16T16:44:51Z

We're planning to point to the latest solo5 src within the next weeks. Hopefully there will be no surprises.

xiaobo55x · 2019-01-17T01:17:53Z

@ricarkol That's great! Thanks!

ananos · 2019-01-24T22:13:31Z

Hi @lumjjb @ricarkol,
We are investigating the possibility to port the Nabla to arm64 platform.

Hi Haibo,

as @ricarkol and @lumjjb have mentioned, there is already an effort to port the nabla containers stack to aarch64.

Although we haven't ported the base-build Makefiles and build system on this repo to support aarch64, you can have a look at our approach at github.com/cloudkernels/rumprun. There's also a series of short, tutorial-like blog posts describing the basic steps to build the toolstack, runnc, and a basic unikernel for a rpi3 at https://blog.cloudkernels.net.

I suspect these changes could be merged upstream (at least for rumprun/src-netbsd) pretty soon (in the coming weeks).

cheers,
Tassos

xiaobo55x · 2019-01-25T03:43:09Z

@ananos
Great! Looking forward to seeing your work in the upstream.

xiaobo55x · 2019-03-06T09:47:53Z

Hi @ananos,

It seems that you are merging your work on aarch64 to upstream. Is it possible to run the Nabla
container on a arm64 machine now?

Regards,

Haibo

ananos · 2019-03-06T11:50:59Z

Hi @xiaobo55x,

yes! initial support is upstream, so you should be able to build the toolstack and run a Nabla container on aarch64.

If you find that useful, we have a post describing the steps: https://blog.cloudkernels.net/posts/build-a-nabla-docker-image

Let us know when you try it, please do ping if you have any issues.

thanks!

lumjjb · 2019-03-06T14:47:41Z

Hi @xiaobo55x you can run aarch with solo5-spt now, we are still upstreaming changes to runnc, this should be done by this week.

nabla-containers/runnc#66

xiaobo55x · 2019-03-07T01:29:04Z

@ananos @lumjjb , Great! Will give a try next week.

xiaobo55x · 2019-03-12T01:12:35Z

Hi @ananos @lumjjb , I have tried to make the nginx-base-build on our arm64 server, and the toolstack can be built successfully, but the nginx rumprun-package build failed. According to the build scripts, it seems that extra work should be done to port the application rumprun-package to arm64.

ananos · 2019-03-12T06:01:22Z

Hi @xiaobo55x,

I suspect that the part that fails is the hardcoded RUMPRUN_TOOLCHAIN_TUPLE=x86_64-rumprun-netbsd entry in rumprun-packages/config.mk.

If this is the case, can you try manually changing it to RUMPRUN_TOOLCHAIN_TUPLE=aarch64-rumprun-netbsd ?

After a succesfull build on nabla-base-build/rumprun-packages/nginx, issuing a make at nabla-base-build/nginx-base would do the trick I guess.

We are porting runnc to aarch64 as we speak. The next step would be to port nabla-base-build. ATM we have been able to manually build the packages.

have a go with the above and let us know!

xiaobo55x · 2019-03-12T06:23:59Z

Hi @ananos, The RUMPRUN_TOOLCHAIN_TUPLE have been changed to aarch64-rumprun-netbsd, and the error happened when building nginx. After checking the makefile in nginx, I found some architecture dependent codes which are not implemented on arm64 yet(for example, the libressl lib build). I have also tried to build node-base package, it also failed. So I guess extra works should be done to port each application on the arm64 platform.

ananos · 2019-03-12T06:51:53Z

well, I'm pretty sure we haven't changed anything in libressl or nginx for it to work.

Just tried it with the tip of rumprun (4ddcdd2) and rumprun-packages (687c6dab) and it seems that there's no x86_64 specific build commands.

For a quick workaround (until we sort out the build issues on nabla-base-build) you can just go to the rumprun-packages/nginx dir (make sure you've got the TUPLE correct), source rumprun/obj/config-PATH.sh and re-try the build. After it succeeds, go back to nginx-base and run make. That should do the trick for now.

xiaobo55x · 2019-03-12T07:16:00Z

The failure happened in the rumprun-packages/libressl/build/crypto/Makefile.in. There are some files suffixed with x86_64. I think there should be the corresponding files suffixed with arm64.

ananos · 2019-03-12T07:37:32Z

I'm clearly missing something. Can you please paste the error you're seeing -- I suspect these files are only built on x86_64.

From a clean rumprun-packages (687c6dab) repo libressl builds fine for me.

xiaobo55x · 2019-03-12T07:46:14Z

./build-rr.sh ran successfully
make[1]: Leaving directory '/home/xuhaibo/Nabla/nabla-base-build/rumprun'
touch rumprun_stamp
install -m 664 -D /home/xuhaibo/Nabla/nabla-base-build/rumprun-packages/config.mk.dist /home/xuhaibo/Nabla/nabla-base-build/rumprun-packages/config.mk
source /home/xuhaibo/Nabla/nabla-base-build/rumprun/obj/config && source /home/xuhaibo/Nabla/nabla-base-build/rumprun/obj/config-PATH.sh &&
make -C /home/xuhaibo/Nabla/nabla-base-build/rumprun-packages/nginx all bin/nginx.spt
make[1]: Entering directory '/home/xuhaibo/Nabla/nabla-base-build/rumprun-packages/nginx'
Makefile.debug:1: warning: overriding recipe for target 'all'
Makefile.bake:1: warning: ignoring old recipe for target 'all'
make -C ../libressl
make[2]: Entering directory '/home/xuhaibo/Nabla/nabla-base-build/rumprun-packages/libressl'
make -C build
make[3]: Entering directory '/home/xuhaibo/Nabla/nabla-base-build/rumprun-packages/libressl/build'
Making all in crypto
make[4]: Entering directory '/home/xuhaibo/Nabla/nabla-base-build/rumprun-packages/libressl/build/crypto'
CCAS aes/aes-elf-x86_64.lo
aes/aes-elf-x86_64.s: Assembler messages:
aes/aes-elf-x86_64.s:5: Error: unknown mnemonic xorl' -- xorl 0(%r15),%eax'
aes/aes-elf-x86_64.s:6: Error: unknown mnemonic xorl' -- xorl 4(%r15),%ebx'
aes/aes-elf-x86_64.s:7: Error: unknown mnemonic xorl' -- xorl 8(%r15),%ecx'
aes/aes-elf-x86_64.s:8: Error: unknown mnemonic xorl' -- xorl 12(%r15),%edx'
aes/aes-elf-x86_64.s:10: Error: unknown mnemonic movl' -- movl 240(%r15),%r13d'

xiaobo55x · 2019-03-12T07:48:19Z

root@bin:/home/xuhaibo/Nabla/nabla-base-build/rumprun-packages# git log
commit 687c6dab278ff3dba68b914e1ed0511eb5525551
Author: Ricardo Koller [email protected]
Date: Thu Feb 21 17:39:36 2019 -0500

python3: adding patch for disabling CAN

The new netbsd (after rumprun pointing into it) includes
some new functionality like PF_CAN sockets. Python is is trying to use
them but we can't quite support them at the moment. This patch makes
python think we don't support them, by undef'ing PF_CAN and AF_CAN.

commit cef2a59f328eb4233aa13b1a94520241f3af176c

ananos · 2019-03-12T08:01:42Z

as I suggested in my previous replies, please try to do a manual build.

Step 1:

nabla-base-build# cd rumprun-packages/
nabla-base-build/rumprun-packages# sed s/x86_64/aarch64/g -i config.mk
root@kmaxgw-arm64:/home/ananos/nabla10/nabla-base-build/rumprun-packages# cat config.mk
#
# rumprun-packages "configuration"
#

# Set to the name of the rumprun compiler toolchain you want to use for
# building packages. (eg. aarch64-rumprun-netbsd, i486-rumprun-netbsdelf).
RUMPRUN_TOOLCHAIN_TUPLE=aarch64-rumprun-netbsd

# Select ssl package (another option is openssl)
#RUMPRUN_SSL= libressl
nabla-base-build/rumprun-packages# cd libressl/
nabla-base-build/rumprun-packages/libressl# . ../../rumprun/obj/config-PATH.sh 
nabla-base-build/rumprun-packages/libressl# make clean
rm -rf build
nabla-base-build/rumprun-packages/libressl# make

See if that works for you -- config.mk is being overwritten by some part of the nabla-base-build Makefiles:
nabla-base-build/nginx-base/Makefile:38

rumprun-packages-nginx: rumprun FORCE
        install -m 664 -D $(RUMPPKGS)/config.mk.dist $(RUMPPKGS)/config.mk

if you still want to initiate make from the nginx-base dir, maybe you should try fixing config.mk.dist to include the correct TUPLE (aarch64 instead of x86_64).

xiaobo55x · 2019-03-12T09:43:16Z

Hi @ananos, The build is successful on our arm64 server with the above methods. Thanks very much for the help!

ananos · 2019-03-12T09:49:08Z

no worries, glad we could resolve this specific issue. We pinpointed the config.mk.dist thing ;-)

I guess TODOs for us is to bring runnc into a stable multi-arch state and then port the changes to nabla-build-base.

Bear in mind that for redis, python3 and probably for any package that uses 128-bit arithmetic operations, libgcc is required to be linked in, so we're updating rumprun's src-netbsd to include the NetBSD version, as well as some makefiles to verify the order of linking. We should have these patches merged-in end of this week or early next week.

xiaobo55x · 2019-03-12T09:58:34Z

Great! One more question, Is it possible to point the rumprun src-netbsd to the latest NETBSD codes which with better support for the aarch64 port.

ananos · 2019-03-12T10:22:26Z

Yeap, that's WiP.

At the moment, rumprun points to https://github.com/cloudkernels/src-netbsd/tree/a9854c10386f81212829f437743ce7ac2e58fad3 which contains patches up to 2018-10-29

We would like to automate the process but that's one more thing in our TODO list ;-)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

port the nabla to aarch64 #21

port the nabla to aarch64 #21

xiaobo55x commented Jan 9, 2019

lumjjb commented Jan 9, 2019

xiaobo55x commented Jan 14, 2019

ricarkol commented Jan 14, 2019

xiaobo55x commented Jan 15, 2019

xiaobo55x commented Jan 15, 2019

ricarkol commented Jan 16, 2019

xiaobo55x commented Jan 17, 2019

ananos commented Jan 24, 2019

xiaobo55x commented Jan 25, 2019

xiaobo55x commented Mar 6, 2019

ananos commented Mar 6, 2019

lumjjb commented Mar 6, 2019 •

edited

Loading

xiaobo55x commented Mar 7, 2019

xiaobo55x commented Mar 12, 2019

ananos commented Mar 12, 2019 •

edited

Loading

xiaobo55x commented Mar 12, 2019 •

edited

Loading

ananos commented Mar 12, 2019

xiaobo55x commented Mar 12, 2019 •

edited

Loading

ananos commented Mar 12, 2019

xiaobo55x commented Mar 12, 2019 •

edited

Loading

xiaobo55x commented Mar 12, 2019

ananos commented Mar 12, 2019

xiaobo55x commented Mar 12, 2019

ananos commented Mar 12, 2019

xiaobo55x commented Mar 12, 2019

ananos commented Mar 12, 2019

port the nabla to aarch64 #21

port the nabla to aarch64 #21

Comments

xiaobo55x commented Jan 9, 2019

lumjjb commented Jan 9, 2019

xiaobo55x commented Jan 14, 2019

ricarkol commented Jan 14, 2019

xiaobo55x commented Jan 15, 2019

xiaobo55x commented Jan 15, 2019

ricarkol commented Jan 16, 2019

xiaobo55x commented Jan 17, 2019

ananos commented Jan 24, 2019

xiaobo55x commented Jan 25, 2019

xiaobo55x commented Mar 6, 2019

ananos commented Mar 6, 2019

lumjjb commented Mar 6, 2019 • edited Loading

xiaobo55x commented Mar 7, 2019

xiaobo55x commented Mar 12, 2019

ananos commented Mar 12, 2019 • edited Loading

xiaobo55x commented Mar 12, 2019 • edited Loading

ananos commented Mar 12, 2019

xiaobo55x commented Mar 12, 2019 • edited Loading

ananos commented Mar 12, 2019

xiaobo55x commented Mar 12, 2019 • edited Loading

xiaobo55x commented Mar 12, 2019

ananos commented Mar 12, 2019

xiaobo55x commented Mar 12, 2019

ananos commented Mar 12, 2019

xiaobo55x commented Mar 12, 2019

ananos commented Mar 12, 2019

lumjjb commented Mar 6, 2019 •

edited

Loading

ananos commented Mar 12, 2019 •

edited

Loading

xiaobo55x commented Mar 12, 2019 •

edited

Loading

xiaobo55x commented Mar 12, 2019 •

edited

Loading

xiaobo55x commented Mar 12, 2019 •

edited

Loading