Build native JNI library with a consolidated CMake toolchain (cross-platform) #232
Comments
|
Any movement on this? |
|
@irbull there seems to be demand for this PR ... what is your word on this ? My goal is that we can get a clear picture of which usecases the build-system needs to support |
|
I have spent a good portion of last week looking at cross platform builds, and if you have something that works, then a big +1 from me!!! Are you just building J2V8, or are you building V8 / node as well. What platform does the build itself run on? How many platforms have you managed to get working? (I was unable to get Node to build for Mac when the build was running on Linux). |
|
In my fork I'm just looking into the native part (V8 / Node) since CMake is primarily designed to build cross platform C/C++ projects. Did you look into something similar for the Java side ? or is a true cross platform build already possible for the Java code ?
I'm uncertain what you mean by that, do you mean that you cross-compile Node on a linux host but targeting a Mac binary output ? The distinction between a cross-platform build toolchain and a cross-compiling toolchain makes a big difference in the complexity of the setup. I saw in the repository some Dockerfiles, are those part of the attempts that you mentioned ? This would mean that we want to also go for cross-compiling ... but I think as long as we can get a working Docker image for each of the supported build + target platform combinations, then the rest will again just be regular CMake scripts (i.e. easy enough). Virtualizing MacOS on Linux using docker does not seem to be supported though, so we might have to go for a full VM virtualization with something like Vagrant. PS: what are the usecases for Maven + Gradle build scripts ? is one of those deprecated or are both still in use ? PSS: which OS are you using to develop / compile J2V8 ? (I'm primarily on Win 10) |
|
The Java side is easy -- compiled Java files can run on all platforms. The scripts are a mess because we have a few different requirements ( The harder part is the cross-compiling. In my case, I was (still am) trying to cross-compile node for Mac from a linux machine. The concept was to encode the steps in a Dockerfile, and run it on Docker. From what I understand, the host will always be linux -- but please correct me if I'm wrong there (I actually hope I'm wrong). What do you mean by:
What does a BTW, Thanks for taking the time and following up on this. I was getting pretty frustrated last week -- and was ready to give up. But you have given me some new hope :) |
The benefit that you get from a cross-platform build toolchain (like CMake) alone is that you can support all the native IDEs & CLI build toolchains that are supported by such a tool ... see here for CMake's supported target toolsets So imho the wins when using CMake over a custom cross-platform build script solution are:
Regarding the MacOS build on a Linux host ... yes, most of the time Docker will run a Linux container, Microsoft has been working on support for Windows Server in Docker containers but I have not tried those out yet 😄 Which docker image did you use for your MacOS build ? Was it one off the Dockerhub or a custom one ?
It's a pleasure ... I really like the idea of this project, I'm a big fan of JavaScript but I also see why Java is needed in enterprises ... so I can imagine many interesting usecases for J2V8 ... also I'm working on a little side project to get production-ready ES6 class interop running with J2V8. |
|
Alright, I'm convinced 😄. To split things out (and hopefully make this repository easier to navigate), I have just created a new project called Static Node Build. This will just include the scripts to build node.js. This is currently a collection of Dockerfiles and I'm hoping that we can run make from the root of this and get the static libraries for all the supported platforms, but that might be wishing for a bit too much. If we could at least run it on the current platform and produce the artifacts for that platform, that would be great. As for the Mac build, yes I tired |
|
Great, I will also look if I can get a Node.js MacOS cross-compile working in Docker ... once we have that figured out I could continue the work on transitioning the build-scripts to a consolidated CMake setup. |
|
I have pushed the |
|
I just finished the trip down the rabbit hole and ended up in the same place as you @irbull 😆 I'm not yet sure if both of those variables should be set to |
|
Alternatively adding |
|
Great job @drywolf! I've made a few changes to the With those changes, plus adding your changes above (both OS mac, host_os mac and adding -DOS=mac to the It appears that --start-group / --end-group is not available on Mac (or in our case Mac toolchains). I'm not sure why it's trying the Linux linking step here. |
|
Also, I set |
|
According to this stackoverflow post, If I remove those from the generated Makefile like so... I'm now stuck at the following linker problem: |
|
I don't think trying to tweak the generated makefiles will be successful. We need to understand why it's creating Linux specific makefiles when using the MacOSX toolchain. I'm hoping there is another parameter (like the OS=mac) that we're just missing. I've updated our progress on nodejs/node#12556 (comment) in case anyone has ideas. Thanks! |
|
I made some further progress looking into what the build currently produces (the error above). This lets the build continue further, but then I seem to run into this issue |
|
One alternative would be using Vagrant as said. I already built node successfully using this Vagrant box, which comes as a download of a 13 GB image though, so not as lightweight as a Docker setup, but at least this one would work right now. |
|
@drywolf I agree. We've put a valiant effort into this, and a docker based solution would be nice, but it might not be worth it. Furthermore, we will likely go through all this again when we try and target the next version of node. Here is what I think we should do.
With this approach someone can either download our giant (it will be big) tarball with the prebuild platforms, or they can run the full node build themselves. I am currently producing the tarball for Android right now. I haven't tried Vagrant before, but it seems like a good time to learn. |
|
Hi @irbull I see that you are already busy, converting parts of the build / publish workflows to a new structure 👍 Here's a short summary on what I have been up to:
To improve on the second point I started converting the scripts to python scripts instead, while also putting in some simple code structures to make the build-system primitives & configs a little bit reusable and more easy to maintain. Based on your docker = DockerBuildSystem()
vagrant = VagrantBuildSystem()
docker_builds = [
#------------------------------------------------------------------------------------------
BuildConfig("android", "x86",
[
MountPoint("$PWD/node.out", "/build/node")
],
"android-gcc-toolchain x86 --api 15 --host gcc-lpthread -C sh -c \"cd jni && ndk-build\""),
#------------------------------------------------------------------------------------------
BuildConfig("linux", "x64",
[
MountPoint("$PWD/node.out", "/build/node"),
MountPoint("$PWD", "/build/."),
]),
#------------------------------------------------------------------------------------------
]
vagrant_builds = [
#------------------------------------------------------------------------------------------
BuildConfig("macos", "x64"),
#------------------------------------------------------------------------------------------
]
docker.build(docker_builds)
vagrant.build(vagrant_builds)PS: I'm thinking about moving the declaration of the Docker filesystem mounts also into the Dockerfile, similar as it is done in Vagrant, which would make the above syntax even more concise. This is already working and runs both the docker and vagrant builds successfully. Before I continue to put more work into it though I would like to hear from you if you would be accepting of using Please let me know of what you think and if I should continue, I would then soon submit a PR with the python equivalents of the current shell scripts + the MacOS build additions. Once we have that, I would then expand the cross-platform support of the cmake scripts that I already wrote for Windows/Visual Studio for the other platforms. |
|
I've continued work on the architecture & code for the build-system that I outline above. The capabilities have already expanded quite a bit, so far the proof-of-concept is capable of supporting the following features / usecases: (MacOS used as example platform, works for the others too)
import build_platform as b
b.execute_build(b.target_macos, b.arch_x86, b.build_all, node_enabled = True, cross_compile = True)
b.execute_build(b.target_macos, b.arch_x64, b.build_all, node_enabled = True, cross_compile = True)@irbull Please let me know if this still looks good to you, I'd not like to put so much effort into this without knowing if I'm headed in the right direction 😉 |
|
To expand on the example of using the API in scripts, this is what it would look like to build just Node.JS for all platforms: # build Node.js only
def build_njs(target, arch):
b.execute_build(target, arch, [b.build_node_js], node_enabled = True, cross_compile = True)
build_njs(b.target_android, b.arch_arm)
build_njs(b.target_android, b.arch_x86)
build_njs(b.target_linux, b.arch_x86)
build_njs(b.target_linux, b.arch_x64)
# needs reboot here to turn Hyper-V off if Host-OS is Windows
build_njs(b.target_macos, b.arch_x86)
build_njs(b.target_macos, b.arch_x64)
# needs reboot here to switch to Windows-Containers
build_njs(b.target_win32, b.arch_x86)
build_njs(b.target_win32, b.arch_x64) |
|
The MacOS build & cross-compile is now working (almost) Before that I even got many more errors, because the maven surefire plugin was trying to run the java unit tests in parallel which I could fix by adding ... ... in the pom.xml, but the errors in the screenshot above are still happening. Also I just pushed all my code in its current state to my J2V8 fork |
|
I figured it out, I was missing the I will clean up the scripts a little bit more tomorrow, and by then the integration for MacOS is pretty much done. |
|
Just tried out my existing windows CMake build and with some small additions for node 7.4.0 it still does produce the shared lib for j2v8. But I seem to be running into the same problem as I was with the linker on MacOS stripping out some symbols from node.lib which results in the Node.js tests failing on windows too...
@TonyRice from what I saw you were already able to build J2V8 for windows after the 7.4.0 update, did you need to make any special changes to the linker options ? Did you build with node-support enabled ? |
|
@drywolf I'm actually currently building with VS2015 and node version v7.10.0. If you checkout the 7.10.0 tag and checkout vcbuild.bat from master you should be able to run I am currently attempting to update the project to the latest versions. My local fork is a little different than the original but I hope to get things working. I am going to try out your cmake builds and see what happens from there. |
|
I found the cause of the problem. Some of the builtin node-module registrations were missing in |
|
I'm currently setting up the android cross-compile ... @irbull do you run the junit tests in some way on android to make sure the jar + native lib are working ? |
|
I'm back for the weekly(ish) progress update. I currently have Android builds working for the two target architectures (armv7a & x86), including the bundling as an AAR package. I have also just now got android instrumentation tests set up to run the existing unit-test suite of J2V8 to see if everything is working as expected with the android build artifacts. And the results look quite promising...
Next up is the linux build configuration, and with that I'm then heading towards the end of the roadmap/todos for what I had planned for this build-system integration. So a PR should be coming up pretty soon 😉 |
|
Sorry, I'm not ignoring you. I just returned from vacation. I am catching up on issues / emails right now. I will look at this tonight. Regarding the unit tests. No, I don't run them "on device". It would be cool to do this, but I don't currently have a setup to make this work. |
|
Some notes: 1. Regarding build scripts vs Python: I completely agree with you. The build scripts I added are pretty simple, and I only added them as control files so I could get a build to run. I think a platform independent build script is a much better idea. I don't think there is any real logic in these files, but I could be mistaken. 2. Selective build stages: Awesome! This is pretty important, especially the part about not building node each time. I was hitting build time limits on Travis. I think this is exactly the right direction. 3. Your on device tests: Some of those (RuntimeNotLoaded and testGetOSLinux) I'm not really worried about. The other ones are interesting though. If you put your work in a PR, I can try it tomorrow and look at the tests. |
No worries, welcome back 😄 |
|
I am closing this since the main focus of this issue is achieved by the merging of #280 |
Hello @drywolf |
|
As mentioned above I only built node using VS 2015 ... using VS 2017 might need adjustments to get working ... you should look into the node.js build documentation if this version is even supposed to compile with VS 2017. TBH I haven't used J2V8 or built node.js from source for over a year now, so others might be more in the loop on the latest details of how to build node with VS. |
|
Thanks a lot. VS 2015 JAR is available? I can try to use it for one of my solution. could you please share it. |
@irbull I started replacing the static Visual Studio project files with CMake scripts in my fork.
This is the groundwork to consolidate all build scripts & steps into a single build script toolchain eventually.
Currently I wrote & tested the scripts only for Windows builds (since that's what I'm using J2V8 on), but I would continue work on support for the other platforms if you would be interested in a PR ?
The text was updated successfully, but these errors were encountered: