Unable to compile binary on linux with Julia 1.5.3 #2
Comments
subes
changed the title
|
I guess it will also fail with julia 1.6 and 1.7, though I did not try. |
|
Thanks for the notice. The mentioned issue is easy to fix. See the branch https://github.com/rssdev10/julia4j/tree/julia1.7_java17 But there are some other issues with Java 17, Julia 1.7. Long time I didn't do update. Looks like For now the native library might be assembled, but unfortunately I cannot check it on home Linux due to pretty old environment. There are issues with visibility of libraries and proper versions. Other issues I see, my main OS is MacOS. Since last time there were changes in java 17 installation path, Julia 1.7 registered paths, and OS Big Sur is with security restrictions and changes in dynamic libraries loading. So, I can build the jar, but not able to configure the environment to provide visibility of dynamic libraries yet. Please check from your side if you can. |
|
Hi, thanks a lot for the modifications. I can now compile it. Though it seems the linker uses wrong paths: Why is the "../bin/.." part in the path? Maybe manually creating this folder makes things work, but this will not be portable. I guess static linking is also not a solution because this will embed the julia installation? Though maybe we could still try this as long as the .so does not become too large because of this? |
|
Nope, creating the folder then makes another error pop up about a different path which can not be workarounded: |
|
Adding the jvm parameter " -Djava.library.path=/usr/lib/x86_64-linux-gnu/julia" also does not help: So the wrong path seems to be hardcoded into the libjulia4j.so as a wrong path. |
|
Here the testcase I am using: https://github.com/invesdwin/invesdwin-context-julia/blob/main/invesdwin-context-julia-parent/invesdwin-context-julia-runtime-julia4j/src/test/java/de/invesdwin/context/julia/runtime/juliacaller/Julia4JJNITest.java If you want to test yourself, installing Ubuntu in VirtualBox could make it testable on linux for you. Otherwise I will try again if you have something testable. |
|
We need to check https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling I want to update my local OpenSUSE which I previously used. Now I'm also not able to update cmake due to breaking dependencies and not able to check the build process properly. But the OS update will take some time. VM is good when there is enough space on an internal laptop's drive... |
|
I looked at the CMake scripts a bit more, I guess the JULIA_HOME part might be detected wrong: The cmake script thinks this is a julia installation which was installed from the website, though this installation comes from the apt/deb package manager and follows normal linux directory layouts where everything lies in a different place. I also tried using: set(CMAKE_INSTALL_RPATH "@loader_path") |
|
Ok, it seems not to be a problem of cmake, instead Julia itself does not handle embedding well when running from debian packaged installations: JuliaInterop/JuliaCall#99 I get the following errors: I guess Julia4j does not work with julia 1.5.3? Or it is again something debian specific. |
|
Ok, finally got it working by installing julia 1.7.1 from the website and using the following test: |
|
CMAKE Julia might be changed - https://github.com/rssdev10/julia4j/blob/julia1.7_java17/swig/FindJULIA.cmake#L2 In terms of variables, there is a check key The way with Btw, check it with And, I updated my local linux. Will try to reanimate it from my side at least for linux. |
|
Ok, thanks for the info. I will work on the binding now since I am able to test. When you have a new version available that works without the System.load workaround, I will upgrade. Thanks! |
|
Also can you please remove the System.out from the NativeUtils? |
Do it locally without a commit. I will remove it after check. |
|
I made some progress: https://github.com/invesdwin/invesdwin-context-julia/blob/main/invesdwin-context-julia-parent/invesdwin-context-julia-runtime-julia4j/src/main/java/de/invesdwin/context/julia/runtime/julia4j/internal/JuliaEngineWrapper.java
Here the output of the crash: |
|
Ok, the author of libjulia-clj documented a workaround to chain signal processing with julia: https://cnuernber.github.io/libjulia-clj/signals.html It seems Julia4j requires binding to a specific java thread similar to JEP for python. Though JEP supports multiple threads/contexts while Julia4j only allows single threaded access to Julia. Julia itself it able to execute commands multithreaded though while python only allows one thread per context. The workaround I implemented is to call Julia4j always from the same thread. This makes all testcases green: https://github.com/invesdwin/invesdwin-context-julia/blob/main/invesdwin-context-julia-parent/invesdwin-context-julia-runtime-julia4j/src/test/java/de/invesdwin/context/julia/runtime/julia4j/Julia4jScriptTaskRunnerJuliaTest.java It would still be great if the error handling could be improved. |
I tried to find a way how to force Java to do self search of dependencies. I changed a way to collect set of paths into I can run the test with |
|
For getting String values back from Julia, looks like we need to have an appropriate low level function to unpack. As an additional function from SWIG. See existing code with a wrapper SWIGEXPORT jstring JNICALL Java_org_julia_jni_swig_Julia4JJNI_jl_1get_1default_1sysimg_1path(JNIEnv *jenv, jclass jcls) {
jstring jresult = 0 ;
char *result = 0 ;
(void)jenv;
(void)jcls;
result = (char *)jl_get_default_sysimg_path();
if (result) jresult = (*jenv)->NewStringUTF(jenv, (const char *)result);
return jresult;
}But in case of getting a value we have a unsafe pointer but not the String. engine.put("x", "hello");
// print global variable "x"
engine.eval("res = join(fill(x, 10), \";\");");
Object result = engine.get("res");
SWIGTYPE_p_void bytes = Julia4J.jl_unbox_voidpointer((SWIGTYPE_p_jl_value_t) result);That I didn't check under debugger yet. |
|
Extracting something from an unsafe pointer does not work. It crashes the JVM. I tried it with an UnsafeBuffer: |
|
Check that branch please - https://github.com/rssdev10/julia4j/tree/jl_string_wrapper I added a simple wrapper: And added appropriate test for that - https://github.com/rssdev10/julia4j/blob/jl_string_wrapper/src/test/java/Julia4JScriptingTest.java#L89 But not able to activate due to issues with |
|
I guess with "activate" you mean not compiling into the libjulia4j.so. I guess that is the reason why I get the following error: |
|
That should be there. |
|
Fixed that part and pushed the commit. The function body was absent despite export table. But that was not a proper way to get the result. jl_unbox_voidpointer cannot process string value. |
|
Would be great if we can fix the string handling in julia4j. I did some performance tests. Currently libjulia-clj is the fastest integration: Though I also created an integration for libpython-clj to compare that against a JNI integration with JEP. libpython-clj is a lot slower than JEP:
By that rationale: when the strings are transported via memory instead of files, julia4j (also JNI) could outperform libjulia-clj by a lot. |
|
I found a workaround for the error handling by using functions that wrap the executions by using try/catch and redirect_stderr. Reusing the functions also makes it rather fast because the evaluation wrappers don't have to be recompiled all the time by Julia. Thus we don't need additonal exception handling functions in julia4j. When it becomes possible to return strings in memory, it will be simple to redirect the stderr/catch output into a string to be returned as an error response. That is also the only thing missing that I currently see. With the current state the performance has improved a lot (due to the precompiled functions): |
|
Hi, I checked the code and found solution how to get variables of any type. And also, added unpack into the For now the code looks like: ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("julia");
engine.put("x", "hello");
// print global variable "x"
engine.eval("res = join(fill(x, 10), \";\");");
SWIGTYPE_p_jl_value_t result = (SWIGTYPE_p_jl_value_t)engine.eval("res");
Julia4J.jl_show(result);
String str = Julia4J.jl_unbox_string((SWIGTYPE_p_jl_value_t) result);
assertEquals(String.join(";", Collections.nCopies(10, "hello")), str);And, some wrapper around The branch is still https://github.com/rssdev10/julia4j/tree/jl_string_wrapper |
|
Unwrapping the string response works great: subes/archive_invesdwin-context-julia@0a227bf It does not seem to make a difference in performance after all: This is the performance with disabled stdout: This is the performance of libjulia-clj:
Though it is still great that we were able to get rid of the output file. I think we can close this issue. |
See screenshot:
I would like to add a native julia integration to: https://github.com/invesdwin/invesdwin-context-julia/
The text was updated successfully, but these errors were encountered: