Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Numpy support w/ python3crystax #882

Closed
wwoods opened this issue Sep 9, 2016 · 19 comments · Fixed by #1343
Closed

Numpy support w/ python3crystax #882

wwoods opened this issue Sep 9, 2016 · 19 comments · Fixed by #1343

Comments

@wwoods
Copy link

wwoods commented Sep 9, 2016

I'm trying to get my app to compile with Python 3 using the Crystax NDK. Everything is set up, and if I try to push my app to my device without specifying that numpy is needed, everything works fine. However, numpy raises a few issues:

  1. (easily fixed) The recipe needs to be modified such that python3crystax is supported. Currently, it only claims python2 support, but that's overridden easily enough (depends = ['python2'] -> depends = [('python3crystax', 'python2)] in the __init__.py).
  2. The compiler claims it can't find -lcrystax:
...
don't know how to compile Fortran code on platform 'posix'
C compiler: /usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-pointer --sysroot /home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/platforms/android-19/arch-arm -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DANDROID -mandroid -fomit-frame-pointer --sysroot /home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/platforms/android-19/arch-arm -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -fPIC

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/home/wwoods/miniconda3/envs/kivy/include/python3.5m -c'
ccache: _configtest.c
/usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-pointer --sysroot /home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/platforms/android-19/arch-arm _configtest.o -o _configtest
/home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.3/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrystax
collect2: error: ld returned 1 exit status
/home/wwoods/walt/dev/kivy/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.3/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrystax
collect2: error: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o

Everything except numpy works fine, so I think that it's just Numpy's recipe that needs a minor alteration. Oddly though, specifying a LIBRARY_PATH or LDFLAGS does not have any effect. In fact, if I add a get_recipe_env to the NumpyRecipe that prints out what super().get_recipe_env(arch) contains, it has LDFLAGS with the appropriate path on there. Running the linker by hand with the given flags find the library fine (ld $(LDFLAGS) -lcrystax --verbose). In the context of python-for-android though, the linker always fails to find libcrystax.a.

Any help appreciated, thanks.
@wwoods
Copy link
Author

wwoods commented Sep 9, 2016

PS - I'm using kivy1.9.2dev and buildozer android_new as the frontend to python-for-android.

@inclement
Copy link
Member

Sorry for the delay replying, I did think about this but I'm not sure offhand what could be wrong. I'll try to check it out when I get back around to some of the python3 things.

@JohnBuckleyGitHub
Copy link

I am guessing that since this issue is still open, the Numpy recipe will not support python3crystax easily?

@inclement
Copy link
Member

inclement commented Feb 25, 2017 via email

@renefritze
Copy link

Isn't it also an issue that the Crystax NDK currently doesn't ship a Fortran compiler for arm-linux-androideabi? Other than that it should be possible to set something like

setup_extra_args = ['-L', 'CRYSTAX/sources/crystax/libs/armeabi-v7a']

in the numpy recipe to pass additional flags to its build_ext.

@inclement
Copy link
Member

@renemilk Fortunately numpy doesn't require any fortran compiling. This is a barrier for scipy though.

@renefritze
Copy link

I tried passing the extra libdir via the mentioned build_ext flag, but it still does not show up in the failing linker command. Maybe putting a site.cfg in the extracted numpy sources can help, but I'm not hopeful. The nuclear option would be substituting the linker with a wrapper script that forcibly adds the flag unconditionally.

@inclement
Copy link
Member

@renemilk You're trying to make the numpy build link properly with libcrystax.so? It may also be plausible to try to remove this linking as I don't think libcrystax should be necessary. I've tried both, also without success.

@renefritze
Copy link

Yes, I do.
I have no idea idea how, why or when libcrystax gets injected into the linker command there, but yeah I also think removing it would do.

@frmdstryr
Copy link

frmdstryr commented Nov 10, 2017 •
edited
Loading

Edit, I got it working for crystax 2.7 here https://github.com/frmdstryr/p4a-numpy . Have not tested on 3.5 but I believe it should update everything to the correct versions.

@inclement
Copy link
Member

Wow, great! I was going to say that your post was very helpful, but getting it all working is even better :p .

Also, I've tried crystax python 2.7 before, but hit some issues with it (I don't remember what). Have you got that working with p4a too?

@frmdstryr
Copy link

frmdstryr commented Nov 10, 2017 •
edited
Loading

Yes python2crystax works fine (been using it for 6+ months now). For some reason it is mentioned that Py_SetPath is unsupported in https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/sdl2/build/jni/src/start.c#L131 however Py_SetPath works fine with the crystax version (however the python2 version does NOT have Py_SetPath!).

Also have a python2crystax recipe with openssl for that here https://github.com/frmdstryr/p4a-crystax (based on a PR someone did a while ago)

Edit: Changes to p4a might be required for it though I have my own fork

@Abysselene
Copy link

Hi,
I'm trying to make numpy work with py3crystax too (internship in a lab and they need numpy on android with kivy and python3) so I'll work on it for some time.

If I get interesting results I'll communicate them here, also if you don't have time to do some tests but have ideas about what to do, share them here.
If I have enough time I'll try your suggestions and post the results.

@Abysselene
Copy link

Abysselene commented Apr 16, 2018 •
edited
Loading

Sharing my discoveries, or at least the ones I think as important:
After editing the recipe for numpy with depends python3crystax only I get the same error as there : #1074
I can set [('python3crystax','python3')] or [('python3crystax','python2')] or ['python3crystax'] I get the same error. (Beware of the order! Setting crystax after python2/3 gives /usr/bin/false command not found)

The problem usually starts with :

blas_mkl_info:
  libraries  not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

openblas_info:
  libraries  not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries  not found in ['/usr/local/lib', '/usr/lib/atlas-base', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
  NOT AVAILABLE
etc.....

If I'm not mistaking the thread says the error has been resolved at that time by editing numpy recipe. There were no flags that's why numpy gave this error and since I have the same error with python3 I guess the libraries are not correctly set.
To be sure I've made another project to test it: python2 kivy numpy, ndk and sdk already downloaded (also tested by setting nothing, buildozer downloads the good one) and it works fine.

So the only difference I have with the working project is python3, more exactly python3crystax in the ndk crystax 10.3.2.
The libraries added in the recipe refer to this with arch arm:

flags = " -L{ctx.ndk_dir}/platforms/android-{api_ver}/arch-{platform}/usr/lib/" \
                " --sysroot={ctx.ndk_dir}/platforms/android-{api_ver}/arch-{platform}" \

I've compared the content between crystax ndk and google r9c ndk.
Some files are missing or added in android-19/arch-arm/usr/lib but that's not what surprised me. Every lib****.a in the crystax version are empty, only 8 bytes!
With an hex editor we can see there is only !<arch>.
I tried to copy paste the content of usr from the r9c ndk (hey don't hit me, I wanted to try) but no changes I get the same error even after a buildozer android clean.

I tried to edit the flags following those instructions : https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-Android

        if flags not in env['CC']:
            env['CC'] += flags + ' NOFORTRAN=1 -mhard-float -D_NDK_MATH_NO_SOFTFP=1'
        if flags not in env['LD']:
            env['LD'] += flags + '-Wl,--no-warn-mismatch -lm_hard' +' -shared'

but I think I did something wrong and I don't have enough time today to continue I'll see this later.

@Abysselene
Copy link

Other important point:
I followed instructions to ignore the blas lapack atlas inclusion, which is possible but we loose a lot of speed with algebra computation (see https://docs.scipy.org/doc/numpy-1.9.2/user/install.html and https://stackoverflow.com/questions/32228967/compile-numpy-without-intel-mkl-blas-atlas-lapack). It should work but I still get the same last error and I think it's this one that halts everything:

C compiler: /usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-pointer -D__ANDROID_API__=19 -I/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -isysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -I/media/axel/reste/buildozer/.buildozer/android/platform/build/build/python-installs/medmanip/include/python3.5 -L/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm/usr/lib/ --sysroot=/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -DANDROID -mandroid -fomit-frame-pointer -D__ANDROID_API__=19 -I/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -isysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -I/media/axel/reste/buildozer/.buildozer/android/platform/build/build/python-installs/medmanip/include/python3.5 -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -mthumb -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python3.5m -c'
ccache: _configtest.c
/usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-pointer -D__ANDROID_API__=19 -I/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -isysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -I/media/axel/reste/buildozer/.buildozer/android/platform/build/build/python-installs/medmanip/include/python3.5 -L/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm/usr/lib/ --sysroot=/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm _configtest.o -o _configtest
/media/axel/reste/.buildozer/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.3/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrystax
collect2: error: ld returned 1 exit status
/media/axel/reste/.buildozer/crystax-ndk-10.3.2/toolchains/arm-linux-androideabi-5/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/5.3/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrystax
collect2: error: ld returned 1 exit status
failure.

lcrystax is missing!
In #1074 it was resolved by adding the libraries (seems like it works with python2crystax) but with the exact same path now it gives the same error they had before. So I keep thinking there is a problem with python3crystax, I would prefer there isn't because I tried to pick the content or android-19/arch-arm in there latest version (b1052) and the .a libs are still empty...

N.B: The warnings saying lapack, blas and atlas are missing don't fail the compilation. The failure is due to crystax missing.

I resolved this error by editing the flags in numpy recipe adding -L{ctx.ndk_dir}/sources/crystax/libs/{arch.arch} the the compilation started to work and again failed with an error over python.h.
After searching for some time I found that it was trying to compile my python3.5 installed in my computer rather than crystax's one.
I don't know why it did that but I edited the flags once more and it resolved this problem -I{ctx.ndk_dir}/sources/python/{py_ver}/include/python/.
By the way, after looking at the flags there https://github.com/frmdstryr/p4a-numpy/blob/master/p4a_numpy/__init__.py (it is what was used to edit the broken recipe of numpy for the py2.7 version last January or December), I saw that it had the correct flags to avoid this so I copy pasted all their flags.

Now the new problem is when building the extensions:

ccache: numpy/core/src/dummymodule.c
[.........]
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/numpy
creating build/lib.linux-x86_64-3.5/numpy/core

It's trying to build a library but I get this error:
........ crystax-ndk-10.3.2/platforms/android-19/arch-arm/usr/lib/../lib/crtbegin_so.o: Relocations in generic ELF (EM: 40)
After searching on internet I found that it's either due to using the wrong crtbegin, by wrong I mean one that has not been compiled with the same compilator, this one is in arch-arm so I hope it's good, or it could be because the compilator doesn't know how to use it and it's resolved with flags (good, until know everything has been resolved with flags).

Now I'm stuck, I don't know what to do because CC and CXX already have what I intended to put : /usr/bin/ccache arm-linux-androideabi-gcc (and g++ respectively)
Unfortunately I don't have anymore time to work on this but I hope my explanations will save a lot of time to the next person trying to resolve this problem.

Here the last error giving me a headache, I'm not used to compilation problems but isn't it strange that it uses linux x86_64 to compile for arm? Until now the commands visible in the console started with C compiler: /usr/bin/ccache arm-linux-androideabi-gcc -DANDROID -mandroid -fomit-frame-poi

error: Command "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro --sysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -lm -L/media/axel/reste/buildozer/.buildozer/android/platform/build/build/libs_collections/medmanip/armeabi-v7a -L/media/axel/reste/.buildozer/crystax-ndk-10.3.2/sources/crystax/libs/armeabi-v7a -lcrystax -DANDROID -mandroid -fomit-frame-pointer -D__ANDROID_API__=19 -I/media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -isysroot /media/axel/reste/.buildozer/crystax-ndk-10.3.2/platforms/android-19/arch-arm -I/media/axel/reste/buildozer/.buildozer/android/platform/build/build/python-installs/medmanip/include/python3.5 -march=armv7-a -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/numpy/core/src/dummymodule.o -Lbuild/temp.linux-x86_64-3.5 -o build/lib.linux-x86_64-3.5/numpy/core/_dummy.cpython-35m-x86_64-linux-gnu.so" failed with exit status 1

@AndreMiras AndreMiras mentioned this issue Jun 25, 2018
Closed
@lehanesa
Copy link

Did anyone have any luck with this?

@krinnewitz
Copy link
Contributor

Hi guys, I think I figured it out!

Here is the fixed recipe with a minimal example application: https://github.com/plapadoo/p4a-numpy

This isn't well-tested. So I will collect more feedback before creating a PR. Please let me know if the fixed recipe works for you.

Thanks to all the people who already put some effort into this in the past. Especially @Abysselene and @frmdstryr

@JohnBuckleyGitHub
Copy link

JohnBuckleyGitHub commented Aug 8, 2018 via email

@AndreMiras AndreMiras mentioned this issue Aug 9, 2018
Closed
@krinnewitz krinnewitz mentioned this issue Aug 28, 2018
Merged
@AndreMiras
Copy link
Member

@johnbolia @wwoods @lehanesa @Abysselene and others the pull request is ready there #1343 🎉
Can you give it a try to speed up merging process?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Extend numpy recipe to support python3 plapadoo/python-for-android
9 participants
@AndreMiras @renefritze @wwoods @frmdstryr @inclement @krinnewitz @JohnBuckleyGitHub @Abysselene @lehanesa