Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

apk not build using crystax NDK #992

Closed
sumitmadhwani opened this issue Jan 24, 2017 · 35 comments
Closed

apk not build using crystax NDK #992

sumitmadhwani opened this issue Jan 24, 2017 · 35 comments

Comments

@sumitmadhwani
Copy link

I am building apk using crystax NDK. I am following the same instructions as mentioned in python-for-android documentation.
I am getting following error for the line from kivy.app import App
logs
@sumitmadhwani sumitmadhwani mentioned this issue Jan 25, 2017
Open
@KarimReefat
Copy link

same issue but with other symbol PyDict_Next in the same module _clock.so
( ImportError: dlopen failed: cannot locate symbol "PyDict_Next" referenced by "_clock.so"... )
my_log.txt
buildozer.txt

@ngoonee ngoonee mentioned this issue Jan 27, 2017
Closed
@FeralBytes
Copy link
Contributor

Also encountering the same error; identical to @KarimReefat

@FeralBytes
Copy link
Contributor

I think it is actually caused by either buildozer or python-for-android; because it is using cython in a python 2.7 virtualenv and yet we are targeting python 3; thus it should be compiled in a python3 virtual environment.

@FeralBytes
Copy link
Contributor

Planning to modify P4A tonight to use either venv or virtualenv3 aka virtualenv but where
--python=python3.5
Offending line of code is Line Number 327: https://github.com/kivy/python-for-android/blob/master/pythonforandroid/build.py#L327
Posting the details in case some one wants to try before I get to it tomorrow.

@FeralBytes
Copy link
Contributor

Also of note Buildozer is also guilty of forcing python 2.7, so I wll probably have to modify it too Line Number 546: https://github.com/kivy/buildozer/blob/master/buildozer/__init__.py#L546
@inclement you stated you were having an issue too; and you referenced python 3.6, are you using Crystax to compile?

@inclement
Copy link
Member

I've been using only crystax for a while, without issues, so I'm not sure what's wrong. The python3 build is quite idiosyncratic though and needs cleaning up, so it's good to have references to ways it's failing for people.

The bit about using a python2.7 virtualenv for python3 builds needs to be fixed, but actually shouldn't in generally stop the build working, as only pure python modules are built this way and most are py2/py3 compatible in the same source code. I've been building python3 apks for a while without any problems of this nature (actually I forgot about this issue!).

With the recent release of python 3.6, it's become particularly important to fix up the python3 build, and I've been doing some preliminary work on this recently. The overall plan is to actually carry the hostpython and targetpython builds, using the build scripts of crystax, rather than using only the prebuilt binaries. I hope that if this is done well, it should both allow support of multiple python versions (in fact, possible most/all recent ones) and make the build more robust for people on different systems. However, things like this always throw up problems, so it's hard to say exactly what will happen when.

@FeralBytes
Copy link
Contributor

@inclement so do you think it is a recent change kivy clock that may be causing the issue then? I have Python 3.5.2. Can you paste your logcat.
Also what exactly should the requirements be at minimum; because I am confused between hostpython3crystax and python3crystax.
Also I found this output from buildozer to be odd:

[INFO]:    Downloading hostpython3crystax
[INFO]:    Skipping hostpython3crystax download as no URL is set
[INFO]:    Downloading python2
[INFO]:    -> running mkdir -p /home/wolfrage/0.Projects/EzTunes/src/EzTunes/.buildozer/android/platform/build/packages/python2
[INFO]:    -> directory context /home/wolfrage/0.Projects/EzTunes/src/EzTunes/.buildozer/android/platform/build/packages/python2
[INFO]:    -> running basename http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2
[INFO]:    -> running rm -f .mark-Python-2.7.2.tar.bz2
[INFO]:    Downloading python2 from http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2

And finally what exactly is hostpython doing; I am trying to close a gap in understanding that is why I am asking.

@FeralBytes FeralBytes mentioned this issue Jan 30, 2017
Closed
@inclement
Copy link
Member

@FeralBytes Probably not, the kivy clock module is just the first cython one that gets imported so normally it's the one that registers as broken even though they all are.

This setup.py file in the python-for-android testapp builds folders should produce a working python3 build. To use it, you'll need to update the ndk-dir path to point to your own local ndk.

You don't need to add hostpython3crystax, although it's a minor bug that it will apparently be allowed alongside python2 in the requirements.

@FeralBytes
Copy link
Contributor

@inclement worked through getting that file setup for my enviroment; and finally got it to build APK; but upon launching the APK on android; it crashes again as soon as kivy is imported with a failure pointing at _clock.so .
At this time I am unable to build anything with Kivy.

@FeralBytes
Copy link
Contributor

@inclement even tried with the Kivy Hello World Example.

I/python  ( 3069): main.py
I/python  ( 3069): [WARNING] [Config      ] Older configuration version detected (0 instead of 17)
I/python  ( 3069): [WARNING] [Config      ] Upgrading configuration in progress.
I/python  ( 3069): [INFO   ] [Logger      ] Record log in /data/data/com.feralbytes.EzTunes/files/app/.kivy/logs/kivy_70-01-01_0.txt
I/python  ( 3069): [INFO   ] [Kivy        ] v1.9.2.dev0, git-Unknown, 20170131
I/python  ( 3069): [INFO   ] [Python      ] v3.5.0 (default, Dec 24 2015, 05:56:06) 
I/python  ( 3069): [GCC 5.3 20151204]
I/python  ( 3069):  Traceback (most recent call last):
I/python  ( 3069):    File "main.py", line 3, in <module>
I/python  ( 3069):      from kivy.app import App
I/python  ( 3069):    File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/app.py", line 319, in <module>
I/python  ( 3069):      from kivy.base import runTouchApp, stopTouchApp
I/python  ( 3069):    File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/base.py", line 29, in <module>
I/python  ( 3069):      from kivy.clock import Clock
I/python  ( 3069):    File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/clock.py", line 362, in <module>
I/python  ( 3069):      from kivy._clock import CyClockBase, ClockEvent, FreeClockEvent, \
I/python  ( 3069):  ImportError: dlopen failed: cannot locate symbol "PyDict_Next" referenced by "_clock.so"...
I/python  ( 3069): Python for android ended.

Are you currently able to compile anything with Kivy?

@FeralBytes
Copy link
Contributor

By pushing Kivy way back in GIT I was able to get the error to change to:
I/python ( 3379): Traceback (most recent call last):
I/python ( 3379): File "main.py", line 3, in
I/python ( 3379): from kivy.app import App
I/python ( 3379): File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/app.py", line 319, in
I/python ( 3379): from kivy.base import runTouchApp, stopTouchApp
I/python ( 3379): File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/base.py", line 30, in
I/python ( 3379): from kivy.event import EventDispatcher
I/python ( 3379): File "/data/data/com.feralbytes.EzTunes/files/app/crystax_python/site-packages/kivy/event.py", line 8, in
I/python ( 3379): import kivy._event
I/python ( 3379): ImportError: dlopen failed: cannot locate symbol "PyObject_GetAttr" referenced by "_event.so"...
I/python ( 3379): Python for android ended.

Still working backwards.

@FeralBytes
Copy link
Contributor

So that every one is aware; at this time; I have stopped trying Crystax builds and Py3. I just am using my code under Py2; since my code is compatable; that is except python-osc. So I am now going to either switch to Kivy-OSC of port python-osc 3to2.
Either way I believe Crystax builds to be broken at this time. I do not see how to fix it and would need some further help before I even try anything else; already lost too much time to it.

@FeralBytes
Copy link
Contributor

For me crystax builds are now working either using Buildozer or a P4A setup.py script. I am not sure what exactly changed; whether it be on my system or else where; but by starting from a basic (not crystax) build with p4a and slowing adding requirements everything began working.
@Malverick and @KarimReefat are you still having this issue? Have you tracked it down any further?
I recommend trying to clean your python for android or buildozer build, dist, and packages directories.

@sumitmadhwani
Copy link
Author

sumitmadhwani commented Feb 10, 2017 •
edited
Loading

@FeralBytes I was implementing a feature for plyer and at that time apk was not building using android ndk so I was suggested to use crystax. After facing this issue, I somehow made my apk build using android ndk. So I continued my work on that feature and thought of looking into this issue sometime later. Now that you have got crystax builds working fine, if you suggest I will close this issue. Or we should wait for @KarimReefat's response?

@FeralBytes
Copy link
Contributor

Well before closing it; do you know if your Crystax build is working again? If Crystax is again working for you; then yes close it; because something got fixed for all of us; but I don't know what. Though it seems the common fact was going back to regular apk build process and working from there.

@inclement
Copy link
Member

Just looking through this issue, does it actually have anything to do with the crystax NDK? Do builds specifically work when not using CrystaX, but fail when using it?

The dlopen error appears to be caused by missing linker arguments under some circumstances, but it depends on the local environment in a way I don't understand. I may try some things at the weekend that could fix it.

@sumitmadhwani
Copy link
Author

Yes @inclement , the builds were working fine when not using CrystaX NDK and py3.

@inclement
Copy link
Member

inclement commented Feb 10, 2017 via email

@sumitmadhwani
Copy link
Author

I have not tested it with py2. I will test it using py2 and will inform you till tomorrow. Will that be fine?
If you want it urgently, I will do it now.

@inclement
Copy link
Member

inclement commented Feb 10, 2017 via email

@FeralBytes
Copy link
Contributor

@inclement for me the regression in being able to build was for more than just Crystax; that is until at least I went all the way back to the old tool chain. Then as I built back up everything was working again. So I really don't know what went wrong at least in my case.

@inclement
Copy link
Member

I've just pushed ed1e1a8 which should (I hope) fix this problem.

@FeralBytes
Copy link
Contributor

Please do not close this bug; I think I now know the issue.
@inclement I think your fix actually just highlighted the issue to me. When building my various setup scripts; (yes I have too many; but I was testing to make sure my code can build many ways) I found the 3.5m reference from your push. I also managed to get a useful java traceback.

I think the real issue is; once you build with crystax all of the libs we depend on: "libSDL2.so", "libpython2.7.so" or "libpython3.5.so" depend on "libcrystax.so". The issue arises when you go back to a non-crystax build; and with enough debugging the error actually shows it's self rather than blowing up on Android as happened to me previously.
The fix that I have applied to my setup.py files is to create in the ~/.local/share/python-for-android/ directory a flag file such as: crystax-ndk-10.3.2.flag
full path: ~/.local/share/python-for-android/crystax-ndk-10.3.2.flag
This allows my same type of builds to be fast but if the build NDK changes to Google's NDK or Crystax from the other then I know because the flag does not match the NDK name.
When it changes I execute the commands below to prevent any corrupt ".so" files that depend on libcrystax.so.
Please note I need to further test these commands to make sure they are correct; only had enough time to understand the problem not to ensure I caught every detail. So I will post back tomorrow with exact commands and then I can start a PR once I get some feedback from kivy-devs.
$ rm -rf ~/.local/share/python-for-android/*.flag
$ rm -rf ~/.local/share/python-for-android/build/bootstrap_builds/
$ rm -rf ~/.local/share/python-for-android/build/other_builds/
$ rm -rf ~/.local/share/python-for-android/build/python-installs/
$ rm -rf ~/.local/share/python-for-android/build/libs_collections/

Also if the dist name is the same for a Crystax and a Google build then the dist folder will also need to be removed because the dist will already contain the corrupted ".so" files and no build will occur.
$ rm -rf ~/.local/share/python-for-android/dist/{dist-name}/

Alright now for the adb logcat output; but with out the full memory dump:

D/dalvikvm( 5140): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2.so 0xadde42a8
E/dalvikvm( 5140): dlopen("/data/app-lib/com.feralbytes.eztunes-1/libSDL2.so") failed: dlopen failed: could not load library "libcrystax.so" needed by "libSDL2.so"; caused by library "libcrystax.so" not found
W/System.err( 5140): dlopen failed: could not load library "libcrystax.so" needed by "libSDL2.so"; caused by library "libcrystax.so" not found
D/dalvikvm( 5140): GC_FOR_ALLOC freed 1118K, 36% free 3159K/4876K, paused 14ms, total 14ms
I/dalvikvm-heap( 5140): Grow heap (frag case) to 3.839MB for 635812-byte allocation
D/dalvikvm( 5140): GC_FOR_ALLOC freed <1K, 23% free 3780K/4876K, paused 33ms, total 33ms
D/dalvikvm( 5140): GC_CONCURRENT freed <1K, 23% free 3787K/4876K, paused 7ms+12ms, total 53ms
W/dalvikvm( 5140): No implementation found for native Lorg/libsdl/app/SDLActivity;.nativeSetEnv:(Ljava/lang/String;Ljava/lang/String;)V
D/AndroidRuntime( 5140): Shutting down VM
W/dalvikvm( 5140): threadid=1: thread exiting with uncaught exception (group=0xadb56730)
E/AndroidRuntime( 5140): FATAL EXCEPTION: main
E/AndroidRuntime( 5140): java.lang.UnsatisfiedLinkError: Native method not found: org.libsdl.app.SDLActivity.nativeSetEnv:(Ljava/lang/String;Ljava/lang/String;)V
E/AndroidRuntime( 5140):     at org.libsdl.app.SDLActivity.nativeSetEnv(Native Method)
E/AndroidRuntime( 5140):     at org.kivy.android.PythonActivity$UnpackFilesTask.onPostExecute(PythonActivity.java:167)
E/AndroidRuntime( 5140):     at org.kivy.android.PythonActivity$UnpackFilesTask.onPostExecute(PythonActivity.java:114)
E/AndroidRuntime( 5140):     at android.os.AsyncTask.finish(AsyncTask.java:631)
E/AndroidRuntime( 5140):     at android.os.AsyncTask.access$600(AsyncTask.java:177)
E/AndroidRuntime( 5140):     at android.os.AsyncTask$InternalHandler.handleMessage(AsyncTask.java:644)
E/AndroidRuntime( 5140):     at android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime( 5140):     at android.os.Looper.loop(Looper.java:137)
E/AndroidRuntime( 5140):     at android.app.ActivityThread.main(ActivityThread.java:5103)
E/AndroidRuntime( 5140):     at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime( 5140):     at java.lang.reflect.Method.invoke(Method.java:525)
E/AndroidRuntime( 5140):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737)
E/AndroidRuntime( 5140):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
E/AndroidRuntime( 5140):     at dalvik.system.NativeStart.main(Native Method)
W/ActivityManager(  912):   Force finishing activity com.feralbytes.eztunes/org.kivy.android.PythonActivity

And this one shows that libpython is also affected:

D/dalvikvm( 5224): GC_CONCURRENT freed 424K, 16% free 4120K/4884K, paused 7ms+5ms, total 27ms
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2.so 0xadde76e8
D/dalvikvm( 5224): Added shared lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2.so 0xadde76e8
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_image.so 0xadde76e8
D/dalvikvm( 5224): Added shared lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_image.so 0xadde76e8
D/dalvikvm( 5224): No JNI_OnLoad found in /data/app-lib/com.feralbytes.eztunes-1/libSDL2_image.so 0xadde76e8, skipping init
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_mixer.so 0xadde76e8
D/dalvikvm( 5224): Added shared lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_mixer.so 0xadde76e8
D/dalvikvm( 5224): No JNI_OnLoad found in /data/app-lib/com.feralbytes.eztunes-1/libSDL2_mixer.so 0xadde76e8, skipping init
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_ttf.so 0xadde76e8
D/dalvikvm( 5224): Added shared lib /data/app-lib/com.feralbytes.eztunes-1/libSDL2_ttf.so 0xadde76e8
D/dalvikvm( 5224): No JNI_OnLoad found in /data/app-lib/com.feralbytes.eztunes-1/libSDL2_ttf.so 0xadde76e8, skipping init
D/dalvikvm( 5224): Trying to load lib /data/app-lib/com.feralbytes.eztunes-1/libpython2.7.so 0xadde76e8
E/dalvikvm( 5224): dlopen("/data/app-lib/com.feralbytes.eztunes-1/libpython2.7.so") failed: dlopen failed: could not load library "libcrystax.so" needed by "libpython2.7.so"; caused by library "libcrystax.so" not found
W/System.err( 5224): Couldn't load python3.5m from loader dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.feralbytes.eztunes-1.apk"],nativeLibraryDirectories=[/data/app-lib/com.feralbytes.eztunes-1, /system/lib]]]: findLibrary returned null

@FeralBytes FeralBytes mentioned this issue Feb 12, 2017
Merged
@FeralBytes
Copy link
Contributor

Update the command must eliminate all of build so now passing to subprocess:

['rm', '-rf', os.path.join(os.path.expanduser('~'), '.local/share/python-for-android/build')]

@rockcastle
Copy link

My tts apk is working fine with crystax ndk
https://github.com/rockcastle/Python-Kivy-Android-app

@yingshaoxo
Copy link
Contributor

In fact, I wonder why author don't ant to builder IMG if Python for Android rely on it.

Because I got wrong when I try to build apk with it.

@rockcastle
Copy link

Check my buildozer spec file.
I hope it helps
https://github.com/rockcastle/Python-Kivy-Android-app?files=1

@yingshaoxo
Copy link
Contributor

yingshaoxo commented May 24, 2017 via email

@rockcastle
Copy link

I have similar sizes for my apps.
import specific libs to make it small size.

@inclement
Copy link
Member

CrystaX-built APKs are about 9MB larger than they need to be due to a bug in what is included. You can make them much smaller by locating the dist directory and deleting the usr directory, which I think is in something like dist_dir/crystax_python/site-packages. I'll check the exact path when I next make a python3 APK.

@yingshaoxo
Copy link
Contributor

yingshaoxo commented May 28, 2017 via email

@rockcastle
Copy link

Hi,
"dist_dir/crystax_python/site-packages"
I have checked to make it smaller size but it did not worked for me.
Is that the place in below?
.buildozer/android/platform/build/dists/app_name/crystax_python/crystax_python/site-packages/

@AndreMiras
Copy link
Member

AndreMiras commented May 19, 2018 •
edited
Loading

CrystaX-built APKs are about 9MB larger than they need to be due to a bug in what is included. You can make them much smaller by locating the dist directory and deleting the usr directory, which I think is in something like dist_dir/crystax_python/site-packages. I'll check the exact path when I next make a python3 APK.

@inclement that's super interesting, I would do anything to have my APKs 9M smaller!
Until we make a built-in fix, is this something that can be workarounded with a blacklist.txt file? Do you have more info, I'm interested in looking at it.
Can we open a dedicated ticket for this topic?

Edit: following up our discussion on IRC, it does not seem to be an issue anymore. For record the problem was ...crystax_python/site-packages/ contained a usr/ directory with a lot of stuff inside. I've checked on my build and this directory was pretty light. So indeed not an issue anymore.

@homdx
Copy link

homdx commented Jan 15, 2019

Hello.
I managed to save files when they were successfully collected and you can download them from my page or from the docker:
https://github.com/homdx/KivyMD/releases

@ghost
Copy link

ghost commented Jan 15, 2019

@homdx thanks, that will be useful for anyone dropping by via google ❤️

@Malverick thanks for the report! I have been not involved in this discussion/ticket, but from what I can see this is mostly around CrystaX which is no longer needed: there is a regular python3 recipe now which uses the regular Google NDK! Also, the discussion appears to have gone astray from the initial issue.

I'll therefore close this for now, but feel absolutely free to poke me if there is still a problem with the particular issue you initially reported and I'll reopen - so far it's to my knowledge still undecided if/when CrystaX will be fully dropped, after all

@ghost ghost closed this as completed Jan 15, 2019
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@AndreMiras @inclement @KarimReefat @FeralBytes @yingshaoxo @rockcastle @sumitmadhwani @homdx