Fix build for case-insensitive FS and add CI test for OSX #1951
Conversation
|
No problem I'm happy to help, I'm very very new to python so I'm just fumbling around. I'll build a new docker with this patch and test it, hopefully it won't take me too long to test it. |
|
Nevermind, I thought the actual python on macOS was somehow installed as |
There was a problem hiding this comment.
Looks fine, except for the lack of testing. I'd rather not merge it without that confirmation, but since the tests prove it doesn't break linux builds I don't think it would be too terrible to merge and hope for the best. I didn't check the details of how you can be sure python.exe is the right filename.
pythonforandroid/python.py
Outdated
| @@ -382,6 +383,22 @@ class HostPythonRecipe(Recipe): | |||
| '''The default url to download our host python recipe. This url will | |||
| change depending on the python version set in attribute :attr:`version`.''' | |||
|
|
|||
| @property | |||
| def python_bin(self): | |||
There was a problem hiding this comment.
I have a vague feeling that it isn't necessarily obvious python_bin will be a name only, not a full path. I'd consider making it an internal property (i.e. just call it _python_bin). It's a minor thing though.
There was a problem hiding this comment.
The circumstances it may be different now since the last force push, now we use this property in PythonRecipe to determine the right hostpython location . If you still thing that we must set this property private let me know and I will change it, no problem.
|
I'd quite like to get this into the next release (sometime in the next few days), or at the very least: we could add a simple log that confirms the python exe name to give confidence in the change. It would then be easy to pull the result from any user's log. |
|
I’m hopefully going to run a test after dinner tonight. running it right now, I'll repost back once it gets to that part, I removed all my work arounds |
|
did not break, but also did not fix it. I'm trying to dig deeper but I'm pretty new to docker and even newer to python-for-android... be sooo much easier if it didn't capture the output :) I can't see where or why it fails, I'm going to attach and run it by hand see if I can figure it out. When it builds python3-libffi-openssl-sqlite3 what make command is used or where do I find the receipt for it? |
|
does this help looks like here it switches from python.exe to python, this might be the case insensitive issue, I'll dig further back and see |
|
so this is the line in the python3 Makefile so it can be modified, or make an alias could be made so that python calls actually call python.exe? |
|
So I'm continuing to investigate, I know I'm posting lots but I keep hoping that I'll post the right thing for someone that understands and knows the build process. oddly enough I do not have python, but the docker image does have python3 and python3.6 interpreters. But the python3 configure script seems to find python. for some reason this code int eh configure returns and there is no option for it to use the hostpython3 which is built just before it, which would then be ../../../../hostpython3/desktop/hostpython3/native-build/python.exe but patching for that would be insane, any other ideas? I still have no idea how it's finding python vs python3 :. Maybe it's being passed in via configure param or ENV? |
|
found the issue, it's the in the python configure script. it continues cause it's not found. BUT interp is already set, so by default it will finish the loop with interp='python' even if it's not found. So no matter what it'll think it found python when in reality it didn't find anything cause it needs to look for python.exe |
|
Testing with this setup, no idea if I'm out to lunch here but figured I'd help try to solve it, I still have the original PR on as well as this patch. fix-interpreter-for-darwin.patch |
|
okay bigger issue, because I'm running this in a docker on Darwin, is_darwin() does not return true but docker is still bleeding enough through to get python.exe and case insensitivity. Maybe host python defaults to python.exe cause it detect the fs being case insensitive? I'm testing that without using is_darwin() right now to make sure then I'll diagnose hostpython3 and see if I can figure out why it builds python.exe vs python. FOUND IT hostpython3, configure so if it's Case Intensive is is python.exe, hence is_darwin() won't help need to test the fs I'm going to end here cause it's late and I'm SURE there are more experienced ppl that can fix it from here. But I'm willing to test once a patch is available. my ideas are, built I'm not sure which is best, I pout them in order of effort not what I think is best.
Sorry for the spam everyone, was trying to work my way through it. |
|
@TheSin- That's great debugging, thanks! |
|
Thanks I just hope it helps. It's an odd issue but it'll be nice to have it solved I think. There are 2 other solutions at least for macOS build things on a USB drive that is formatted to be case sensitive (or install on case sensitive) or just run a VM which is what I did to get my build done. But native would be nice in the long run, and it seems like it should be easy enough to solve so long as that is the only issue since I didn't get to finish testing cause I wasn't sure on the direction the team would want to take to solve it. |
|
@inclement, so sorry for the delay in the response in here, I was on vacations until yesterday, later I will force push the changes which will fix this issue once and for all (I hope...) @TheSin-, so many thanks for your debugging session and for the pyton3's patch 👍 (I will include that in the next |
|
yeah it took me a bit to figure it all out since I'm so new to all this, docker and python-for-android, but it's a great tool so I wanted to help fix it for everyone. I'm glad I was able to help and once the push is available I'll give it all an other shot and let you know. I still have it all setup and ready. |
opacam
force-pushed
the
hotfix-macos-build
branch
from
6d26d07
to
84adc33
Compare
|
@inclement, I force pushed some changes and added a new commit, now I'm pretty sure that the build success, because I created a CI test for Mac OS X which will force the Anyway, at least now we are sure that the build for Mac OS X succeeds 😄...it will work the generated apk? ...I guess that we will find out...🤞 @TheSin-, I think with the added CI test it may not be needed to test it as you did, but I will be happy to read and comment your reflexions 😄 if you test it...we may found other issues related with Mac OSX build... note: I patched python2 for Mac OSX as @TheSin- did, but didn't test the build...teorically it should work... |
pythonforandroid/python.py
Outdated
|
|
||
| ''' | ||
| if is_darwin(): | ||
| return 'python.exe' |
There was a problem hiding this comment.
I'd like to go back to my comments, this will break some Darwin installs.
This patch should not be based on darwin tests at all, darwin can has both insensitive or sensitive for both HFS+ and AFS, so in the case of a casesensitive darwin install, this will break. The issue isn't purely darwin/macOS, it's just easier to see cause the default install is case insensitive.
A new test for FS case sensitivity should be made (is_caseinsensitivefs()) and it's results should get cached so it can be used for both he patches and the binary name IMHO
thus all places that is_darwin() is used in this PR should be is_caseinsensitivefs()
See
https://www.howtogeek.com/wp-content/uploads/2017/10/format-drive.png
|
here is the results of my current tests, but I need to revisit a bit of it. Other then what I already commented on. so it did build it, but it's trying to run python instead of python.exe and causing the build to fail. I'm hading home but I'll diagnose more once I'm there. |
|
Hi guys, Tested this (first time in months) and it built ok. my app is however crashing out at startup with no code changes so wondering if its something here. here's my logcat https://pastebin.com/raw/nX6VfmAC looks like somthing to do with lib python so |
|
aaa...that's good news @OptimusGREEN , we have the build... but I still not convinced of the work I made in here, I'm testing in another branch a different approach (considering the last comments of @TheSin- ) and it will simplify the whole stuff...I will push it later... regarding the runtime error...what kivy version do you use... if that is the case I would recommend you to stick to p4a version...there was some conversation on discord (yesterday) and the error I see it's pretty similar (hard to tell with this kind of libc errors...) If not...well...we have another problem... |
.travis.yml
Outdated
| # download/extract android's SDK | ||
| - curl --location --progress-bar --insecure $ANDROID_SDK_TOOLS_DL_URL --output $ANDROID_SDK_TOOLS_ARCHIVE | ||
| - mkdir -p $ANDROID_SDK_HOME | ||
| - unzip -q $ANDROID_SDK_TOOLS_ARCHIVE -d $ANDROID_SDK_HOME |
There was a problem hiding this comment.
It feels like this whole download thing could go to some helper scripts maybe under ci/. And if not too painful, it could also be OS agnostic. That could also actually be part of some Makefile that could be easily reused inside or outside CI. For instance something like https://github.com/AndreMiras/dockerfiles/blob/51a88e3/buildozer_android/android_sdk.mk
.travis.yml
Outdated
| - mkdir -p $ANDROID_SDK_HOME | ||
| - unzip -q $ANDROID_SDK_TOOLS_ARCHIVE -d $ANDROID_SDK_HOME | ||
| - rm -rf $ANDROID_SDK_TOOLS_ARCHIVE | ||
| - ls $HOME |
There was a problem hiding this comment.
Minor: is this for debugging purpose?
pythonforandroid/python.py
Outdated
| ''' | ||
| if not self.version: | ||
| raise BuildInterruptingException( | ||
| 'The hostpython recipe must has set version' |
| self.ctx.hostpython = join(build_dir, 'python') | ||
| with current_directory(recipe_build_dir): | ||
| # Configure the build | ||
| with current_directory(build_dir): |
There was a problem hiding this comment.
Why do we switch current directory twice in a row? Is it because the second time it's a relative path?
Edit: I mean why are these context manger nested
There was a problem hiding this comment.
Why do we switch current directory twice in a row? Is it because the second time it's a relative path?
I'm not pretty sure of this, since I only removed one indentation level, but, apart of relative paths, I think that it's because we need to write config.status inside native-build folder...also all the thing makes sense in one context because it's a part of the build process of python.
Anyway...maybe @inclement can add some words or correct me. He did the original hostpython3/python3 recipes 😄 (I only refactored a little, some months back, to have the python2 version of the thing based on the work He made...but the concept and logic are the @inclement's ones 😉)
There was a problem hiding this comment.
Thanks, I was just curious, maybe if I get to curious I'll play with it later. But let's get this PR merged first 😄
| 'patches/fix-ctypes-util-find-library.patch'] | ||
| 'patches/fix-ctypes-util-find-library.patch', | ||
| 'patches/fix-interpreter-version.patch', | ||
| ] |
There was a problem hiding this comment.
Wait are we still spending time trying to fix Python2 related things?
Well now it's done, but I wouldn't have bothered
There was a problem hiding this comment.
Wait are we still spending time trying to fix Python2 related things?
Well...we still support it right?...2020 is almost here...:smile:
There was a problem hiding this comment.
Not tested for regression, but looking good overall.
I can't think of what could go wrong at run time (well probably everything as usual 😛).
I can give a building & runtime check later this week if I make some time.
My only concern is the .travis.yml file starts to look ugly
There was a problem hiding this comment.
Thanks @opacam, great work - especially not having osx to test on!
I'm inclined to merge this, but @AndreMiras' concerns about the travis file seem reasonable, what are your thoughts on that? It does feel like the preparation of p4a dependencies on osx should be abstracted somewhere else.
|
@AndreMiras and @inclement, thanks for your reviews 😄 ...I was hoping that you guys don't want this Mac OS X test 😆. I already expressed my feelings about mac, so the main goal was only to be sure that the thing worked (at least at build time). But since there is interest on have this test, I agree with you guys, the Mac OS X logic has to be moved somewhere else, isolating the android part (so we can refactor as well the android part of our docker files). I liked the @AndreMiras's idea of |
It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way To achieve this goal we refactor a little our `HostPythonRecipe`: - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version) - add public property `HostPythonRecipe.python_exe` (the full path of the python executable) - implement `HostPythonRecipe.should_build` And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`: - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...) - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS We also need: - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version - fix python2 interpreter (fix-interpreter-version.patch) Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions) Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!!
opacam
force-pushed
the
hotfix-macos-build
branch
from
511caea
to
7ce1e3d
Compare
ci/makefiles/android.mk
Outdated
| # Those android NDK/SDK variables can be override when running the file | ||
| ifndef ANDROID_NDK_VERSION | ||
| ANDROID_NDK_VERSION=17c | ||
| endif |
There was a problem hiding this comment.
What about the shorthand default? e.g.
ANDROID_NDK_VERSION ?= 17c
There was a problem hiding this comment.
Also changed other entries from android.mk and osx.mk files 😉
@AndreMiras, thank you for the tip!!! 😄
| USER ${USER} | ||
|
|
||
| # Download and install android's NDK/SDK | ||
| RUN make -f ci/makefiles/android.mk target_os=linux |
There was a problem hiding this comment.
Nice! I'm just wondering, this layer would be cached entirely then correct? This is fine, I wasn't sure if it would be cached or not
There was a problem hiding this comment.
...this layer would be cached entirely then correct? This is fine, I wasn't sure...
I hope so...but I don't know for sure...so...:no_mouth:
| extract_android_sdk: | ||
| mkdir -p $(ANDROID_SDK_HOME) \ | ||
| && unzip -q $(ANDROID_SDK_TOOLS_ARCHIVE) -d $(ANDROID_SDK_HOME) \ | ||
| && rm -f $(ANDROID_SDK_TOOLS_ARCHIVE) |
There was a problem hiding this comment.
Minor for later.
I'm wondering if we do want to clean it or use it as a target for not re-downloading it next time. So basically the idea wold be to have something like:
$(ANDROID_SDK_TOOLS_ARCHIVE):
curl --location --progress-bar --continue-at - \
$(ANDROID_SDK_TOOLS_DL_URL) --output $(ANDROID_SDK_TOOLS_ARCHIVE
download_android_sdk: $(ANDROID_SDK_TOOLS_ARCHIVE)That way running the target twice won't run the download twice if already present. Same story for other targets/dependencies
There was a problem hiding this comment.
I was thinking in keeping docker's layer smallest as possible...since we don't to re use the removed files right now...so ...maybe better to change/test this if we need it, right? 😉
There was a problem hiding this comment.
Yes let's keep it that way 😄
| @@ -0,0 +1,78 @@ | |||
| # Downloads and installs the Android SDK depending on supplied platform: darwin or linux | |||
There was a problem hiding this comment.
Nice DRY file! Pretty useful 👏
opacam
force-pushed
the
hotfix-macos-build
branch
from
7ce1e3d
to
b195952
Compare
.travis.yml
Outdated
| docker run | ||
| -e CI | ||
| -e TRAVIS_JOB_ID="$TRAVIS_JOB_ID" | ||
| -e TRAVIS_BRANCH="$TRAVIS_BRANCH" |
There was a problem hiding this comment.
Just for your info, you can simplify to -e TRAVIS_BRANCH since variables are the same you can skip the = part. Just like we did for -e CI
Dockerfile.py2
Outdated
| && apt -y clean | ||
|
|
||
| # Install Java, set JAVA_HOME (to accept android's SDK licenses) and clean | ||
| RUN ${RETRY} apt -y install openjdk-8-jdk && apt -y autoremove && apt -y clean |
There was a problem hiding this comment.
Just wondering, what about --no-install-recommends
There was a problem hiding this comment.
Better, so I changed this line with -qq --no-install-recommends (so we keep our logs smaller) 👍
|
|
||
| all: install_java upgrade_cython install_android_ndk_sdk install_p4a | ||
|
|
||
| install_java: |
There was a problem hiding this comment.
This is to think about later, but installing system requirements could actually be part Linux Makefile as well
| before_script: | ||
| # installs java 1.8, android's SDK/NDK and p4a | ||
| - make -f ci/makefiles/osx.mk | ||
| - export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home |
There was a problem hiding this comment.
Just wondering, is needed for building the app in the script part below?
There was a problem hiding this comment.
Yes, we need it because the default java version for travis's OS X image (xcode11) has another version of java
See also: kivy/buildozer#945
| - export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home | ||
| # Run a background process to make sure that travis will not kill our tests in | ||
| # case that the travis log doesn't produce any output for more than 10 minutes | ||
| - while sleep 540; do echo "==== Still running (travis, don't kill me) ===="; done & |
There was a problem hiding this comment.
That might not be needed for the OSX build actually, because I don't think it would run more than 20 or 30 minutes without log printing.
travis_wait waits 20 minutes by default and can be increased up to 30 minutes by passing time as parameter.
There was a problem hiding this comment.
I maintained the background process...I think that could be useful..., but I removed one line of the above comment (since we already commented before)...but we can remove it if you feel this is useless...just let me know 😉
Dockerfile.py2
Outdated
| && apt -y autoremove \ | ||
| && apt -y clean | ||
|
|
||
| # Install Java, set JAVA_HOME (to accept android's SDK licenses) and clean |
There was a problem hiding this comment.
I lost track of this accept part, but it actually is in the Makefile (update_android_sdk). Maybe at some point the system dep could also move to the Makefile. I don't know if that would be a good idea or not
| $(info Android NDK download url is : $(ANDROID_NDK_DL_URL)) | ||
| $(info Android API level is : $(ANDROID_API_LEVEL)) | ||
| $(info Android NDK version is : $(ANDROID_NDK_VERSION)) | ||
| $(info JAVA_HOME is : $(JAVA_HOME)) |
| self.ctx.hostpython = join(build_dir, 'python') | ||
| with current_directory(recipe_build_dir): | ||
| # Configure the build | ||
| with current_directory(build_dir): |
There was a problem hiding this comment.
Thanks, I was just curious, maybe if I get to curious I'll play with it later. But let's get this PR merged first 😄
opacam
force-pushed
the
hotfix-macos-build
branch
from
b195952
to
a60eed8
Compare
To do so we: - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk` - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk` - refactor docker files: make use of android's makefile - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build.
opacam
force-pushed
the
hotfix-macos-build
branch
from
a60eed8
to
5a4484b
Compare
* [python] Fix build for case-insensitive FS 🍎 It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way To achieve this goal we refactor a little our `HostPythonRecipe`: - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version) - add public property `HostPythonRecipe.python_exe` (the full path of the python executable) - implement `HostPythonRecipe.should_build` And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`: - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...) - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS We also need: - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version - fix python2 interpreter (fix-interpreter-version.patch) Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions) Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!! * [ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation To do so we: - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk` - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk` - refactor docker files: make use of android's makefile - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build.
* [recipes] Fix compilation for regex recipe The error was: build/other_builds/hostpython3/desktop/hostpython3/Include/Python.h:39:19: fatal error: crypt.h: No such file or directory * [recipes] Update regex's recipe * put the sdk manager command in a code block (#1956) to avoid quotes being replaced by fancy ones * Updated version number for develop branch following 2019.08.09 release (#1960) * Fix build for case-insensitive FS and add CI test for OSX (#1951) * [python] Fix build for case-insensitive FS 🍎 It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way To achieve this goal we refactor a little our `HostPythonRecipe`: - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version) - add public property `HostPythonRecipe.python_exe` (the full path of the python executable) - implement `HostPythonRecipe.should_build` And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`: - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...) - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS We also need: - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version - fix python2 interpreter (fix-interpreter-version.patch) Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions) Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!! * [ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation To do so we: - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk` - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk` - refactor docker files: make use of android's makefile - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build. * customizability (#1869) * customizability options - added compile-options * apptheme option for all bootstraps * Initial migration to NDK r19 (Part I - The core) (#1722) * [ndk19] Rework of the arch environment to work with ndk r19+ Things done here: - Modifies target so we can use the toolchains `in-place` (as if we were using a standalone toolchain) - Removes some cflags that we don't need anymore - Move `macros` and `includes` from `CFLAGS` to `CPPFLAGS` (because this is the way it's supposed to be) - Move libraries from `LDFLAGS` to `LDLIBS` (again, the same as above) - Refactor `Arch`'s variables into class attributes, to make `get_env` smaller and to be more readable/maintainable: + class attributes for flags: * common_cflags * common_cppflags * common_ldflags * common_ldlibs * common_ldshared * arch_cflags + class attributes for compiler: * clang_path * clang_exe * clang_exe_cxx - Remove hardcoded cpu count in `MAKE` - Remove `gcc` of environment (because it has been partially removed in ndk r18) - Shorten long line to be pep8 friendly (line where we define env['BUILDLIB_PATH']) - Commented lines for `ArchAarch_64` that it seems tha we don't need anymore (not removed because I'm not sure if we need them) Note: This will force us to use a minimal ndk version of 19 See also: https://developer.android.com/ndk/guides/other_build_systems * [ndk19] Update `test_archs` to ndk's r19+ build system Also add some more tests. Now we check if some of the functions we mock, `glob` and `find_executable`, are called with the expected arguments * [ndk19] Update `test_bootstrap` to ndk's r19+ build system Because `clang` is the new default compiler and the affected tests were made thinking in `gcc` compiler * [ndk19] Remove `clang` of `get_recipe_env` Because as per ndk r19 the default compiler is clang and gcc has been partially removed * [ndk19] Adapt python recipes to ndk's r19+ build system We move python recipe to use a minimal ndk version of 19 because it will simplify the python's cross compilation (this way we can remove some environment variables that we don't need anymore). The only side effect detected is that the python's `zlib` module isn't built anymore. This is caused because python3 and python2 build systems contains libz's hardcoded version, which does not match the version used in ndk 19 (may be this the cause of some `zip/unzip` errors reported in python-for-android's issues?). We will take care later of this zlib problem, or we will get a runtime crash...so it seems that zlib is an `essential module` to successfully run a python-for-android app * [ndk19] Update libffi to version post `v3.3RC0` and adapt it to be ndk r19+ compatible * [ndk19] Fix build of python's zlib module Because python has a hardcoded zlib version which does not match android's ndk r19, and that prevents the build of zlib's module Note: it seems that we have to point to the right lib and headers, otherwise we won't succeed in building python's zlib module * [ndk19] Fix build of python2's _ctypes module for ndk r19+ * [ndk19] Fix numpy for ndk r19+ Note: because of the new build system recently introduced, we can safely remove some of the patches that we needed because it seems that it builds and runs fine without them * [ndk19] Update docs with the new ndk requirements * [ndk19] Set min/max/recommended android's NDK * [ndk19] Make CI tests with android's NDK `r19b` Because it's our minimum supported android NDK * [ndk19] Fix png recipe for NDK r19+ We remove `get_recipe_env` because we don't need anymore to specify the flags set in there. Note: As per new NDK r19+ build system implemented in p4a, the removed `flags` should be automatically set by the android's NDK * [ndk19] Add `-fPIC` to `armeabi-v7a`'s arch cflags... ...force `Position-Independent Code` To solve the kivy's build error detected in travis `OSX` test (https://travis-ci.org/kivy/python-for-android/jobs/576204088#L1105). But also detected the same error in my local tests (linux os in this case) for `freetype` and `harfbuzz` recipes...so I have the feeling that this is better to set as global 🤞..is it? * [LIBS - PART I] Initial refactor of library recipes (#1944) This commit will introduce a new cls attribute: `built_libraries`. This `built_libraries` attribute must be a dictionary. The keys are the full library name, e.g.: `libffi.so`. And the values must be the relative path of the library, e.g: `.libs`. So the this cls attribute for libffi recipe it would look like: ```built_libraries = {'libffi.so': '.libs'}``` This new cls attribute will allow us to detect library recipes and to refactor common operations that we do for those kind of recipes: - copy library into right location - implement `should_build` for library recipes Those two things will make that our rebuilds to be more consistent and I hope that it will be easier to maintain. **So, once explained a little this PR, in here we do:** - [x] Add attribute `built_libraries` - [x] Add methods to refactor common library operations: - [x] install_libraries - [x] get_libraries - [x] implement `should_build` for library recipes - [x] implement basic tests for the newly introduced `Recipe` methods - [x] Make use of those Library attributes/methods for some basic set of recipes: - [x] libffi - [x] openssl - [x] png - [x] jpeg - [x] freetype - [x] harfbuzz - [x] libcurl - [x] libzbar - [x] libiconv - [x] libexpat - [x] libogg - [x] libxml2 - [x] libxslt - [x] libshine - [x] libx264 - [x] libglob - [x] libsodium - [x] libsecp256k1 **Things to do in separate PRs**, to be easy to review and because those points will start to conflict with the `NDK r19 migration` thing just merged: - Make use of Library attributes/methods introduced in this PR for un covered library recipes in here. Here we have two situations: - library recipes that depends on android's STL library (almost all the work done in here will depend of the `NDK r19 migration` thing) - all remaining library recipes, which are not STL lib dependent and that are not implemented in here (because I was unable to perform a successful build with them using arch `arm64-v8a`...so I think it would be better to deal with those recipes in separate PRs and later...with the `NDK r19 migration` thing merged) **Notes about changed recipes:** all the recipes touched in here almost have the same changes: - remove `should_build` method (to make use of the one implemented in base class for library recipes) - remove the copy of the library (because that will be done automatically by the method `install_libraries` implemented in base class) - fixed the imports due to refactoring * [recipe-lib] Add attribute Recipe.built_libraries so we can refactor some essential operations performed for library recipes (like copy the library into the right location, or decide if we should build the library recipe again) * [recipe-lib] Add method Recipe.get_libraries This function will allows us to get a list of the built/context libraries. This will be useful to check if we have to built a library recipe or which library should be rebuild * [recipe-lib] Make `Recipe.should_build` decide the library build In case that the attribute built_libraries has been set, we will check the need of the recipe build, otherwise we will act as usual, forcing the build unless overwrote in subclass * [recipe-lib] Add method `Recipe.install_libraries` So we can: - control the copy of the library recipe - refactor the install of the lib for all library recipes * [recipe-lib] Make libffi a library recipe * [recipe-lib] Make openssl a library recipe and ... also make the imports from the right module * [recipe-lib] Make png a library recipe * [recipe-lib] Make jpeg a library recipe * [recipe-lib] Make freetype a library recipe and ... also make the imports from the right module * [recipe-lib] Make harfbuzz a library recipe and ... also make the imports from the right module * [recipe-lib] Make libcurl a library recipe and ... also make the imports from the right module * [recipe-lib] Make libzbar a library recipe and ... also make the imports from the right module * [recipe-lib] Make libiconv a library recipe and ... also make the imports from the right module * [recipe-lib] Make libexpat a library recipe and ... also: - make the imports from the right module - remove hardcoded arch * [recipe-lib] Make libogg a library recipe * [recipe-lib] Make libxml2 a library recipe and ... also make the imports from the right module * [recipe-lib] Make libxslt a library recipe and ... also make the imports from the right module * [recipe-lib] Make libshine a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libx264 a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libglob a library recipe * [recipe-lib] Make libsodium a library recipe and ... also: - make the imports from the right module - fix hardcoded arch - enable cpu count when compiling * [recipe-lib] Make libsecp256k1 a library recipe and ... also make the imports from the right module * [tests] Add tests for library recipe * [NDK19] Fix libglob for android NDK r19 - change the `ARG_MAX` define, because it's already defined at `sysroot/usr/include/linux/limits.h` - Replaced `size_t` by Including the <stddef.h> header. Because found the solution at stackoverflow: As per C99, §7.17, size_t is not a builtin type but defined in <stddef.h> See also: - https://travis-ci.org/kivy/python-for-android/jobs/576392711#L5992-L6013 - https://stackoverflow.com/questions/26410466/gcc-linaro-compiler-throws-error-unknown-type-name-size-t * [NDK19] Remove `--sysroot` from LDFLAGS for cffi and pymunk (#1965) Because `--sysroot` flag is not needed anymore and make it fails the build since we migrated to new android's build system (implemented in NDK r19) See also: - https://developer.android.com/ndk/guides/other_build_systems * Added libffi headers troubleshooting note to doc (#1972) * [LIBS - PART II] Part II of NDK r19 migration - Initial STL lib migration (#1947) * [recipe-stl] Add android's STL lib support to `Recipe` To allow us to refactor some common operations that we use in our recipes that depends on android's STL library. Note: This commit will allow us to begin the migration to `c++_shared`. This is a must when we move to android's NDK r19+, because as for android NDK >= 18 is the only one supported STL library. * [recipe-stl] Make CppCompiledComponentsPythonRecipe use Recipe's STL support * [recipe-stl] Make icu a library recipe with STL support (rework) Also done here: - Remove hardcoded version in url - Disable versioned shared libraries - Make it to be build as a shared libraries (instead of static) - Disable the build of static libraries (because we build them as shared ones, so we avoid to link with them without our consents) - Shorten long lines to be pep8's friendly - Remove icu from ci/constants - Remove `RuntimeError` about the need to use NDK api <= 19 (because that is not the case anymore) - consider host's number of cpu to perform the build * [recipe-stl] Rework pyicu recipe to match latest icu changes Also done here: - Remove icu.patch because now we don't have the version in our icu libraries - Shorten long lines to be pep8's friendly * [tests] Add tests for recipe with STL support * [tests] Add tests for icu recipe * [tests] Add tests for pyicu recipe * [WIP][LIBS - PART III] Rework of pyleveldb, leveldb and snappy (#1966) * [recipe-lib] Make snappy a library recipe * [recipe-stl] Make leveldb a library recipe and... make it work with the reworked snappy recipe * [recipe-stl] Fix pyleveldb for reworked leveldb/snappy * [WIP][LIBS - PART IV] Rework of shapely and libgeos (#1967) * [recipe-stl] Make libgeos a library recipe * [recipe-stl] Fix shapely for reworked libgeos * [recipe-stl] Make libzmq a library recipe (#1968) * [recipe-stl] Rework of libtorrent and boost (#1971) which: - fix build for android's NDK r19+ - allow us to build `boost+libtorrent` for any arch - Update boost to version `1.69.0` - update libtorrent to version `1.2.1` The build method needs to be changed because one of the scripts used in `boost+libtorrent` build (make-standalone-toolchain.sh), has been removed from android's NDK. [This is because](https://developer.android.com/ndk/guides/standalone_toolchain): `As of r19, the NDK's default toolchains are standalone toolchains, which renders this process unnecessary.` Note: since 3887d2b, `python-for-android` uses android's NDK r19 as the minimum supported and also changed the build method used by p4a as [described in here](https://developer.android.com/ndk/guides/other_build_systems) * Also copy the service/main.py when building with setup.py (#1936) * [recipe-stl] Rework of protobuf_cpp recipe (#1969) In here we do: - inherit from CppCompiledComponentsPythonRecipe Because depends on android's STL library - make use of the base class methods for library recipes - Split build_arch into proper methods - Shorten some long lines (to be PEP8 friendly) - make generated library shared - remove recipe from CI/constants * [CI] Fix CI's test for `arm64-v8a` (#1977) * Updated README.md to clarify NDK versions (#1981) * 📚 Updated README.dm to clarify NDK versions * 🔧 Change back to r19b * A bunch of tests for library recipes (#1982) The main idea is to test as many recipes as we can with the less code possible and without creating any file/directory so our tests can be performed as fast as possible (all this tests will only add between 2 and 3 seconds to our CI tests and will cover almost 100% of the code for each tested recipe) To do so, we create a couple of modules: tests.recipe_ctx: allow us to create a proper Context to test our recipes tests.recipe_lib_test: which holds some base classes to be used to test a recipe depending on the build method used. For starters we introduce two kind of base classes: BaseTestForMakeRecipe: To test an standard library build (this will iinclude the recipes which requires the classical build commandsconfigure/make) BaseTestForCmakeRecipe: To test an library recipe which is compiled with cmake We also refactor the existing recipes tests, so we can remove some lines in there...the ones that creates a Context. * [test] Add module `tests.recipe_ctx` A helper module to test recipes. Here we will initialize a `Context` to test recipes. * [test] Refactor `setUp/tearDown` for `test_icu` * [test] Refactor `setUp/tearDown` for `test_pyicu` * [test] Refactor `setUp` for `test_reportlab` * [test] Refactor `setUp` for `test_gevent` * [test] Add module `tests.recipe_lib_test` A helper module to test recipes which will allow to test any recipe using `configure/make` commands. * [test] Add test for `libffi` * [test] Add test for `libexpat` * [test] Add test for `libcurl` * [test] Add test for `libiconv` * [test] Add test for `libogg` * [test] Add test for `libpq` * [test] Add test for `libsecp256k1` * [test] Add test for `libshine` * [test] Add test for `libvorbis` * [test] Add test for `libx264` * [test] Add test for `libxml2` * [test] Add test for `libxslt` * [test] Add test for `png` * [test] Add test for `freetype` * [test] Add test for `harfbuzz` * [test] Add test for `openssl` * [test] Add `BaseTestForCmakeRecipe` * [test] Add test for `jpeg` and clean code We can remove the `get_recipe_env` because the environment that we use is already using a clang as default compiler (not the case when we migrated the jpeg recipe to use `cmake`)...so we can do a little refactor :) * [test] Add test for `snappy` * [test] Add test for `leveldb` * [test] Add test for `libgeos` * [test] Add test for `libmysqlclient` * [test] Add test for `openal` * [test] Make the `super` calls Python3 style * [test] Move mock tests outside context manager... Because there is no need to do it there. Also rewrote the inline comments. * Recipes tests enhancements (#1984) This is a follow up of #1982 In here we do: - Move from `__new__` to `__init__` for `BaseTestForMakeRecipe` ([discussed at here](#1982 (comment))) - Convert cls attributes to instance attributes ([discussed at here](#1982 (comment))) - Remove `mock` dependency for py3 tests (Because, as per Python 3.3+, `mock` it's a built-in-module). (Unfortunately we still will have a couple of `import mock` entries that we cannot remove until we completely remove python2 from our tests) * [test] From `__new__` to `__init__` for `BaseTestForMakeRecipe` * [test] Remove mock dependency for py3 tests Because, as per Python 3.3+, `mock` it's a built-in-module * [test] Convert cls attributes to instance attributes Because we may have some side effects without willing, considering that cls attributes are shared between instances. * Fixes test_virtualenv and test_venv failing, closes #1994 (#1995) Recent pep517 release seems to break tests. * Made p4a use per-arch dist build dirs (#1986) * Made p4a use per-arch dist build dirs * Fixed up check on existing folder when overwriting distributions * Improved Distribution.get_distribution api * Fixed test_archs.py * Fixed test_bootstrap.py * Fixed test_distribution.py * Fixed recipes tests * Fixed formatting * Made sure whole build process accesses dist_dir correctly * Fixed apk output name formatting by moving version to end of string * Further test fixes following dist_dir changes * Removed unnecessary variable * Fixes libiconv & libzbar configure host Having `--host=x86` was making the configure script fail at enabling shared library build on `x86`. This was resulting in failing to copy `libiconv.so` and `libzbar.so` on `x86` arch. * Updates Java version troubleshooting (#1991) Adds the NoClassDefFoundError XmlSchema case and groups related cases together. Gives instructions for both Ubuntu and macOS to fix. * Made on-device unit tests app use the develop branch by default (#1985) * Updated version to 2019.10.06 Co-authored-by: Pol Canelles <[email protected]> Co-authored-by: Gabriel Pettier <[email protected]> Co-authored-by: Philipp Auersperg-Castell <[email protected]> Co-authored-by: Richard Larkin <[email protected]> Co-authored-by: Andre Miras <[email protected]>
* Release 2019.10.06 (#1998) * [recipes] Fix compilation for regex recipe The error was: build/other_builds/hostpython3/desktop/hostpython3/Include/Python.h:39:19: fatal error: crypt.h: No such file or directory * [recipes] Update regex's recipe * put the sdk manager command in a code block (#1956) to avoid quotes being replaced by fancy ones * Updated version number for develop branch following 2019.08.09 release (#1960) * Fix build for case-insensitive FS and add CI test for OSX (#1951) * [python] Fix build for case-insensitive FS 🍎 It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way To achieve this goal we refactor a little our `HostPythonRecipe`: - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version) - add public property `HostPythonRecipe.python_exe` (the full path of the python executable) - implement `HostPythonRecipe.should_build` And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`: - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...) - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS We also need: - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version - fix python2 interpreter (fix-interpreter-version.patch) Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions) Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!! * [ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation To do so we: - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk` - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk` - refactor docker files: make use of android's makefile - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build. * customizability (#1869) * customizability options - added compile-options * apptheme option for all bootstraps * Initial migration to NDK r19 (Part I - The core) (#1722) * [ndk19] Rework of the arch environment to work with ndk r19+ Things done here: - Modifies target so we can use the toolchains `in-place` (as if we were using a standalone toolchain) - Removes some cflags that we don't need anymore - Move `macros` and `includes` from `CFLAGS` to `CPPFLAGS` (because this is the way it's supposed to be) - Move libraries from `LDFLAGS` to `LDLIBS` (again, the same as above) - Refactor `Arch`'s variables into class attributes, to make `get_env` smaller and to be more readable/maintainable: + class attributes for flags: * common_cflags * common_cppflags * common_ldflags * common_ldlibs * common_ldshared * arch_cflags + class attributes for compiler: * clang_path * clang_exe * clang_exe_cxx - Remove hardcoded cpu count in `MAKE` - Remove `gcc` of environment (because it has been partially removed in ndk r18) - Shorten long line to be pep8 friendly (line where we define env['BUILDLIB_PATH']) - Commented lines for `ArchAarch_64` that it seems tha we don't need anymore (not removed because I'm not sure if we need them) Note: This will force us to use a minimal ndk version of 19 See also: https://developer.android.com/ndk/guides/other_build_systems * [ndk19] Update `test_archs` to ndk's r19+ build system Also add some more tests. Now we check if some of the functions we mock, `glob` and `find_executable`, are called with the expected arguments * [ndk19] Update `test_bootstrap` to ndk's r19+ build system Because `clang` is the new default compiler and the affected tests were made thinking in `gcc` compiler * [ndk19] Remove `clang` of `get_recipe_env` Because as per ndk r19 the default compiler is clang and gcc has been partially removed * [ndk19] Adapt python recipes to ndk's r19+ build system We move python recipe to use a minimal ndk version of 19 because it will simplify the python's cross compilation (this way we can remove some environment variables that we don't need anymore). The only side effect detected is that the python's `zlib` module isn't built anymore. This is caused because python3 and python2 build systems contains libz's hardcoded version, which does not match the version used in ndk 19 (may be this the cause of some `zip/unzip` errors reported in python-for-android's issues?). We will take care later of this zlib problem, or we will get a runtime crash...so it seems that zlib is an `essential module` to successfully run a python-for-android app * [ndk19] Update libffi to version post `v3.3RC0` and adapt it to be ndk r19+ compatible * [ndk19] Fix build of python's zlib module Because python has a hardcoded zlib version which does not match android's ndk r19, and that prevents the build of zlib's module Note: it seems that we have to point to the right lib and headers, otherwise we won't succeed in building python's zlib module * [ndk19] Fix build of python2's _ctypes module for ndk r19+ * [ndk19] Fix numpy for ndk r19+ Note: because of the new build system recently introduced, we can safely remove some of the patches that we needed because it seems that it builds and runs fine without them * [ndk19] Update docs with the new ndk requirements * [ndk19] Set min/max/recommended android's NDK * [ndk19] Make CI tests with android's NDK `r19b` Because it's our minimum supported android NDK * [ndk19] Fix png recipe for NDK r19+ We remove `get_recipe_env` because we don't need anymore to specify the flags set in there. Note: As per new NDK r19+ build system implemented in p4a, the removed `flags` should be automatically set by the android's NDK * [ndk19] Add `-fPIC` to `armeabi-v7a`'s arch cflags... ...force `Position-Independent Code` To solve the kivy's build error detected in travis `OSX` test (https://travis-ci.org/kivy/python-for-android/jobs/576204088#L1105). But also detected the same error in my local tests (linux os in this case) for `freetype` and `harfbuzz` recipes...so I have the feeling that this is better to set as global 🤞..is it? * [LIBS - PART I] Initial refactor of library recipes (#1944) This commit will introduce a new cls attribute: `built_libraries`. This `built_libraries` attribute must be a dictionary. The keys are the full library name, e.g.: `libffi.so`. And the values must be the relative path of the library, e.g: `.libs`. So the this cls attribute for libffi recipe it would look like: ```built_libraries = {'libffi.so': '.libs'}``` This new cls attribute will allow us to detect library recipes and to refactor common operations that we do for those kind of recipes: - copy library into right location - implement `should_build` for library recipes Those two things will make that our rebuilds to be more consistent and I hope that it will be easier to maintain. **So, once explained a little this PR, in here we do:** - [x] Add attribute `built_libraries` - [x] Add methods to refactor common library operations: - [x] install_libraries - [x] get_libraries - [x] implement `should_build` for library recipes - [x] implement basic tests for the newly introduced `Recipe` methods - [x] Make use of those Library attributes/methods for some basic set of recipes: - [x] libffi - [x] openssl - [x] png - [x] jpeg - [x] freetype - [x] harfbuzz - [x] libcurl - [x] libzbar - [x] libiconv - [x] libexpat - [x] libogg - [x] libxml2 - [x] libxslt - [x] libshine - [x] libx264 - [x] libglob - [x] libsodium - [x] libsecp256k1 **Things to do in separate PRs**, to be easy to review and because those points will start to conflict with the `NDK r19 migration` thing just merged: - Make use of Library attributes/methods introduced in this PR for un covered library recipes in here. Here we have two situations: - library recipes that depends on android's STL library (almost all the work done in here will depend of the `NDK r19 migration` thing) - all remaining library recipes, which are not STL lib dependent and that are not implemented in here (because I was unable to perform a successful build with them using arch `arm64-v8a`...so I think it would be better to deal with those recipes in separate PRs and later...with the `NDK r19 migration` thing merged) **Notes about changed recipes:** all the recipes touched in here almost have the same changes: - remove `should_build` method (to make use of the one implemented in base class for library recipes) - remove the copy of the library (because that will be done automatically by the method `install_libraries` implemented in base class) - fixed the imports due to refactoring * [recipe-lib] Add attribute Recipe.built_libraries so we can refactor some essential operations performed for library recipes (like copy the library into the right location, or decide if we should build the library recipe again) * [recipe-lib] Add method Recipe.get_libraries This function will allows us to get a list of the built/context libraries. This will be useful to check if we have to built a library recipe or which library should be rebuild * [recipe-lib] Make `Recipe.should_build` decide the library build In case that the attribute built_libraries has been set, we will check the need of the recipe build, otherwise we will act as usual, forcing the build unless overwrote in subclass * [recipe-lib] Add method `Recipe.install_libraries` So we can: - control the copy of the library recipe - refactor the install of the lib for all library recipes * [recipe-lib] Make libffi a library recipe * [recipe-lib] Make openssl a library recipe and ... also make the imports from the right module * [recipe-lib] Make png a library recipe * [recipe-lib] Make jpeg a library recipe * [recipe-lib] Make freetype a library recipe and ... also make the imports from the right module * [recipe-lib] Make harfbuzz a library recipe and ... also make the imports from the right module * [recipe-lib] Make libcurl a library recipe and ... also make the imports from the right module * [recipe-lib] Make libzbar a library recipe and ... also make the imports from the right module * [recipe-lib] Make libiconv a library recipe and ... also make the imports from the right module * [recipe-lib] Make libexpat a library recipe and ... also: - make the imports from the right module - remove hardcoded arch * [recipe-lib] Make libogg a library recipe * [recipe-lib] Make libxml2 a library recipe and ... also make the imports from the right module * [recipe-lib] Make libxslt a library recipe and ... also make the imports from the right module * [recipe-lib] Make libshine a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libx264 a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libglob a library recipe * [recipe-lib] Make libsodium a library recipe and ... also: - make the imports from the right module - fix hardcoded arch - enable cpu count when compiling * [recipe-lib] Make libsecp256k1 a library recipe and ... also make the imports from the right module * [tests] Add tests for library recipe * [NDK19] Fix libglob for android NDK r19 - change the `ARG_MAX` define, because it's already defined at `sysroot/usr/include/linux/limits.h` - Replaced `size_t` by Including the <stddef.h> header. Because found the solution at stackoverflow: As per C99, §7.17, size_t is not a builtin type but defined in <stddef.h> See also: - https://travis-ci.org/kivy/python-for-android/jobs/576392711#L5992-L6013 - https://stackoverflow.com/questions/26410466/gcc-linaro-compiler-throws-error-unknown-type-name-size-t * [NDK19] Remove `--sysroot` from LDFLAGS for cffi and pymunk (#1965) Because `--sysroot` flag is not needed anymore and make it fails the build since we migrated to new android's build system (implemented in NDK r19) See also: - https://developer.android.com/ndk/guides/other_build_systems * Added libffi headers troubleshooting note to doc (#1972) * [LIBS - PART II] Part II of NDK r19 migration - Initial STL lib migration (#1947) * [recipe-stl] Add android's STL lib support to `Recipe` To allow us to refactor some common operations that we use in our recipes that depends on android's STL library. Note: This commit will allow us to begin the migration to `c++_shared`. This is a must when we move to android's NDK r19+, because as for android NDK >= 18 is the only one supported STL library. * [recipe-stl] Make CppCompiledComponentsPythonRecipe use Recipe's STL support * [recipe-stl] Make icu a library recipe with STL support (rework) Also done here: - Remove hardcoded version in url - Disable versioned shared libraries - Make it to be build as a shared libraries (instead of static) - Disable the build of static libraries (because we build them as shared ones, so we avoid to link with them without our consents) - Shorten long lines to be pep8's friendly - Remove icu from ci/constants - Remove `RuntimeError` about the need to use NDK api <= 19 (because that is not the case anymore) - consider host's number of cpu to perform the build * [recipe-stl] Rework pyicu recipe to match latest icu changes Also done here: - Remove icu.patch because now we don't have the version in our icu libraries - Shorten long lines to be pep8's friendly * [tests] Add tests for recipe with STL support * [tests] Add tests for icu recipe * [tests] Add tests for pyicu recipe * [WIP][LIBS - PART III] Rework of pyleveldb, leveldb and snappy (#1966) * [recipe-lib] Make snappy a library recipe * [recipe-stl] Make leveldb a library recipe and... make it work with the reworked snappy recipe * [recipe-stl] Fix pyleveldb for reworked leveldb/snappy * [WIP][LIBS - PART IV] Rework of shapely and libgeos (#1967) * [recipe-stl] Make libgeos a library recipe * [recipe-stl] Fix shapely for reworked libgeos * [recipe-stl] Make libzmq a library recipe (#1968) * [recipe-stl] Rework of libtorrent and boost (#1971) which: - fix build for android's NDK r19+ - allow us to build `boost+libtorrent` for any arch - Update boost to version `1.69.0` - update libtorrent to version `1.2.1` The build method needs to be changed because one of the scripts used in `boost+libtorrent` build (make-standalone-toolchain.sh), has been removed from android's NDK. [This is because](https://developer.android.com/ndk/guides/standalone_toolchain): `As of r19, the NDK's default toolchains are standalone toolchains, which renders this process unnecessary.` Note: since 3887d2b, `python-for-android` uses android's NDK r19 as the minimum supported and also changed the build method used by p4a as [described in here](https://developer.android.com/ndk/guides/other_build_systems) * Also copy the service/main.py when building with setup.py (#1936) * [recipe-stl] Rework of protobuf_cpp recipe (#1969) In here we do: - inherit from CppCompiledComponentsPythonRecipe Because depends on android's STL library - make use of the base class methods for library recipes - Split build_arch into proper methods - Shorten some long lines (to be PEP8 friendly) - make generated library shared - remove recipe from CI/constants * [CI] Fix CI's test for `arm64-v8a` (#1977) * Updated README.md to clarify NDK versions (#1981) * 📚 Updated README.dm to clarify NDK versions * 🔧 Change back to r19b * A bunch of tests for library recipes (#1982) The main idea is to test as many recipes as we can with the less code possible and without creating any file/directory so our tests can be performed as fast as possible (all this tests will only add between 2 and 3 seconds to our CI tests and will cover almost 100% of the code for each tested recipe) To do so, we create a couple of modules: tests.recipe_ctx: allow us to create a proper Context to test our recipes tests.recipe_lib_test: which holds some base classes to be used to test a recipe depending on the build method used. For starters we introduce two kind of base classes: BaseTestForMakeRecipe: To test an standard library build (this will iinclude the recipes which requires the classical build commandsconfigure/make) BaseTestForCmakeRecipe: To test an library recipe which is compiled with cmake We also refactor the existing recipes tests, so we can remove some lines in there...the ones that creates a Context. * [test] Add module `tests.recipe_ctx` A helper module to test recipes. Here we will initialize a `Context` to test recipes. * [test] Refactor `setUp/tearDown` for `test_icu` * [test] Refactor `setUp/tearDown` for `test_pyicu` * [test] Refactor `setUp` for `test_reportlab` * [test] Refactor `setUp` for `test_gevent` * [test] Add module `tests.recipe_lib_test` A helper module to test recipes which will allow to test any recipe using `configure/make` commands. * [test] Add test for `libffi` * [test] Add test for `libexpat` * [test] Add test for `libcurl` * [test] Add test for `libiconv` * [test] Add test for `libogg` * [test] Add test for `libpq` * [test] Add test for `libsecp256k1` * [test] Add test for `libshine` * [test] Add test for `libvorbis` * [test] Add test for `libx264` * [test] Add test for `libxml2` * [test] Add test for `libxslt` * [test] Add test for `png` * [test] Add test for `freetype` * [test] Add test for `harfbuzz` * [test] Add test for `openssl` * [test] Add `BaseTestForCmakeRecipe` * [test] Add test for `jpeg` and clean code We can remove the `get_recipe_env` because the environment that we use is already using a clang as default compiler (not the case when we migrated the jpeg recipe to use `cmake`)...so we can do a little refactor :) * [test] Add test for `snappy` * [test] Add test for `leveldb` * [test] Add test for `libgeos` * [test] Add test for `libmysqlclient` * [test] Add test for `openal` * [test] Make the `super` calls Python3 style * [test] Move mock tests outside context manager... Because there is no need to do it there. Also rewrote the inline comments. * Recipes tests enhancements (#1984) This is a follow up of #1982 In here we do: - Move from `__new__` to `__init__` for `BaseTestForMakeRecipe` ([discussed at here](#1982 (comment))) - Convert cls attributes to instance attributes ([discussed at here](#1982 (comment))) - Remove `mock` dependency for py3 tests (Because, as per Python 3.3+, `mock` it's a built-in-module). (Unfortunately we still will have a couple of `import mock` entries that we cannot remove until we completely remove python2 from our tests) * [test] From `__new__` to `__init__` for `BaseTestForMakeRecipe` * [test] Remove mock dependency for py3 tests Because, as per Python 3.3+, `mock` it's a built-in-module * [test] Convert cls attributes to instance attributes Because we may have some side effects without willing, considering that cls attributes are shared between instances. * Fixes test_virtualenv and test_venv failing, closes #1994 (#1995) Recent pep517 release seems to break tests. * Made p4a use per-arch dist build dirs (#1986) * Made p4a use per-arch dist build dirs * Fixed up check on existing folder when overwriting distributions * Improved Distribution.get_distribution api * Fixed test_archs.py * Fixed test_bootstrap.py * Fixed test_distribution.py * Fixed recipes tests * Fixed formatting * Made sure whole build process accesses dist_dir correctly * Fixed apk output name formatting by moving version to end of string * Further test fixes following dist_dir changes * Removed unnecessary variable * Fixes libiconv & libzbar configure host Having `--host=x86` was making the configure script fail at enabling shared library build on `x86`. This was resulting in failing to copy `libiconv.so` and `libzbar.so` on `x86` arch. * Updates Java version troubleshooting (#1991) Adds the NoClassDefFoundError XmlSchema case and groups related cases together. Gives instructions for both Ubuntu and macOS to fix. * Made on-device unit tests app use the develop branch by default (#1985) * Updated version to 2019.10.06 Co-authored-by: Pol Canelles <[email protected]> Co-authored-by: Gabriel Pettier <[email protected]> Co-authored-by: Philipp Auersperg-Castell <[email protected]> Co-authored-by: Richard Larkin <[email protected]> Co-authored-by: Andre Miras <[email protected]> * Bumps openssl to 1.1.1f * Updated version to 2019.10.06.post0 Co-authored-by: Pol Canelles <[email protected]> Co-authored-by: Gabriel Pettier <[email protected]> Co-authored-by: Philipp Auersperg-Castell <[email protected]> Co-authored-by: Richard Larkin <[email protected]> Co-authored-by: Andre Miras <[email protected]> Co-authored-by: Andre Miras <[email protected]>
It turns out that the generated binary for
MacOSandCygwinis notpython...it'spython.exe...this leads to not set correctly thectx.hostpythonvariable forMac OS. This rename of the python binary forMacOSis to, probably, avoid issues with theMac OSfilesystem which could be Case Insensitive and it would conflict with the build folder namedPython).I hope that this solve the
Mac OSbuild issue 🤞Note: this should be tested by some mac user, unfortunately I could not test it because I don't have a mac...so I recommend to merge after someone test this, but all seems indicate that this is the problem of p4a with Mac OS...thanks to this comment we got the solution 😄,Thanks @TheSin-
See also: https://github.com/python/cpython/blob/3.7/README.rst#build-instructions
Closes: #1817
Closes: #1800
Closes: #1682
Closes: #1647