Skip to content

Commit

Permalink
Fix build for case-insensitive FS and add CI test for OSX (kivy#1951)
Browse files Browse the repository at this point in the history 
* [python] Fix build for case-insensitive FS 🍎

It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way

To achieve this goal we refactor a little our `HostPythonRecipe`:
  - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version)
  - add public property `HostPythonRecipe.python_exe` (the full path of the python executable)
  - implement `HostPythonRecipe.should_build`

And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`:
  - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...)
  - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS

We also need:
  - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version
  - fix python2 interpreter (fix-interpreter-version.patch)

Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions)

Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!!

* [ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation

To do so we:
  - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk`
  - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk`
  - refactor docker files: make use of android's makefile
  - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs
  - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build.
  • Loading branch information
@opacam @joergbrech
opacam authored and joergbrech committed Nov 10, 2019
1 parent b3215a6 commit df77774
Show file tree
Hide file tree
Showing 9 changed files with 221 additions and 144 deletions.
39 changes: 28 additions & 11 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@ before_install:
- git remote set-branches --add origin develop
- git fetch

env:
global:
- ANDROID_SDK_HOME=/opt/android/android-sdk
- ANDROID_NDK_HOME=/opt/android/android-ndk

jobs:
include:
- &linting
Expand Down Expand Up @@ -65,19 +60,41 @@ jobs:
# case that the travis log doesn't produce any output for more than 10 minutes
- while sleep 540; do echo "==== Still running (travis, don't kill me) ===="; done &
script:
- docker run -e CI -e TRAVIS_JOB_ID="$TRAVIS_JOB_ID" -e TRAVIS_BRANCH="$TRAVIS_BRANCH" p4a /bin/sh -c "$COMMAND"
- >
docker run
-e CI
-e TRAVIS_JOB_ID
-e TRAVIS_BRANCH
-e ANDROID_SDK_HOME="/home/user/.android/android-sdk"
-e ANDROID_NDK_HOME="/home/user/.android/android-ndk"
p4a /bin/sh -c "$COMMAND"
after_script:
# kill the background process started before run docker
- kill %1
name: Python 3 armeabi-v7a
name: Python 3 arm64-v8a
# overrides requirements to skip `peewee` pure python module, see:
# https://github.com/kivy/python-for-android/issues/1263#issuecomment-390421054
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=armeabi-v7a
env:
COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=arm64-v8a
- <<: *testing
name: Python 3 arm64-v8a
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=arm64-v8a
name: Python 3 armeabi-v7a
os: osx
osx_image: xcode11 # since xcode1.3, python3 is the default interpreter
before_script:
# installs java 1.8, android's SDK/NDK and p4a
- make -f ci/makefiles/osx.mk
- export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
# Run a background process (like we do with linux tests)
- while sleep 540; do echo "==== Still running (travis, don't kill me) ===="; done &
script:
- >
cd testapps && python3 setup_testapp_python3_sqlite_openssl.py apk
--sdk-dir $HOME/.android/android-sdk
--ndk-dir $HOME/.android/android-ndk
--requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools
--arch=armeabi-v7a
- <<: *testing
name: Python 2 basic
name: Python 2 armeabi-v7a (with numpy)
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python2_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements sdl2,pyjnius,kivy,python2,openssl,requests,sqlite3,setuptools,numpy'
- <<: *testing
name: Rebuild updated recipes
Expand Down
62 changes: 11 additions & 51 deletions Dockerfile.py2
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,64 +37,17 @@ ENV RETRY="retry -t 3 --"
RUN curl https://raw.githubusercontent.com/kadwanev/retry/1.0.1/retry \
--output /usr/local/bin/retry && chmod +x /usr/local/bin/retry

ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
ENV ANDROID_NDK_VERSION="17c"
ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"

# get the latest version from https://developer.android.com/ndk/downloads/index.html
ENV ANDROID_NDK_ARCHIVE="android-ndk-r${ANDROID_NDK_VERSION}-linux-x86_64.zip"
ENV ANDROID_NDK_DL_URL="https://dl.google.com/android/repository/${ANDROID_NDK_ARCHIVE}"

# download and install Android NDK
RUN ${RETRY} curl --location --progress-bar --insecure \
"${ANDROID_NDK_DL_URL}" \
--output "${ANDROID_NDK_ARCHIVE}" \
&& mkdir --parents "${ANDROID_NDK_HOME_V}" \
&& unzip -q "${ANDROID_NDK_ARCHIVE}" -d "${ANDROID_HOME}" \
&& ln -sfn "${ANDROID_NDK_HOME_V}" "${ANDROID_NDK_HOME}" \
&& rm -rf "${ANDROID_NDK_ARCHIVE}"


ENV ANDROID_SDK_HOME="${ANDROID_HOME}/android-sdk"

# get the latest version from https://developer.android.com/studio/index.html
ENV ANDROID_SDK_TOOLS_VERSION="4333796"
ENV ANDROID_SDK_BUILD_TOOLS_VERSION="28.0.2"
ENV ANDROID_SDK_TOOLS_ARCHIVE="sdk-tools-linux-${ANDROID_SDK_TOOLS_VERSION}.zip"
ENV ANDROID_SDK_TOOLS_DL_URL="https://dl.google.com/android/repository/${ANDROID_SDK_TOOLS_ARCHIVE}"

# download and install Android SDK
RUN ${RETRY} curl --location --progress-bar --insecure \
"${ANDROID_SDK_TOOLS_DL_URL}" \
--output "${ANDROID_SDK_TOOLS_ARCHIVE}" \
&& mkdir --parents "${ANDROID_SDK_HOME}" \
&& unzip -q "${ANDROID_SDK_TOOLS_ARCHIVE}" -d "${ANDROID_SDK_HOME}" \
&& rm -rf "${ANDROID_SDK_TOOLS_ARCHIVE}"

# update Android SDK, install Android API, Build Tools...
RUN mkdir --parents "${ANDROID_SDK_HOME}/.android/" \
&& echo '### User Sources for Android SDK Manager' \
> "${ANDROID_SDK_HOME}/.android/repositories.cfg"

# Download and accept Android licenses (JDK necessary!)
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
&& apt -y autoremove
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" > /dev/null
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" > /dev/null

# Set avdmanager permissions (executable)
RUN chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"


ENV USER="user"
ENV HOME_DIR="/home/${USER}"
ENV ANDROID_HOME="${HOME_DIR}/.android"
ENV WORK_DIR="${HOME_DIR}" \
PATH="${HOME_DIR}/.local/bin:${PATH}"

# install system dependencies
RUN ${RETRY} apt -y install -qq --no-install-recommends \
python virtualenv python-pip wget lbzip2 patch sudo \
&& apt -y autoremove
&& apt -y autoremove \
&& apt -y clean

# build dependencies
# https://buildozer.readthedocs.io/en/latest/installation.html#android-on-ubuntu-16-04-64bit
Expand All @@ -113,6 +66,10 @@ RUN ${RETRY} apt -y install -qq --no-install-recommends \
&& apt -y autoremove \
&& apt -y clean

# Install Java, set JAVA_HOME (to accept android's SDK licenses) and clean
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
&& apt -y autoremove && apt -y clean
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

# prepare non root env
RUN useradd --create-home --shell /bin/bash ${USER}
Expand All @@ -124,9 +81,12 @@ RUN echo "%sudo ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

WORKDIR ${WORK_DIR}
COPY --chown=user:user . ${WORK_DIR}
RUN chown --recursive ${USER} ${ANDROID_SDK_HOME}
RUN mkdir ${ANDROID_HOME} && chown --recursive ${USER} ${ANDROID_HOME}
USER ${USER}

# Download and install android's NDK/SDK
RUN make -f ci/makefiles/android.mk target_os=linux

# install python-for-android from current branch
RUN virtualenv --python=python venv \
&& . venv/bin/activate \
Expand Down
61 changes: 9 additions & 52 deletions Dockerfile.py3
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@

FROM ubuntu:18.04

ENV ANDROID_HOME="/opt/android"

# configure locale
RUN apt update -qq > /dev/null && apt install -qq --yes --no-install-recommends \
locales && \
Expand All @@ -37,57 +35,9 @@ ENV RETRY="retry -t 3 --"
RUN curl https://raw.githubusercontent.com/kadwanev/retry/1.0.1/retry \
--output /usr/local/bin/retry && chmod +x /usr/local/bin/retry

ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
ENV ANDROID_NDK_VERSION="17c"
ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"

# get the latest version from https://developer.android.com/ndk/downloads/index.html
ENV ANDROID_NDK_ARCHIVE="android-ndk-r${ANDROID_NDK_VERSION}-linux-x86_64.zip"
ENV ANDROID_NDK_DL_URL="https://dl.google.com/android/repository/${ANDROID_NDK_ARCHIVE}"

# download and install Android NDK
RUN ${RETRY} curl --location --progress-bar --insecure \
"${ANDROID_NDK_DL_URL}" \
--output "${ANDROID_NDK_ARCHIVE}" \
&& mkdir --parents "${ANDROID_NDK_HOME_V}" \
&& unzip -q "${ANDROID_NDK_ARCHIVE}" -d "${ANDROID_HOME}" \
&& ln -sfn "${ANDROID_NDK_HOME_V}" "${ANDROID_NDK_HOME}" \
&& rm -rf "${ANDROID_NDK_ARCHIVE}"


ENV ANDROID_SDK_HOME="${ANDROID_HOME}/android-sdk"

# get the latest version from https://developer.android.com/studio/index.html
ENV ANDROID_SDK_TOOLS_VERSION="4333796"
ENV ANDROID_SDK_BUILD_TOOLS_VERSION="28.0.2"
ENV ANDROID_SDK_TOOLS_ARCHIVE="sdk-tools-linux-${ANDROID_SDK_TOOLS_VERSION}.zip"
ENV ANDROID_SDK_TOOLS_DL_URL="https://dl.google.com/android/repository/${ANDROID_SDK_TOOLS_ARCHIVE}"

# download and install Android SDK
RUN ${RETRY} curl --location --progress-bar --insecure \
"${ANDROID_SDK_TOOLS_DL_URL}" \
--output "${ANDROID_SDK_TOOLS_ARCHIVE}" \
&& mkdir --parents "${ANDROID_SDK_HOME}" \
&& unzip -q "${ANDROID_SDK_TOOLS_ARCHIVE}" -d "${ANDROID_SDK_HOME}" \
&& rm -rf "${ANDROID_SDK_TOOLS_ARCHIVE}"

# update Android SDK, install Android API, Build Tools...
RUN mkdir --parents "${ANDROID_SDK_HOME}/.android/" \
&& echo '### User Sources for Android SDK Manager' \
> "${ANDROID_SDK_HOME}/.android/repositories.cfg"

# Download and accept Android licenses (JDK necessary!)
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
&& apt -y autoremove
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" > /dev/null
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" > /dev/null

# Set avdmanager permissions (executable)
RUN chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"


ENV USER="user"
ENV HOME_DIR="/home/${USER}"
ENV ANDROID_HOME="${HOME_DIR}/.android"
ENV WORK_DIR="${HOME_DIR}" \
PATH="${HOME_DIR}/.local/bin:${PATH}"

Expand All @@ -114,6 +64,10 @@ RUN ${RETRY} apt -y install -qq --no-install-recommends \
&& apt -y autoremove \
&& apt -y clean

# Install Java and set JAVA_HOME (to accept android's SDK licenses)
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
&& apt -y autoremove && apt -y clean
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

# prepare non root env
RUN useradd --create-home --shell /bin/bash ${USER}
Expand All @@ -127,9 +81,12 @@ RUN pip2 install --upgrade Cython==0.28.6

WORKDIR ${WORK_DIR}
COPY --chown=user:user . ${WORK_DIR}
RUN chown --recursive ${USER} ${ANDROID_SDK_HOME}
RUN mkdir ${ANDROID_HOME} && chown --recursive ${USER} ${ANDROID_HOME}
USER ${USER}

# Download and install android's NDK/SDK
RUN make -f ci/makefiles/android.mk target_os=linux

# install python-for-android from current branch
RUN virtualenv --python=python3 venv \
&& . venv/bin/activate \
Expand Down
68 changes: 68 additions & 0 deletions ci/makefiles/android.mk
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Downloads and installs the Android SDK depending on supplied platform: darwin or linux

# We must provide a platform (darwin or linux) and we need JAVA_HOME defined
ifndef target_os
$(error target_os is not set...aborted!)
endif

# Those android NDK/SDK variables can be override when running the file
ANDROID_NDK_VERSION ?= 17c
ANDROID_SDK_TOOLS_VERSION ?= 4333796
ANDROID_SDK_BUILD_TOOLS_VERSION ?= 28.0.2
ANDROID_HOME ?= $(HOME)/.android
ANDROID_API_LEVEL ?= 27

ANDROID_SDK_HOME=$(ANDROID_HOME)/android-sdk
ANDROID_SDK_TOOLS_ARCHIVE=sdk-tools-$(target_os)-$(ANDROID_SDK_TOOLS_VERSION).zip
ANDROID_SDK_TOOLS_DL_URL=https://dl.google.com/android/repository/$(ANDROID_SDK_TOOLS_ARCHIVE)

ANDROID_NDK_HOME=$(ANDROID_HOME)/android-ndk
ANDROID_NDK_FOLDER=$(ANDROID_HOME)/android-ndk-r$(ANDROID_NDK_VERSION)
ANDROID_NDK_ARCHIVE=android-ndk-r$(ANDROID_NDK_VERSION)-$(target_os)-x86_64.zip
ANDROID_NDK_DL_URL=https://dl.google.com/android/repository/$(ANDROID_NDK_ARCHIVE)

$(info Target install OS is : $(target_os))
$(info Android SDK home is : $(ANDROID_SDK_HOME))
$(info Android NDK home is : $(ANDROID_NDK_HOME))
$(info Android SDK download url is : $(ANDROID_SDK_TOOLS_DL_URL))
$(info Android NDK download url is : $(ANDROID_NDK_DL_URL))
$(info Android API level is : $(ANDROID_API_LEVEL))
$(info Android NDK version is : $(ANDROID_NDK_VERSION))
$(info JAVA_HOME is : $(JAVA_HOME))

all: install_sdk install_ndk

install_sdk: download_android_sdk extract_android_sdk update_android_sdk

install_ndk: download_android_ndk extract_android_ndk

download_android_sdk:
curl --location --progress-bar --continue-at - \
$(ANDROID_SDK_TOOLS_DL_URL) --output $(ANDROID_SDK_TOOLS_ARCHIVE)

download_android_ndk:
curl --location --progress-bar --continue-at - \
$(ANDROID_NDK_DL_URL) --output $(ANDROID_NDK_ARCHIVE)

# Extract android SDK and remove the compressed file
extract_android_sdk:
mkdir -p $(ANDROID_SDK_HOME) \
&& unzip -q $(ANDROID_SDK_TOOLS_ARCHIVE) -d $(ANDROID_SDK_HOME) \
&& rm -f $(ANDROID_SDK_TOOLS_ARCHIVE)


# Extract android NDK and remove the compressed file
extract_android_ndk:
mkdir -p $(ANDROID_NDK_FOLDER) \
&& unzip -q $(ANDROID_NDK_ARCHIVE) -d $(ANDROID_HOME) \
&& ln -sfn $(ANDROID_NDK_FOLDER) $(ANDROID_NDK_HOME) \
&& rm -f $(ANDROID_NDK_ARCHIVE)

# updates Android SDK, install Android API, Build Tools and accept licenses
update_android_sdk:
touch $(ANDROID_HOME)/repositories.cfg
yes | $(ANDROID_SDK_HOME)/tools/bin/sdkmanager --licenses > /dev/null
$(ANDROID_SDK_HOME)/tools/bin/sdkmanager "build-tools;$(ANDROID_SDK_BUILD_TOOLS_VERSION)" > /dev/null
$(ANDROID_SDK_HOME)/tools/bin/sdkmanager "platforms;android-$(ANDROID_API_LEVEL)" > /dev/null
# Set avdmanager permissions (executable)
chmod +x $(ANDROID_SDK_HOME)/tools/bin/avdmanager
23 changes: 23 additions & 0 deletions ci/makefiles/osx.mk
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# installs java 1.8, android's SDK/NDK, cython and p4a

# The following variable/s can be override when running the file
ANDROID_HOME ?= $(HOME)/.android

all: install_java upgrade_cython install_android_ndk_sdk install_p4a

install_java:
brew tap adoptopenjdk/openjdk
brew cask install adoptopenjdk8
/usr/libexec/java_home -V

upgrade_cython:
pip3 install --upgrade Cython==0.28.6

install_android_ndk_sdk:
mkdir -p $(ANDROID_HOME)
make -f ci/makefiles/android.mk target_os=darwin JAVA_HOME=`/usr/libexec/java_home -v 1.8`

install_p4a:
# check python version and install p4a
python3 --version
pip3 install -e .
Loading

0 comments on commit df77774

Please sign in to comment.