cffi UnicodeEncodeError: 'ascii' codec can't encode character '\u2018'

[AndreMiras]

Versions

• Python: host python3, target python3
• OS: Ubuntu 18.04
• Kivy: N/A
• Cython: 0.28.6

Description

In a Docker environment, compiling cffi will fail. The compilation command used:

python setup_testapp_python3.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --bootstrap sdl2 --requirements python3,cffi

buildozer.spec

Command:

python setup_testapp_python3.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --bootstrap sdl2 --requirements python3,cffi

Logs

(venv) user@5bbd34e1299b:~/testapps$ p4a clean_recipe_build cffi
[INFO]:    Cleaning build for cffi recipe.
[INFO]:    Deleting /home/user/.local/share/python-for-android/build/other_builds/cffi-python3
(venv) user@5bbd34e1299b:~/testapps$ python setup_testapp_python3.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --bootstrap sdl2 --requirements python3,cffi
packages are []
running apk
WARNING: You did not supply an Android package identifier, trying org.test.testapp_python3_googlendk instead.
         This may fail if this is not a valid identifier
Including testapp/main.py
Including testapp/colours.png
[WARNING]: --ndk-version is deprecated and no longer necessary, the value you passed is ignored
[INFO]:    Will compile for the following archs: armeabi-v7a
[INFO]:    Getting Android API version from user argument: 27
[INFO]:    Available Android APIs are (19, 27)
[INFO]:    Requested API target 27 is available, continuing.
[INFO]:    Getting NDK dir from from user argument
[INFO]:    Found NDK revision 17.2.4988734
[INFO]:    Getting NDK API version (i.e. minimum supported API) from user argument
[INFO]:    Found virtualenv at /usr/bin/virtualenv
[INFO]:    Found the following toolchain versions: ['4.9']
[INFO]:    Picking the latest gcc toolchain, here 4.9
[INFO]:    No existing dists meet the given requirements!
[INFO]:    No dist exists that meets your requirements, so one will be built.
[INFO]:    Found a single valid recipe set: ['hostpython3', 'libffi', 'python3', 'setuptools', 'pycparser', 'cffi']
[INFO]:    The selected bootstrap is sdl2
[INFO]:    # Creating dist with sdl2 bootstrap
[INFO]:    Dist will have name bdisttest_python3_googlendk and recipes (python3, cffi)
[INFO]:    Dist will also contain modules () installed from pip
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Recipe build order is ['hostpython3', 'libffi', 'python3', 'setuptools', 'pycparser', 'cffi']
[INFO]:    # Downloading recipes 
[INFO]:    Downloading hostpython3
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/hostpython3
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/hostpython3
[INFO]:    -> running basename https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
[INFO]:    hostpython3 download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading libffi
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/libffi
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/libffi
[INFO]:    -> running basename https://github.com/libffi/libffi/archive/v3.2.1.tar.gz
[INFO]:    libffi download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading python3
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/python3
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/python3
[INFO]:    -> running basename https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
[INFO]:    python3 download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading setuptools
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/setuptools
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/setuptools
[INFO]:    -> running basename https://pypi.python.org/packages/source/s/setuptools/setuptools-40.0.0.zip
[INFO]:    setuptools download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading pycparser
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/pycparser
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/pycparser
[INFO]:    -> running basename https://pypi.python.org/packages/source/p/pycparser/pycparser-2.14.tar.gz
[INFO]:    pycparser download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading cffi
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/cffi
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/cffi
[INFO]:    -> running basename https://pypi.python.org/packages/source/c/cffi/cffi-1.11.5.tar.gz
[INFO]:    cffi download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    # Building all recipes for arch armeabi-v7a
[INFO]:    # Unpacking recipes
[INFO]:    Unpacking hostpython3 for armeabi-v7a
[INFO]:    -> running basename https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop
[INFO]:    hostpython3 is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking libffi for armeabi-v7a
[INFO]:    -> running basename https://github.com/libffi/libffi/archive/v3.2.1.tar.gz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/libffi/armeabi-v7a__ndk_target_21
[INFO]:    libffi is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking python3 for armeabi-v7a
[INFO]:    -> running basename https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/python3-libffi/armeabi-v7a__ndk_target_21
[INFO]:    python3 is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking setuptools for armeabi-v7a
[INFO]:    -> running basename https://pypi.python.org/packages/source/s/setuptools/setuptools-40.0.0.zip
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/setuptools-python3/armeabi-v7a__ndk_target_21
[INFO]:    setuptools is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking pycparser for armeabi-v7a
[INFO]:    -> running basename https://pypi.python.org/packages/source/p/pycparser/pycparser-2.14.tar.gz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/pycparser-python3/armeabi-v7a__ndk_target_21
[INFO]:    pycparser is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking cffi for armeabi-v7a
[INFO]:    -> running basename https://pypi.python.org/packages/source/c/cffi/cffi-1.11.5.tar.gz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21
[INFO]:    -> running mv cffi-1.11.5 /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi
[INFO]:    <- directory context /home/user/testapps
[INFO]:    # Prebuilding recipes
[INFO]:    Prebuilding hostpython3 for armeabi-v7a
[INFO]:    hostpython3 has no prebuild_armeabi_v7a, skipping
[INFO]:    Prebuilding libffi for armeabi-v7a
[INFO]:    libffi has no prebuild_armeabi_v7a, skipping
[INFO]:    Applying patches for libffi[armeabi-v7a]
[INFO]:    libffi already patched, skipping
[INFO]:    Prebuilding python3 for armeabi-v7a
[INFO]:    python3 has no prebuild_armeabi_v7a, skipping
[INFO]:    Applying patches for python3[armeabi-v7a]
[INFO]:    python3 already patched, skipping
[INFO]:    Prebuilding setuptools for armeabi-v7a
[INFO]:    setuptools has no prebuild_armeabi_v7a, skipping
[INFO]:    Prebuilding pycparser for armeabi-v7a
[INFO]:    pycparser has no prebuild_armeabi_v7a, skipping
[INFO]:    Prebuilding cffi for armeabi-v7a
[INFO]:    cffi has no prebuild_armeabi_v7a, skipping
[INFO]:    Applying patches for cffi[armeabi-v7a]
[INFO]:    Applying patch disable-pkg-config.patch
[INFO]:    -> running patch -t -d /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi -p1 -i /home/user/pythonforandroid/recipes/cffi/disable-pkg-config.patch
[INFO]:    -> running touch /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi/.patched
[INFO]:    # Building recipes
[INFO]:    Building hostpython3 for armeabi-v7a
[INFO]:    Skipping hostpython3 (3.7.1) build, as it has already been completed
[INFO]:    Building libffi for armeabi-v7a
[INFO]:    libffi said it is already built, skipping
[INFO]:    Building python3 for armeabi-v7a
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/python3-libffi/armeabi-v7a__ndk_target_21/python3/android-build
[WARNING]: Doing some hacky stuff to link properly
[INFO]:    -> running cp /opt/android/android-ndk-r17c/platforms/android-21/arch-arm/usr/lib/crtbegin_so.o ./
[INFO]:    -> running cp /opt/android/android-ndk-r17c/platforms/android-21/arch-arm/usr/lib/crtend_so.o ./
[INFO]:    Activating flags for libffi
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Building setuptools for armeabi-v7a
[INFO]:    setuptools apparently isn't already in site-packages
[INFO]:    Installing setuptools into site-packages
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/setuptools-python3/armeabi-v7a__ndk_target_21/setuptools
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/python-installs/bdisttest_python3_googlendk --install-lib=.
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build --install-lib=Lib/site-packages
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Building pycparser for armeabi-v7a
[INFO]:    pycparser apparently isn't already in site-packages
[INFO]:    Installing pycparser into site-packages
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/pycparser-python3/armeabi-v7a__ndk_target_21/pycparser
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/python-installs/bdisttest_python3_googlendk --install-lib=.
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build --install-lib=Lib/site-packages
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Building cffi for armeabi-v7a
[INFO]:    cffi apparently isn't already in site-packages
[INFO]:    Building compiled components in cffi
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi
[INFO]:    -> running python setup.py clean --all
[INFO]:    -> running python setup.py build_ext -v
[INFO]:    -> running find build/lib.linux-x86_64-3.7 -name "*.o" -exec arm-linux-androideabi-strip --strip-unneeded {} ;
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Installing cffi into site-packages
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/python-installs/bdisttest_python3_googlendk --install-lib=.
[INFO]:    Rebuilding compiled components in cffi
[INFO]:    -> running python setup.py clean --all
[INFO]:    -> running python setup.py build_ext -v
           working: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/home/user/.local/share/python-for-android/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include -I/home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/Include -I/home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o

                                                                                                                                           Traceback (most recent call last):
  File "setup_testapp_python3.py", line 28, in <module>
    package_data={'testapp': ['*.py', '*.png']}
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/user/pythonforandroid/bdistapk.py", line 80, in run
    main()
  File "/home/user/pythonforandroid/toolchain.py", line 1051, in main
    ToolchainCL()
  File "/home/user/pythonforandroid/toolchain.py", line 560, in __init__
    getattr(self, args.subparser_name.replace('-', '_'))(args)
  File "/home/user/pythonforandroid/toolchain.py", line 151, in wrapper_func
    build_dist_from_args(ctx, dist, args)
  File "/home/user/pythonforandroid/toolchain.py", line 192, in build_dist_from_args
    build_recipes(build_order, python_modules, ctx)
  File "/home/user/pythonforandroid/build.py", line 566, in build_recipes
    recipe.build_arch(arch)
  File "/home/user/pythonforandroid/recipe.py", line 890, in build_arch
    self.install_python_package(arch)
  File "/home/user/pythonforandroid/recipe.py", line 863, in install_python_package
    self.install_hostpython_package(arch)
  File "/home/user/pythonforandroid/recipe.py", line 908, in install_hostpython_package
    self.rebuild_compiled_components(arch, env)
  File "/home/user/pythonforandroid/recipe.py", line 917, in rebuild_compiled_components
    *self.setup_extra_args)
  File "/home/user/pythonforandroid/logger.py", line 193, in shprint
    shorten_string(msg, msg_width), width=msg_width))
UnicodeEncodeError: 'ascii' codec can't encode character '\u2018' in position 70: ordinal not in range(128)

The error was also seen on Travis:
https://travis-ci.org/kivy/python-for-android/jobs/487456674

[AndreMiras]

So this is happening because in the Docker environment:

>>> sys.stdout.encoding
'ANSI_X3.4-1968'

I'm wondering why the enforcement was removed in #604 🤔
Anyway it can be fixed by setting the locals in Docker.