Add limited android test support #132

malbarbo · 2017-09-29T13:57:51Z

We could provide android test support by launching an android emulator and running tests in the emulator (like how it is done in libc and rust), but there is licenses issues in redistributing the sdk (see 3.4 of https://developer.android.com/studio/terms.html). Off course, I maybe be wrong about this.

So we download and build bionic libc from source, which I think allows we to distribute the binaries. Having the android libc allows running native binaries using qemu, which can be enough for projects like, nix, ripgrep, etc.

Related #82

japaric

Awesome, @malbarbo.

I think this is a great start towards supporting testing on android targets.

I'm wondering if we can workaround this potential redistribution issue by downloading the SDK on user machine (and cache it somehow) instead of distributing it with our docker images. But that's something we can look into later.

So, I left some questions in the comments. But, other than that this looks good to me. Excellent work as usual. 💯

japaric · 2017-10-02T13:42:46Z

docker/aarch64-linux-android/Dockerfile

@@ -29,5 +35,7 @@ ENV CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc \
    OPENSSL_DIR=/openssl \
    OPENSSL_INCLUDE_DIR=/openssl/include \
    OPENSSL_LIB_DIR=/openssl/lib \
-    RUST_TEST_THREADS=1 \


Is this intentional? Does QEMU have good threading support for android / bionic? Or are all Rust (or bionic?) applications single threaded?

(This comment applies to all the RUST_TEST_THREADS removals)

This was unintentional, sorry.

japaric · 2017-10-02T13:44:13Z

docker/android-system.sh

+    # script ask for it
+    cat << EOF > /usr/bin/java
+#!/bin/bash
+echo "java version \"1.7.0\""


This is an interesting hack. I suppose that all android programs read the java version (e.g. by calling java --version) at runtime or something like that?

Yes, you are right.

japaric · 2017-10-02T13:44:31Z

docker/android-system.sh

+    rm bionic/linker/tests/Android.mk bionic/tests/Android.mk
+
+    source build/envsetup.sh
+    lunch aosp_$arch-user


lunch? not launch? (probably intentional)

lunch is a command of the android build system.

japaric · 2017-10-02T13:49:46Z

docker/i686-linux-android/Dockerfile

@@ -12,22 +12,39 @@ RUN apt-get update && \
 COPY xargo.sh /
 RUN bash /xargo.sh

+# We could supposedly directly run i686 binaries like we do for x86_64, but


This is ... interesting. Any idea why this happen? IIRC, this also happens with the i686-musl target. A panicking program cross compiled for this target will fail in a similar way when executed on x86_64. But I never tried running running that binary under qemu-i386. Cross compiling that binary for i686-gnu and running it on x86_64 works fine though.

Well, it seams I got this wrong. Using qemu is enough to pass the ci/script.sh testing, but it fails in in the ripgrep tests. I was unable to reproduce the i686-musl failure... Running cross test --target i686-unknown-linux-musl in the ripgrep repository worked as expected. I tested a simple program that panics, and it also worked.

Maybe this is a std issue? How should we proceed?

Actually, it seems that unwinding on i686-unknown-linux-musl was fixed a while ago but didn't notice.

i686-unknown-linux-musl is not tested in rust-lang/rust so what you are reporting may indeed be a std bug.

japaric · 2017-10-02T13:52:27Z

docker/i686-linux-android/Dockerfile

 COPY android-ndk.sh /
 RUN bash /android-ndk.sh x86 21
 ENV PATH=$PATH:/android-ndk/bin

+# Build with no-asm to make openssl linked binaries position-independent (PIE)


Interesting finding. I wonder if no-asm would fix OpenSSL builds on i686-musl. IIRC, when cross compiling OpenSSL for i686-musl caused some PIC/PIE errors later on in the linking process.

I've tried and it didn't work...

OK. Doesn't matter; it seems that there may be a solution for that issue in #27 (comment) .

japaric · 2017-10-02T13:59:20Z

docker/qemu.sh

+    # Allow qemu to run android (bionic libc) binaries
+    if [[ "$target" == "android" ]]; then
+      patch -p1 <<'EOF'
+diff -ur qemu-2.10.0/linux-user/elfload.c qemu-2.10.0.new/linux-user/elfload.c


Is this an upstream QEMU commit that was not part of the 2.10.0 release? Or did you come up with this patch yourself? If the latter is there any change you could upstream it. If the former could you include a link to the commit change?

I wrote this patch my self some time ago. I updated qemu to 2.10 to keep it up to date. I thought to send it upstream, but I gave up reading https://wiki.qemu.org/Contribute/SubmitAPatch (Later I found https://wiki.qemu.org/Contribute/TrivialPatches but I was already tired...). I will soon ask in the users list why qemu does not support ioctl with RNDGETENTCNT.

japaric · 2017-10-02T14:04:21Z

docker/qemu.sh


    local arch=$1 \
+          target=$2 \


maybe this should be named os?

Much better!

japaric · 2017-10-02T14:05:10Z

.travis.yml

@@ -41,7 +41,7 @@ matrix:

    # BSD
    - env: TARGET=i686-unknown-freebsd                  DYLIB=1 STD=1 OPENSSL=0.7.17
-    - env: TARGET=x86_64-unknown-dragonfly               DYLIB=1       OPENSSL=0.5.5


nit: alignment regression :-)

This is a fix, isn't it?

Also fix openssl arch for armv7-linux-androideabi

malbarbo · 2017-10-02T21:37:32Z

I'm wondering if we can workaround this potential redistribution issue by downloading the SDK on user machine (and cache it somehow) instead of distributing it with our docker images.

An alternative is to build the docker image in the user machine. In a more general away, we can pack all Docker files inside the cross binary and give the user the option of downloading the pre-build docker image or build it locally. We can also let the user specify a Docker file.

malbarbo · 2017-10-10T17:54:47Z

I submitted the qemu patch upstream.

Running ripgrep tests in an i686 android emulator works as expected. So I think std is correct. Anyway, the error seems only to happen when Command is used.

@japaric Anything else? Can we test this?

japaric · 2017-10-20T17:35:23Z

Sorry for the delay @malbarbo. Could you please add:

A note about testing support being limited for these target. Noting that Command doesn't work, for example, would be great.
Add the _RUNNER variable to the Dockerfiles so that these targets work without binfmt-misc support (which requires sudo). You can find an example of that variable here.

After that you can r+ this.

@homunkulus delegate+

homunkulus · 2017-10-20T17:35:25Z

✌️ @malbarbo can now approve this pull request

malbarbo · 2017-10-20T18:55:46Z

Thanks @japaric!

@homunkulus +r

malbarbo · 2017-10-20T18:56:43Z

Ops, let's try again.

@homunkulus r+

homunkulus · 2017-10-20T18:56:44Z

📌 Commit 140aff3 has been approved by malbarbo

Add limited android test support We could provide android test support by launching an android emulator and running tests in the emulator (like how it is done in libc and rust), but there is licenses issues in redistributing the sdk (see 3.4 of https://developer.android.com/studio/terms.html). Off course, I maybe be wrong about this. So we download and build bionic libc from source, which I think allows we to distribute the binaries. Having the android libc allows running native binaries using qemu, which can be enough for projects like, nix, ripgrep, etc. Related #82

homunkulus · 2017-10-20T18:56:51Z

⌛ Testing commit 140aff3 with merge 3de557c...

homunkulus · 2017-10-20T21:00:13Z

☀️ Test successful - status-travis
Approved by: malbarbo
Pushing 3de557c to master...

malbarbo added 2 commits September 29, 2017 09:43

Add android test support

cff2ebc

Enable android tests on travis and update README

03d0187

japaric reviewed Oct 2, 2017

View reviewed changes

malbarbo added 2 commits October 2, 2017 17:55

Add info about the qemu android patch

5d8d75b

Run android tests with one thread (in qemu)

dd33f39

Also fix openssl arch for armv7-linux-androideabi

Update qemu android patch

44aa61a

malbarbo mentioned this pull request Oct 20, 2017

Some tests fails on i686-linux-android #140

Closed

malbarbo added 2 commits October 20, 2017 16:51

Add missing cargo runners for some android targets

4aed911

Add a note about failing tests on i686-linux-android

140aff3

homunkulus merged commit 140aff3 into cross-rs:master Oct 20, 2017

homunkulus mentioned this pull request Oct 20, 2017

Add armv5 support #113

Closed

malbarbo mentioned this pull request Oct 27, 2017

Fix android ci rust-lang/libc#825

Merged

jamesmunns mentioned this pull request Mar 17, 2018

Add full android test support #82

Closed

malbarbo deleted the android-test branch October 16, 2018 13:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add limited android test support #132

Add limited android test support #132

malbarbo commented Sep 29, 2017

japaric left a comment

japaric Oct 2, 2017

malbarbo Oct 2, 2017

japaric Oct 2, 2017

malbarbo Oct 2, 2017

japaric Oct 2, 2017

malbarbo Oct 2, 2017

japaric Oct 2, 2017

malbarbo Oct 2, 2017

japaric Oct 3, 2017

japaric Oct 2, 2017

malbarbo Oct 2, 2017

japaric Oct 3, 2017 •

edited

Loading

japaric Oct 2, 2017

malbarbo Oct 2, 2017

japaric Oct 2, 2017

malbarbo Oct 2, 2017

japaric Oct 2, 2017

malbarbo Oct 2, 2017

malbarbo commented Oct 2, 2017

malbarbo commented Oct 10, 2017

japaric commented Oct 20, 2017

homunkulus commented Oct 20, 2017

malbarbo commented Oct 20, 2017

malbarbo commented Oct 20, 2017

homunkulus commented Oct 20, 2017

homunkulus commented Oct 20, 2017

homunkulus commented Oct 20, 2017

Add limited android test support #132

Add limited android test support #132

Conversation

malbarbo commented Sep 29, 2017

japaric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

japaric Oct 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

malbarbo commented Oct 2, 2017

malbarbo commented Oct 10, 2017

japaric commented Oct 20, 2017

homunkulus commented Oct 20, 2017

malbarbo commented Oct 20, 2017

malbarbo commented Oct 20, 2017

homunkulus commented Oct 20, 2017

homunkulus commented Oct 20, 2017

homunkulus commented Oct 20, 2017

japaric Oct 3, 2017 •

edited

Loading