32-bit arm-linux-android compilation for OpenCL broken after PR #4657 ([CodeGen] Generate blob use LLVM directly)

[trevor-m]

I have the script below which I was using to compile a mobilenet model for an android target with a 32-bit ARM processor and a Mali gpu, using a much older commit of TVM.
When I updated to a newer commit of TVM, I will get this error during compilation:

RuntimeError: Compilation error:
Android (6317467 based on r365631c1) clang version 9.0.8 (https://android.googlesource.com/toolchain/llvm-project e0caee08e5f09b374a27a676d04978c81fcb1928) (based on LLVM 9.0.8svn)
Target: armv7a-unknown-linux-android27
Thread model: posix
InstalledDir: /data/standalone-ndk-r21b/bin
Found candidate GCC installation: /data/standalone-ndk-r21b/bin/../lib/gcc/arm-linux-androideabi/4.9.x
Selected GCC installation: /data/standalone-ndk-r21b/bin/../lib/gcc/arm-linux-androideabi/4.9.x
Candidate multilib: thumb;@mthumb
Candidate multilib: armv7-a;@march=armv7-a
Candidate multilib: armv7-a/thumb;@march=armv7-a@mthumb
Candidate multilib: .;
Selected multilib: armv7-a;@march=armv7-a
Found CUDA installation: /usr/local/cuda-10.0, version 10.0
 "/data/standalone-ndk-r21b/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld" --sysroot=/data/standalone-ndk-r21b/sysroot -z noexecstack -EL --warn-shared-textrel -z now -z relro -X --hash-style=gnu --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -shared -o compiled.so /data/standalone-ndk-r21b/sysroot/usr/lib/arm-linux-androideabi/27/crtbegin_so.o -L/data/standalone-ndk-r21b/lib64/clang/9.0.8/lib/linux/arm -L/data/standalone-ndk-r21b/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a -L/data/standalone-ndk-r21b/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a -L/data/standalone-ndk-r21b/sysroot/usr/lib/arm-linux-androideabi/27 -L/data/standalone-ndk-r21b/sysroot/usr/lib/arm-linux-androideabi -L/data/standalone-ndk-r21b/sysroot/usr/lib/../lib -L/data/standalone-ndk-r21b/sysroot/usr/lib/arm-linux-androideabi/../../lib -L/data/standalone-ndk-r21b/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a -L/data/standalone-ndk-r21b/sysroot/usr/lib /tmp/tmp6x5vf9t2/lib0.o /tmp/tmp6x5vf9t2/devc.o -Bstatic -lc++ -Bdynamic -lm -lgcc -ldl -lc -lgcc -ldl /data/standalone-ndk-r21b/sysroot/usr/lib/arm-linux-androideabi/27/crtend_so.o
/data/standalone-ndk-r21b/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: /tmp/tmp6x5vf9t2/devc.o uses VFP register arguments, output does not
clang90++: error: linker command failed with exit code 1 (use -v to see invocation)

I did a binary search on the TVM commits to find when this error was introduced, and I traced it down to the following PR. #4657 Before this PR, my script worked fine. After this PR, I get the error above.

Question: Is there is a bug introduced by that PR? If not, how should my compilation script be updated? I have tried many, many combinations of compiler flags (-mfloat-abi=hard), usingndk.create_shared, etc and could not find a solution.

Also, this only happens with the OpenCL target. If I set the target to LLVM (same as target_host), compilation succeeds with and without #4657.

import tvm
from tvm import relay
import mxnet
from mxnet.gluon.model_zoo.vision import get_model
import gluoncv
import os

block = get_model('mobilenetv2_1.0', pretrained=True)
input_shape = (1, 3, 224, 224)
mod, params = relay.frontend.from_mxnet(block, shape={'data': input_shape}, dtype='float32')

# Compile
target = tvm.target.mali()
target_host = "llvm -target=arm-linux-android -mattr=+neon -mfloat-abi=soft"
with relay.build_config(opt_level=3):
    graph, lib, params = relay.build(mod, params=params, target=target, target_host=target_host)

# Save
with open('compiled.json', 'w') as f_graph_json:
    f_graph_json.write(graph)
with open('compiled.params', 'wb') as f_params:
    f_params.write(relay.save_param_dict(params))
cc = "/data/standalone-ndk-r21b/bin/armv7a-linux-androideabi27-clang++"
options = ["-v", "-static-libstdc++", "--sysroot", "/data/standalone-ndk-r21b/sysroot"]
lib.export_library('compiled.so', cc=cc, options=options)

[tqchen]

Thanks @trevor-m , please open a new troubleshooting thread on https://discuss.tvm.ai/

[trevor-m]

I found the source of the bug: https://discuss.tvm.ai/t/opencl-target-for-32-bit-arm-linux-android-broken-after-pr-4657/7252/4?u=trevor-m