aarch64 AOT module fails to load on android

[zoraaver]

When trying to load an aarch64 AOT file on android, the module fails to load due to an overflow check failing, AOT module load failed: target address out of range..

Expected behaviour

aarch64 AOT module loads without error

Actual behaviour

aarch64 AOT module fails to load

Steps to reproduce

I've attached the example android studio project which reproduces the issue consistently on an arm64 emulator:
WasmAOT.zip

C++ code to load the AOT module:

#include <android/asset_manager_jni.h>
#include <android/log.h>
#include <jni.h>

#include <cstring>
#include <string>
#include <vector>

#include "wasm_export.h"

extern "C" JNIEXPORT jstring JNICALL
Java_com_example_wasmaot_MainActivity_stringFromJNI(
        JNIEnv* env,
        jobject /* this */) {
    std::string hello = "Hello from wasm aot example";
    return env->NewStringUTF(hello.c_str());
}

extern "C" JNIEXPORT jint JNICALL
Java_com_example_wasmaot_MainActivity_loadWasmAOTModule(
        JNIEnv* env,
        jobject /* this */,
        jobject jasset_manager) {
    auto asset_manager = AAssetManager_fromJava(env, jasset_manager);

    auto aot_file_name = std::string("wamr_aot_test_") + AOT_FILE_SUFFIX + std::string(".aot");

    __android_log_print(ANDROID_LOG_INFO,"example.wasmaot", "%s", ("Loading AOT file " + aot_file_name).c_str());

    auto asset = AAssetManager_open(asset_manager, aot_file_name.c_str(), AASSET_MODE_BUFFER);

    if (!asset) {
        __android_log_print(ANDROID_LOG_ERROR,"example.wasmaot", "Failed to open AOT asset");
        return 1;
    }

    auto size = AAsset_getLength(asset);
    auto asset_buffer = AAsset_getBuffer(asset);

    std::vector<uint8_t> buf;
    buf.reserve(size);
    buf.assign(static_cast<const uint8_t*>(asset_buffer), static_cast<const uint8_t*>(asset_buffer) + size);

    AAsset_close(asset);

    wasm_runtime_init();

    char error_buf[128] = {};

    auto module = wasm_runtime_load(buf.data(), size, error_buf, sizeof(error_buf));

    if (module) {
        wasm_runtime_unload(module);
    } else {
        __android_log_print(ANDROID_LOG_ERROR,"example.wasmaot" ,"%s", error_buf);
        return 1;
    }

    wasm_runtime_destroy();

    return 0;
}

This is the rust code for the AOT module:

use std::os::raw::c_char;

#[no_mangle]
pub extern "C" fn main(_argv: *const *const c_char, _argc: i32) -> i32 {
    let mut a = 0;
    for i in 0..10 {
        a += i;
    }

    let mut buffer = Vec::new();

    buffer.extend([1, 2, 3]);

    println!("Hello world {:?}", buffer);
    println!("Hello world {:?}", a);

    0
}

which was compiled via cargo build --target=wasm32-wasi and

wamrc \
    --disable-simd \
    --target=aarch64 \
    --size-level=3 \
    -o "wamr_aot_test_aarch64.aot" \
    "hello_world.wasm"

The same rust code, when compiled for armv7 loads without issue on an armv7 android build.

[wenyongh]

Hi, could you try using --size-level=1 or --size-level=2 for wamrc?

[zoraaver]

Hi, thanks for the quick response. I get the following error when trying to compile with --size-level=1 or --size-level=2:
LLVM ERROR: Only small, tiny and large code models are allowed on AArch64.

However, passing --size-level=0 works and the resulting AOT file is successfully loaded. Is this expected?

[wenyongh]

Yes, it is expected, I have a test, LLVM AArch64 does not support code model medium (i.e. --size-level=1) and code model kernel (i.e. --size-level=2). Code model large (i.e. --size-level=0) is supported.

[zoraaver]

Sorry, I should have been more clear. I meant is it expected that the AOT module fails to load when size-level is set to 3?

[zoraaver]

Also, is there any way to estimate how much the size of the AOT file would increase (relatively) changing the size-level from 3 to 0?

[wenyongh]

Hi, I tested some cases, it seems the size increased is not very much (bytes):

            size-level 3    size-level 0
CoreMark:   64276      64680
dhrystone:  64020      64576
gcc-loops:  419668     421560
hashset:    847616     847964
quicksort:  15292      15508

You can also test these cases under <wamr_root>/tests/benchmarks.

[zoraaver]

Thanks, that's good to know. I'm still a bit concerned that the AOT module fails to load when size-level=3, is there any possible explanation for this behaviour?

[wenyongh]

It may be due to that size-level=3 generates some relocation types which require the address of target symbol is in range 0-2GB. Could you try the following patch:

diff --git a/core/iwasm/aot/aot_loader.c b/core/iwasm/aot/aot_loader.c
index 480a00b9..84d73497 100644
--- a/core/iwasm/aot/aot_loader.c
+++ b/core/iwasm/aot/aot_loader.c
@@ -1475,11 +1475,7 @@ load_object_data_sections(const uint8 **p_buf, const uint8 *buf_end,
     /* Create each data section */
     for (i = 0; i < module->data_section_count; i++) {
         int map_prot = MMAP_PROT_READ | MMAP_PROT_WRITE;
-#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_AMD_64) \
-    || defined(BUILD_TARGET_RISCV64_LP64D)                       \
-    || defined(BUILD_TARGET_RISCV64_LP64)
-        /* aot code and data in x86_64 must be in range 0 to 2G due to
-           relocation for R_X86_64_32/32S/PC32 */
+#if UINTPTR_MAX == UINT64_MAX
         int map_flags = MMAP_MAP_32BIT;
 #else
         int map_flags = MMAP_MAP_NONE;
@@ -3021,11 +3017,7 @@ create_sections(AOTModule *module, const uint8 *buf, uint32 size,
                 if ((section_size > 0) && !module->is_indirect_mode) {
                     int map_prot =
                         MMAP_PROT_READ | MMAP_PROT_WRITE | MMAP_PROT_EXEC;
-#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_AMD_64) \
-    || defined(BUILD_TARGET_RISCV64_LP64D)                       \
-    || defined(BUILD_TARGET_RISCV64_LP64)
-                    /* aot code and data in x86_64 must be in range 0 to 2G due
-                       to relocation for R_X86_64_32/32S/PC32 */
+#if UINTPTR_MAX == UINT64_MAX
                     int map_flags = MMAP_MAP_32BIT;
 #else
                     int map_flags = MMAP_MAP_NONE;

[zoraaver]

I'm still getting the same error AOT module load failed: target address out of range. I'll try to investigate further.

[bianchui]

I run into this issue too.
In my case, there is a check in R_AARCH64_ADR_PREL_PG_HI21, because of aot_module->data_sections->data and aot_module->code both mmaped memory and distance maybe larger than 4GB.
I'm trying to merge both mmap into one large mmap so their distance will small enough.

[bianchui]

@zoraaver Would you do a test to see that fix your issue too?

[bianchui]

@wenyongh i see the branch is mark as merged and my commit is totally lost in the final history
and issues also not mark as fixed
is that intent?

should i fire another pr to merge the final change?

[wenyongh]

@bianchui the code changes had been merged into main branch, but I chose the Squash and Merge mode when merging to main, so the commit logs are lost, please don't create another PR to merge again. I usually do that way because there may be lots of commit logs in the development branch (e.g. a lot of commits/fixes when developing a feature, merging them all to main makes main looks a little ugly), do you need to merge branch dev/merge_aot_data_text to main again to keep your commit log?

BTW, could you test this issue again? If it is fixed, maybe we can mark this issue as fixed and close it?

[wenyongh]

@bianchui I merged branch dev/merge_aot_data_text into main again without squash, now your commit can be found in the main branch:

please have a check, thanks.

[bianchui]

BTW, could you test this issue again? If it is fixed, maybe we can mark this issue as fixed and close it?

@wenyongh fix this issue was the original reason for merging text and data
and i do the test, it has been fixed
the probability of failure before and after is 15% & 0% of nearly 1million devces.

and thank you for your reply and the merge

[wenyongh]

Welcome. And let's close this issue as it is fixed.

[wenyongh]

Glad to know that WAMR is tested in nearly 1million devices, it is cool!