AOT compiled WASM couldn't be loaded on macOS M1 environment

[cosmo0920]

Step to replicates

1. Build ./sample/basic WASM application
2. Move to wamr-compiler and build wamrc
3. Built AOT object from ./out/wasm-apps/testapp.wasm with wamrc:

/path/to/wamr-compiler/build/wamrc --cpu=apple-m1 --size-level=3 -o ./out/wasm-apps/testapp.aot ./out/wasm-apps/testapp.wasm
Create AoT compiler with:
  target:        arm64
  target cpu:    apple-m1
  cpu features:  
  opt level:     3
  size level:    3
  output format: AoT file
Compile success, file ./out/wasm-apps/testapp.aot was generated.

4. Execute basic sample application with AOT object.
5. Then, 💥!

$  ./out/basic -f out/wasm-apps/testapp.aot
Load wasm module failed. error: AOT module load failed: mmap memory failed

Environment

Software versions:

$ sw_vers
ProductName:	macOS
ProductVersion:	12.5
BuildVersion:	21G72

Host Compiler version:

$ clang --version
Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

WASI SDK's compiler version:

$ ~/work/wasi-sdk-16.0/bin/clang --version
clang version 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d054140bf2556d4c4d)
Target: wasm32-unknown-wasi
Thread model: posix
InstalledDir: ~/work/wasi-sdk-16.0/bin

[cosmo0920]

I tried to take a look on M1 Mac issue and I found the several issues before calling WASM function part:

1. With macOS arm64 runtime, it is always hardened. We shouldn't use PROT_EXEC flags in mmap. Otherwise, mmap is failed.

diff --git a/core/shared/platform/common/posix/posix_memmap.c b/core/shared/platform/common/posix/posix_memmap.c
index 2dfbee4..1de331e 100644
--- a/core/shared/platform/common/posix/posix_memmap.c
+++ b/core/shared/platform/common/posix/posix_memmap.c
@@ -16,6 +16,12 @@ static size_t total_size_munmapped = 0;
 
 #define HUGE_PAGE_SIZE (2 * 1024 * 1024)
 
+#if defined(__APPLE__) && defined(__arm64__)
+#define EXEC_FLAGS 0
+#else
+#define EXEC_FLAGS PROT_EXEC
+#endif
+
 #if !defined(__APPLE__) && !defined(__NuttX__) && defined(MADV_HUGEPAGE)
 static inline uintptr_t
 round_up(uintptr_t v, uintptr_t b)
@@ -66,7 +72,7 @@ os_mmap(void *hint, size_t size, int prot, int flags)
         map_prot |= PROT_WRITE;
 
     if (prot & MMAP_PROT_EXEC)
-        map_prot |= PROT_EXEC;
+        map_prot |= EXEC_FLAGS;
 
 #if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_AMD_64)
 #ifndef __APPLE__
@@ -243,7 +249,7 @@ os_mprotect(void *addr, size_t size, int prot)
         map_prot |= PROT_WRITE;
 
     if (prot & MMAP_PROT_EXEC)
-        map_prot |= PROT_EXEC;
+        map_prot |= EXEC_FLAGS;
 
     return mprotect(addr, request_size, map_prot);
 }

2. Default target name of M1 macOS is arm64 not aarch64v8

diff --git a/core/iwasm/aot/arch/aot_reloc_aarch64.c b/core/iwasm/aot/arch/aot_reloc_aarch64.c
index 4c46eaa..dc3cb8f 100644
--- a/core/iwasm/aot/arch/aot_reloc_aarch64.c
+++ b/core/iwasm/aot/arch/aot_reloc_aarch64.c
@@ -53,7 +53,11 @@ get_target_symbol_map(uint32 *sym_num)
     return target_sym_map;
 }
 
+#if defined(__APPLE__) && defined(__arm64__)
+#define BUILD_TARGET_AARCH64_DEFAULT "arm64"
+#else
 #define BUILD_TARGET_AARCH64_DEFAULT "aarch64v8"
+#endif
 void
 get_current_target(char *target_buf, uint32 target_buf_size)
 {

[cosmo0920]

With the above change, I still got bus error:

$ lldb --  ./out/basic -f out/wasm-apps/testapp.aot
(lldb) target create "./out/basic"
Current executable set to '/Users/cosmo/GitHub/wasm-micro-runtime/samples/basic/out/basic' (arm64).
(lldb) settings set -- target.run-args  "-f" "out/wasm-apps/testapp.aot"
(lldb) r
Process 27342 launched: '/Users/cosmo/GitHub/wasm-micro-runtime/samples/basic/out/basic' (arm64)
Process 27342 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x1002dc004)
    frame #0: 0x00000001002dc004
->  0x1002dc004: str    d8, [sp, #-0x70]!
    0x1002dc008: stp    x29, x30, [sp, #0x10]
    0x1002dc00c: stp    x28, x27, [sp, #0x20]
    0x1002dc010: stp    x26, x25, [sp, #0x30]
Target 0: (basic) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x1002dc004)
  * frame #0: 0x00000001002dc004
    frame #1: 0x000000010001c224 basic`call_func + 8
    frame #2: 0x0000000100030fb0 basic`invoke_native_with_hw_bound_check + 444
    frame #3: 0x0000000100030bc0 basic`aot_call_function + 420
    frame #4: 0x00000001000199a4 basic`wasm_runtime_call_wasm_a + 740
    frame #5: 0x0000000100001288 basic`main + 612
    frame #6: 0x00000001000d508c dyld`start + 520

[eloparco]

Hello, I'm getting the same error on MacOS M1, did you manage to find a solution?

[cosmo0920]

No, I didn't. Currently, I disable to use AOT execution on M1 Mac.

[eloparco]

@wenyongh Any hints on what changes would be needed to use AOT on M1 macs?
After reading the docs, I was changing

diff --git a/core/iwasm/aot/aot_loader.c b/core/iwasm/aot/aot_loader.c
index b5fa33c0..854453ad 100644
--- a/core/iwasm/aot/aot_loader.c
+++ b/core/iwasm/aot/aot_loader.c
@@ -3232,6 +3232,7 @@ aot_load_from_aot_file(const uint8 *buf, uint32 size, char *error_buf,
     if (!module)
         return NULL;
 
+    pthread_jit_write_protect_np(false);
     if (!load(buf, size, module, error_buf, error_buf_size)) {
         aot_unload(module);
         return NULL;
diff --git a/core/iwasm/aot/arch/aot_reloc_aarch64.c b/core/iwasm/aot/arch/aot_reloc_aarch64.c
index 4c46eaa0..10ec3f5a 100644
--- a/core/iwasm/aot/arch/aot_reloc_aarch64.c
+++ b/core/iwasm/aot/arch/aot_reloc_aarch64.c
@@ -53,7 +53,12 @@ get_target_symbol_map(uint32 *sym_num)
     return target_sym_map;
 }
 
+#if defined(__APPLE__) && defined(__arm64__)
+#define BUILD_TARGET_AARCH64_DEFAULT "arm64"
+#else
 #define BUILD_TARGET_AARCH64_DEFAULT "aarch64v8"
+#endif
+
 void
 get_current_target(char *target_buf, uint32 target_buf_size)
 {
diff --git a/core/shared/platform/common/posix/posix_memmap.c b/core/shared/platform/common/posix/posix_memmap.c
index 2dfbee45..9c403d66 100644
--- a/core/shared/platform/common/posix/posix_memmap.c
+++ b/core/shared/platform/common/posix/posix_memmap.c
@@ -36,7 +36,7 @@ void *
 os_mmap(void *hint, size_t size, int prot, int flags)
 {
     int map_prot = PROT_NONE;
-    int map_flags = MAP_ANONYMOUS | MAP_PRIVATE;
+    int map_flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_JIT;
     uint64 request_size, page_size;
     uint8 *addr = MAP_FAILED;
     uint32 i;

And with that I managed fix some problems and get farther in the execution, but I still get

thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x100430008)
  * frame #0: 0x0000000100430008
    frame #1: 0x000000010006f5f0 iwasm`invoke_native_with_hw_bound_check(exec_env=0x0000000103600800, func_ptr=0x0000000100430008, func_type=0x0000000102400590, signature=0x0000000000000000, attachment=0x0000000000000000, argv=0x0000000000000000, argc=0, argv_ret=0x0000000000000000) at aot_runtime.c:1392:19
    frame #2: 0x000000010006f160 iwasm`aot_call_function(exec_env=0x0000000103600800, function=0x000000016fdfdc80, argc=0, argv=0x0000000000000000) at aot_runtime.c:1601:15
    frame #3: 0x000000010006e7b4 iwasm`execute_post_instantiate_functions(module_inst=0x0000000102703380, is_sub_inst=false, exec_env_main=0x0000000000000000) at aot_runtime.c:1022:14
    frame #4: 0x000000010006d0b8 iwasm`aot_instantiate(module=0x0000000101ed0a40, parent=0x0000000000000000, exec_env_main=0x0000000000000000, stack_size=65536, heap_size=0, error_buf="", error_buf_size=128) at aot_runtime.c:1231:10
    frame #5: 0x00000001000372a0 iwasm`wasm_runtime_instantiate_internal(module=0x0000000101ed0a40, parent=0x0000000000000000, exec_env_main=0x0000000000000000, stack_size=65536, heap_size=0, error_buf="", error_buf_size=128) at wasm_runtime_common.c:1340:44
    frame #6: 0x0000000100037314 iwasm`wasm_runtime_instantiate(module=0x0000000101ed0a40, stack_size=65536, heap_size=0, error_buf="", error_buf_size=128) at wasm_runtime_common.c:1354:12
    frame #7: 0x0000000100006d3c iwasm`main(argc=1, argv=0x000000016fdfecc8) at main.c:933:15
    frame #8: 0x000000019a953f28 dyld`start + 2236

I tried adding the address sanitizer to get more info:

./iwasm -v=5 wasm-apps/no_pthread.aot
[5/5] Linking C executable iwasm
[10:45:10:477 - 1F610DB40]: Found malloc function, name: malloc, index: 17
[10:45:10:626 - 1F610DB40]: Found free function, name: free, index: 19
[10:45:10:632 - 1F610DB40]: Load module success.

[10:45:10:654 - 1F610DB40]: Memory instantiate:
[10:45:10:656 - 1F610DB40]:   page bytes: 65536, init pages: 1, max pages: 30
[10:45:10:658 - 1F610DB40]:   data offset: 5808, stack size: 32768
[10:45:10:660 - 1F610DB40]:   heap offset: 65536, heap size: 0

AddressSanitizer:DEADLYSIGNAL
=================================================================
==37835==ERROR: AddressSanitizer: BUS on unknown address (pc 0x000100618008 bp 0x00016fc19800 sp 0x00016fc195d0 T0)
==37835==The signal is caused by a UNKNOWN memory access.
==37835==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x100618008  (<unknown module>)
    #1 0x1002535ec in invoke_native_with_hw_bound_check aot_runtime.c:1392
    #2 0x10025315c in aot_call_function aot_runtime.c:1601
    #3 0x1002527b0 in execute_post_instantiate_functions aot_runtime.c:1022
    #4 0x1002510b4 in aot_instantiate aot_runtime.c:1231
    #5 0x10021b29c in wasm_runtime_instantiate_internal wasm_runtime_common.c:1340
    #6 0x10021b310 in wasm_runtime_instantiate wasm_runtime_common.c:1354
    #7 0x1001ead38 in main main.c:933
    #8 0x19a953f24  (<unknown module>)

==37835==Register values:
 x[0] = 0x0000000103600800   x[1] = 0x0000000000000000   x[2] = 0x0000000000000000   x[3] = 0x0000000000000000  
 x[4] = 0x0000000000000000   x[5] = 0x0000000000000000   x[6] = 0x0000000000000000   x[7] = 0x0000000000000000  
 x[8] = 0x0000000100221cf0   x[9] = 0x0000000000000000  x[10] = 0x0000000000000000  x[11] = 0x0000000000000001  
x[12] = 0x0000000000000000  x[13] = 0xffffff8fd205cd1f  x[14] = 0x0000000000007e01  x[15] = 0x0000000000000006  
x[16] = 0x000000019acda3d0  x[17] = 0x00000001006b45e0  x[18] = 0x0000000000000000  x[19] = 0x0000000100618008  
x[20] = 0x000000010021f1b0  x[21] = 0x0000000000000000  x[22] = 0x000000016fc195d0  x[23] = 0x000000019a9ce366  
x[24] = 0x000000016fc1a9d0  x[25] = 0x0000000000000001  x[26] = 0x0000000000000000  x[27] = 0x0000000000000000  
x[28] = 0x0000000000000000     fp = 0x000000016fc19800     lr = 0x0000000100221d74     sp = 0x000000016fc195d0  
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: BUS (<unknown module>) 
==37835==ABORTING
[1]    37835 abort      ./iwasm -v=5 wasm-apps/no_pthread.aot

I don't know much about AOT, so I would appreciate any inputs or suggestions on what to try next.

[eloparco]

Managed to get it to work, opening a PR

[cosmo0920]

Seems to be resolved in #2618.