Attempts to mprotect() with MAP_JIT failing on Apple Silicon as of macOS 11.2

[logancollins]

We are using the PCRE2 regular expression library (version 10.36, and by extension, SLJIT), and have noticed a change in behavior after macOS 11.2 came out. Now, any attempt to allocate memory with SLJIT fails. It's okay for us, as PCRE2 can fall back to not using JIT for an expression. However, I wanted to dig in and figure out why it was suddenly failing.

It turns out that, with sljitExecAllocator.c, the mmap() with MAP_JIT call is succeeding, but the subsequent call to mprotect() (on line 185) fails with a generic "permission denied" error from the system. This was working in macOS 11.0 and 11.1, but something changed with the 11.2 update. After spending a day not really being able to tell "why", we filed a feedback issue with Apple DTS.

Apple DTS provided the following response:

I think the documentation could state this more clearly, but I think the
basic answer here is that this configuration:

const int prot = PROT_READ | PROT_WRITE | PROT_EXEC;

...is inherently invalid on Apple Silicon. That is, it is inherently
invalid to mark a page "rwx". The documentation doesn't QUITE say that,
but you can see the implications inside this article:

https://developer.apple.com/documentation/apple_silicon/porting_just-in-time_compilers_to_apple_silicon?language=objc

"When memory protection is enabled, a thread cannot write to a memory
region and execute instructions in that region at the same time. Apple
silicon enables memory protection for all apps, regardless of whether
they adopt the Hardened Runtime."

"Each thread has its own set of access permissions for a given memory
region. When you call mmap to create the memory region, the system
initially configures it as readable and executable (R-X) for all
threads. Calling pthread_jit_write_protect_np with the value false
offers a secure and efficient way to simultaneously remove the
executable permission and add the writable permission (RW-) for the
current thread."

"Important
Because pthread_jit_write_protect_np changes only the current thread’s
permissions, avoid accessing the same memory region from multiple
threads. Giving multiple threads access to the same memory region opens
up a potential attack vector, in which one thread has write access and
another has executable access to the same region."

If you read between the lines there, particularly with the last
"important" block, the underlying issue here is that you simply CAN'T*
have a "rwx" page on Apple Silicon. I'm not sure what your mprotect
call was doing pre-11.2, but I don't think it was behaving correctly.

*You might ask how Rosetta supports apps that DO rely on RWX on the same
thread, and the answer is basically that it toggles between RW/RX on the
"fly" based on what's actually happening. In rough terms, you can
basically think of it as catching the mach exception generated by being
the wrong state and then toggling the page state to the required state.
Not something I would recommend trying to implement.

Long story short, I think my recommendation here would basically be to
pthread_jit_write_protect_np instead of mprotect when working with
MAP_JIT pages on Apple Silicon.

We already apply the pthread_jit_write_protect_np() calls before and after invoking PCRE2 to compile a regex. But, it sounds like they are saying that SLJIT needs to not call mprotect() when on Apple Silicon no matter what.

We're planning on making the relevant change in our copy of PCRE2 and SLJIT, but I wanted to bring this up to you in case you have other consumers of the library hit this issue in the future.

Thank you for your time!

[zherczeg]

I don't know much about macs, but if you submit a patch, and people are happy with it, I am gladly accept your contribution.

[carenas]

there is no need to run PCRE under rosetta when there is support for it to run in Apple Silicon natively if you apply #90; agree though that we should improve the detection code as well, but as you had noticed macOS is a moving target (even making API breaking changes within minor versions) and we haven't been given access to M1 hardware

[logancollins]

Thanks for the note! However, we are running it under Apple Silicon natively (we aren't using Rosetta). I will review that pull request to see if it improves things, but it was all calls to mprotect() that were failing outright always under Apple Silicon (line 185 in the pull request's updated code).

I'm happy to put a merge request together with anything that might be necessary for support under macOS 11.2.

[carenas]

FWIW, the mprotect() test is doing EXACTLY what it was designed to do; and it is to detect if the OS is lying to us and will otherwise segfault when we write/exec it because the page was not marked writable/executable (even if mmap returned without error).

you can find more details in the discusion in the downstream PCRE bug:

https://bugs.exim.org/show_bug.cgi?id=2618

note that for macOS mprotect would normally fail for Apple Silicon for the reason you explained starting with 11.2, but can also fail in x86 depending on the way the library was compiled and the entitlements the application using it might had been given.

the issue predates Big Sur an it has been around since the last versions of 10.13 when MAP_JIT made its undocumented release and as explained in:

https://bugs.exim.org/show_bug.cgi?id=2334

[logancollins]

Completely understandable! I definitely don't know much about how all of this works. I'm just coming through with the information we've gathered and what Apple was willing to send along. Thanks for referencing everything and your continued help in this!

[carenas]

it would seem this is because macOS not failing the mprotect call when requested an RWX that was silently converted to RX by mmap was considered a "bug" and so now there is at least a way to detect it was lying and would had crashed (as it still does at least in HardenedBSD).

an alternative (which shouldn't be used unless there is really no better option) and that wastes a lot of memory is to use the alternative WX allocator (-DSLJIT_WX_EXECUTABLE_ALLOCATOR) that should be also available starting with PCRE 10.36 using the corresponding configure/cmake flag.