mmap() fails on rpi5, but same call succeeds on x86_64

[andrewbird]

Describe the bug

I'm trying to run Dosemu2 on raspberry pi 5 with latest up-to-date RaspiOS. I've created all the packages required but it fails during startup due to an mmap() failure. I have been in contact with the Dosemu2 developer and he has located the failure to a single mmap() call. He's produced a simple test case that succeeds on x86_64, but fails on arm64 of rpi 5.

Steps to reproduce the behaviour

Use this test program

ajb@raspberrypi:/local/src/dosemu2.git $ cat mprot.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>

int main(void)
{
    int err;
    char buf[1024];
    char *addr2;
    char *addr = mmap(NULL, 4096*3, PROT_READ | PROT_WRITE,
            MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (addr == MAP_FAILED) {
        perror("mmap()");
        return 1;
    }
    printf("mapped %p\n", addr);
    addr2 = addr + 4096;
    err = mprotect(addr2, 4096, PROT_NONE);
    if (err) {
        perror("mprotect()");
        return 1;
    }
    printf("mprotect %p worked\n", addr2);
    snprintf(buf, sizeof(buf), "cat /proc/%i/maps", getpid());
    system(buf);
    return 0;
}

compile and run

ajb@raspberrypi:/local/src/dosemu2.git $ gcc -o mprot mprot.c
ajb@raspberrypi:/local/src/dosemu2.git $ ./mprot
mapped 0x7fff9d664000
mprotect(): Invalid argument

compare against same program run on x86_64 (Linux calypso 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux)

ajb@calypso:~/Desktop$ gcc -o mprot mprot.c
ajb@calypso:~/Desktop$ ./mprot
mapped 0x795b20a98000
mprotect 0x795b20a99000 worked
63f12acef000-63f12acf0000 r--p 00000000 00:4c 6689407                    /home/ajb/Desktop/mprot
63f12acf0000-63f12acf1000 r-xp 00001000 00:4c 6689407                    /home/ajb/Desktop/mprot
63f12acf1000-63f12acf2000 r--p 00002000 00:4c 6689407                    /home/ajb/Desktop/mprot
63f12acf2000-63f12acf3000 r--p 00002000 00:4c 6689407                    /home/ajb/Desktop/mprot
63f12acf3000-63f12acf4000 rw-p 00003000 00:4c 6689407                    /home/ajb/Desktop/mprot
63f12cc24000-63f12cc45000 rw-p 00000000 00:00 0                          [heap]
795b20800000-795b20828000 r--p 00000000 103:02 26374064                  /usr/lib/x86_64-linux-gnu/libc.so.6
795b20828000-795b209b0000 r-xp 00028000 103:02 26374064                  /usr/lib/x86_64-linux-gnu/libc.so.6
795b209b0000-795b209ff000 r--p 001b0000 103:02 26374064                  /usr/lib/x86_64-linux-gnu/libc.so.6
795b209ff000-795b20a03000 r--p 001fe000 103:02 26374064                  /usr/lib/x86_64-linux-gnu/libc.so.6
795b20a03000-795b20a05000 rw-p 00202000 103:02 26374064                  /usr/lib/x86_64-linux-gnu/libc.so.6
795b20a05000-795b20a12000 rw-p 00000000 00:00 0 
795b20a7e000-795b20a81000 rw-p 00000000 00:00 0 
795b20a98000-795b20a99000 rw-p 00000000 00:00 0 
795b20a99000-795b20a9a000 ---p 00000000 00:00 0 
795b20a9a000-795b20a9d000 rw-p 00000000 00:00 0 
795b20a9d000-795b20a9e000 r--p 00000000 103:02 26374061                  /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
795b20a9e000-795b20ac9000 r-xp 00001000 103:02 26374061                  /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
795b20ac9000-795b20ad3000 r--p 0002c000 103:02 26374061                  /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
795b20ad3000-795b20ad5000 r--p 00036000 103:02 26374061                  /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
795b20ad5000-795b20ad7000 rw-p 00038000 103:02 26374061                  /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffe37857000-7ffe37878000 rw-p 00000000 00:00 0                          [stack]
7ffe3789e000-7ffe378a2000 r--p 00000000 00:00 0                          [vvar]
7ffe378a2000-7ffe378a4000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

Device (s)

Raspberry Pi 5

System

ajb@raspberrypi:/local/src/dosemu2.git $ cat /etc/rpi-issue
Raspberry Pi reference 2024-11-19
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 891df1e21ed2b6099a2e6a13e26c91dea44b34d4, stage4

ajb@raspberrypi:/local/src/dosemu2.git $ vcgencmd version
2024/09/23 14:02:56 
Copyright (c) 2012 Broadcom
version 26826259 (release) (embedded)

ajb@raspberrypi:/local/src/dosemu2.git $ uname -a
Linux raspberrypi 6.6.62+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64 GNU/Linux

Logs

Nothing appears in dmesg at the time this test program is run

Additional context

No response

[pelwell]

The default Pi 5 kernel uses 16kB pages for increased performance - try with kernel=kernel8.img in config.txt.

[andrewbird]

Cool, thanks @pelwell that fixed both the test case, and the application.

ajb@raspberrypi:/local/src/dosemu2.git $ ./mprot 
mapped 0x7fb923e000
mprotect 0x7fb923f000 worked
5586950000-5586951000 r-xp 00000000 08:02 1849096                        /local/src/dosemu2.git/mprot
558696f000-5586970000 r--p 0000f000 08:02 1849096                        /local/src/dosemu2.git/mprot
5586970000-5586971000 rw-p 00010000 08:02 1849096                        /local/src/dosemu2.git/mprot
55c5f06000-55c5f27000 rw-p 00000000 00:00 0                              [heap]
7fb9040000-7fb91c7000 r-xp 00000000 08:02 6048                           /usr/lib/aarch64-linux-gnu/libc.so.6
7fb91c7000-7fb91dc000 ---p 00187000 08:02 6048                           /usr/lib/aarch64-linux-gnu/libc.so.6
7fb91dc000-7fb91e0000 r--p 0018c000 08:02 6048                           /usr/lib/aarch64-linux-gnu/libc.so.6
7fb91e0000-7fb91e2000 rw-p 00190000 08:02 6048                           /usr/lib/aarch64-linux-gnu/libc.so.6
7fb91e2000-7fb91ef000 rw-p 00000000 00:00 0 
7fb9208000-7fb922f000 r-xp 00000000 08:02 5758                           /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
7fb923e000-7fb923f000 rw-p 00000000 00:00 0 
7fb923f000-7fb9240000 ---p 00000000 00:00 0 
7fb9240000-7fb9243000 rw-p 00000000 00:00 0 
7fb9243000-7fb9245000 r--p 00000000 00:00 0                              [vvar]
7fb9245000-7fb9246000 r-xp 00000000 00:00 0                              [vdso]
7fb9246000-7fb9248000 r--p 0002e000 08:02 5758                           /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
7fb9248000-7fb924a000 rw-p 00030000 08:02 5758                           /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
7fcffc4000-7fcffe5000 rw-p 00000000 00:00 0                              [stack]

Now do you know of any way to change the source of the test case to not require this change to config.txt?

[pelwell]

The mmap() size must be an integral number of pages, so I would replace the instances of 4096 with a macro/variable and make its value 16384 - at least for your test.

[andrewbird]

Thank you again, I'll take that back to the developer and see if he can make the change.

[popcornmix]

sysconf(_SC_PAGESIZE); may be the run time function he wants.
See here

[andrewbird]

Thanks @popcornmix , I'd already looked up getpagesize() but not spotted that it was deprecated. I've passed it on now.

Many thanks to both of you for your help,

Andrew