libc: Activate mmap for malloc

[d3m3vilurr]

this patch needs user level testing.
at least we got the more than 90MB heap after the patch without overriding _newlib_heap_size_user variables.

I got 96MBs heap with this test code.

#include <psp2/kernel/threadmgr.h>
#include <psp2/kernel/processmgr.h>
#include <stdio.h>
#include <stdlib.h>

#include "debugScreen.h"

#define printf psvDebugScreenPrintf

int main(int argc, char *argv[]) {
    psvDebugScreenInit();
    void *p = NULL;
    int size = 0;
    while (1) {
        // alloc 1MB
        p = malloc(1 * 1024 * 1024);
        if (!p) {
            break;
        }
        size++;
        printf("alloc %d MB\n", size);
        sceKernelDelayThread(100000); // Wait for 0.1 seconds
    }

    printf("check done\n");
    sceKernelDelayThread(3*1000000); // Wait for 3 seconds
    sceKernelExitProcess(0);
    return 0;
}

pr requires #44

[d3m3vilurr]

@Rinnegatamante I checked the vita-quake case.

imo main problem is vitaGL code set.

1. maybe you want to access every heap memory on the gpu.
2. and you override _sbrk_r for it

I don't agree that this hack approach is safe, but it's out of subject.

main problem is this action is only designed every memchunk allow from gpu, but some new chunk (made by mmap) wouldn't allow it until calling sceGxmMapMemory.

actually this hack is brilliant and it could make the easy code. but sadly i don't know good solution to support it.

also imo, only vitagl homebrews had the problem. vita2d wouldn't get the same problem.
i didn't build retroarch but imo it would be work well.

tl;dr

• vitagl's memory handling has the problem that occurs to be unable to support the musl toolchain.
• malloc pr also touches this, and it seems broken changes. this reason why i split from [WIP] Implement mmap/munmap #44

[Rinnegatamante]

vitaQuake case? every single vitaGL made will be compliant with it. Btw do whatever u want. I'm not going to update newlib regardless and i'm not in Henkaku discord server so i won't have to deal with people coming cause vita2d apparently stopped allocating textures for no reasons.

Maybe reading my comment was too much of a request to understand where the problem resides in allocating shittons of memblocks for no reason.

[d3m3vilurr]

yeah. i know event vitagl application will get the same problem after merge it.
I just tested vita-quake to cross check broken feature.

and i was misunderstanding your comment, i thought comment is vita2d will break after merge.
but might be your comment is vita2d couldn't work with big/many textures.

my opinions

• vita2d problem is vita2d's; it's out of this project.
• vitaGL problem is also vitaGL's; same as vita2d. but the hack code is efficient both of perf & readability. but it also makes limitation to every using applications. maybe we are able to solve this, it would be better.
  do you think real os heap/page memory area is linear? :)
• but this malloc pr would be holding until reading the allocation steps/checking some more applications.

[Rinnegatamante]

You can revert vitaGL to when it used to alloc a single memblock everytime a texture is allocated (the way vita2d works) and experience some juicy crash with whatever homebrew in existence using vitaGL due to running out of memblocks (which is what allocating memblocks randomly with malloc would cause.
As for vitaGL:

1. i've no idea why you keep taking reference of it when i didn't even mention it.
2. vitaGL mem code has no issues since there's specifically a ram_threshold arg at vitaGL initialization that allows you to leave mem for extra memblocks if required.

[d3m3vilurr]

ah sorry. @frangarcj suggested to me two application for testing. one of them is vita-quake. (another one is retroarch)
so I commented testing result

if we wanted testing this pr, we have to remove overriding _newlib_heap_size_user first. anyway it's next step.

[Rinnegatamante]

I can give you a simple test case of the faultiness of this thing. Take the code you put at the first comment of this PR. Outside of the while try to alloc a memblock of USER_RW.
The alloc will fail even if there's enough free memory (you can check free memory being available with sceKernelGetFreeMemorySize).
That's cause you ran out of memblock handles.