[boot] Expand /bootopts to 1023 bytes #2100
Conversation
Also for speed read only one sector for superblock
Also release setup data segment (REL_INITSEG) to main memory
|
Thank you @ghaerr, this is great. You beat me to it, and I'm glad you did: The restructuring of bootopts-processing and the release of the segment after use - very nice! And thanks for the hint about Hex Fiend! I used it way back and have since forgotten completely about it. Extremely powerful - and immediately applicable to the rabbit chase I'm in at the moment. |
I got a little ahead of myself trying to put all the bootopts processing variables into a struct and reverted portions of the release commit. Some of them are used as argv/slen values to sys_exec when starting /bin/init and thus can't be released by itself. I've determined releasing the variables later by the idle task is unsafe. The final result works well, and only the options[] and umb[] setup buffers are released, but that's a large part of the total - a little over 1K returned to the kernel data segment. An unforeseen nice enhancement is that after kernel startup, but before /bin/init, adding the ~1K back to the kernel heap ends up being used for a bunch of smaller allocations, which otherwise would fragment the older, larger sections of the kernel heap. This can be seen below in the allocations occurring at kernel data segment location 3E12, which used to be the options parsing buffer: |
Really nice, @ghaerr. It will be interesting to see how this works out on TLVC where the small heap allocations are made from the top of the heap to avoid the very same fragmentation. My first thought is that it should be quite simple to fill the bottom segment first, then continue from the top. We'll see when we get there. Thanks. |
The ELKS near heap allocator always uses best fit. Now, instead of having a single initial (very large) heap segment, the second heap_add of approximately 1050 bytes causes that segment to fill first (from low addresses). If the TLVC allocator has been modified to fill from high addresses first but still uses best fit, then it would seem to perform as you say, except likely fill from the high address of the small segment first, then the high addresses of the larger segment? Does TLVC still use best fit, or does it always place allocations less than a certain size at highest address instead? Given this interesting behavior, its on my list to try some other arena-size experimentation, perhaps allocating the initial heap into two or three arena sizes from the start. The real test will be to look at fragmentation after quite a few processes have run. |
|
The TLVC allocator https://github.com/Mellvik/TLVC/pull/36 was modified to check for SEG descriptor allocations and allocate those from the top since they are small, dynamic and tend to cause fragmentation. That has worked out really well. Best fit still rules, so if there is space, a small TTY buffer allocation will end up in the middle of the SEG descriptors, but that's fine. Changing this to take advantage of split heap segments should really be a walk in the park - and really useful, maybe even reserve the first (low men) heap segment to small allocations.
I agree, this is an interesting experiment. The heap fragmentation is very much activity dependent, as would be expected. Unusual activity like switching between network interfaces on a running system, i.e. releasing and reallocating network & ntty buffers may still cause some interesting fragmentation, but I suspect a more 'regular' environment would be fine with the current setup, possibly reserving the low mem segment to small allocations. |
Actually, the low memory segment is the last heap segment. Undoubtably since it was added last but contains .data address offsets from the options[] buffer. It seems that the heap allocator isn't sorting new segments when added to the heap, which could be considered a bug with regards to any later block merging (although that won't happen in this case). Nonetheless, as can be seen in my screenshot above, the lower addressed segment comes later in the meminfo listing, which is a bit strange to look at. I may fix this but I'm suspecting that the heap manager was written with the idea of only a single heap_add to be used, and perhaps its only working because the second heap_add uses lower address. I'm not aware of any code prohibiting merging between the added segments (like libc malloc) so I'm thinking its a definite possibility that more than two heap_add's won't work without allocator enhancements. FYI! |
Requested by @Mellvik some time ago.
This PR adds the ability to use a larger /bootopts file for system configuration, on both MINIX and FAT filesystems. This will allow for more text, explanations or multiple scenarios to be saved. The current only caveat is that when /bootopts is larger than 512 bytes, both disk sectors must be contiguous. This should normally not be a problem since /bootopts is now distributed as a file > 512 bytes using contiguous sectors, and editing or rewriting the file will keep the same sector numbers on FAT. MINIX uses 1K blocks so this is not an issue.
The larger /bootopts file was already almost working, as MINIX already reads two 512-byte disk sectors per block. While looking deeper into the implementation, some subtle bugs were found and fixed, noted below. In addition, several related features were added at the same time, including the release of all the low memory and kernel data segment memory used for /bootopts buffers back to main memory and the kernel heap respectively.
Features added:
Bugs fixed:
Caveats:
@Mellvik: Interestingly, all this work started as the result of playing with a cool macOS hex editor. I tried it out and happened to be looking at the hex dump image/fd1440.img and immediately noticed two "linux" strings in the boot_minix.o second boot sector payload. I figured the boot sector code size could be decreased, and that started the process of noticing exactly how /bootopts was being loaded, and the overwrite occurring at REL_INITSEG... The attention was still focused on the config.h DMASEG and REL_SYSSEG areas and down the rabbit hole I went :)
I highly recommend Hex Fiend, if you set the column width to 32, filesystem layouts become very apparent while scrolling through floppy disk images. Seeing filesystem layouts visually really helps see what is going on.