Skip to content

KFS-3
Compare
Choose a tag to compare
@Orycterope Orycterope released this 25 Aug 18:49
· 1262 commits to master since this release
kfs3
356c24e

MMU and gifs

Who said you can't display gifs in the kernel ?

gif of a gif of a gif

serial output:

Screenshot_20190824_212800

This is the third release of KFS. Here's what we've been working on:

Virtual memory

The main focus of this release has been toward supporting virtual memory.

Physical memory allocation

First of all, for every virtual memory allocation we need to allocate some physical memory. It is the job of the Frame Allocator to keep track of which page frame is occupied and which ones aren't.

A page frame is a 4 KiB bytes physical memory block where pages will be laid out.

To do that we implement a dead simple bitmap allocator, tracking for every frame if it is free or occupied.

The 4 GiB / 4 KiB / 8 = 128 KiB array lives in the .bss.

The frame allocator returns RAII Frame types, that automatically tell the frame allocator to free them when they are no longer used (dropped).

Even though it is an astonishingly stupid design, we still managed to get it wrong, and spent several days debugging a seemingly unrelated problem, just because we were giving off occupied frames like candy:

gif of a video of a screenrecording of a gif

What you're seeing here, is the gif mistakenly uncompressing itself in BIOS ROM instead of RAM, where the corrupted black strip happens. From top to bottom: RAM, BIOS ROM, and finally RAM again. Qemu is surprisingly tolerant with that, and just ignores writes to ROM. The actual hardware we tested it on, not so much...

Strongly typed pointers

We always make sure to never mangle Virtual and Physical memory addresses/pointers thanks to rust's strong type system. A physical memory pointer is wrapped in the PhysicalAddress type, virtual ones in the VirtualAddress one, and you can't cast from one into the other without deconstructing them. This is a good safety net that prevents us from doing some really stupid or dangerous stuff without noticing.

MMU

Next, we enable the MMU and use it to provide a contiguous view of the fragmented physical memory and check memory accesses.

Our paging module provides endpoints to create a mapping at a given address with the given permissions, unmap one, get the physical address of a virtual address by parsing the page tables, and find a free virtual region of any given size and alignment, used for creating mappings.

This is all provided by the PageTablesSet trait, which only requires a mean of getting a pointer to the page directory, and from the directory to a page table.

The PageTablesSet trait is implemented by 3 kinds of page table hierarchies, that differ on the way they access the page tables for writing:

  • an ActivePageDirectory will want to use recursive mapping,
  • an InactivePageDirectory will want to temporarily map the table in the current address space, modify it, and then unmap it,
  • a PagingOffPageDirectory will point to physical memory, and directly write to it.

This way we can handle all kinds of page table hierarchies with one single api.

When KFS starts, grub hasn't enabled the paging for us, and we live in physical memory. To enable the paging we must first create a set of page tables somewhere in the physical memory, and enable the MMU by making it use this hierarchy. But as soon as we do this, the kernel itself would become unmapped, and we would page-fault. So we must not forget in this hierarchy to identity map the frames were the kernel was loaded by grub.

Thanks to our PageTablesSet trait being implemented on PagingOffPageDirectory, we can do this really easily by using the same api as we would use once the paging has been enabled.

Lands

We divide the 4 GiB virtual memory space of x86_32 as followed:

0x00000000..0x3fffffff, 1 GiB: Kernel memory
0x40000000..0xffffffff, 3 GiB: User memory

Ultimately we want the kernel to live in high memory, so we will redefine it in the future.

Heap

In kernel memory, we reserve a 512 MiB memory region for the heap. We use a linked list allocator to manage this memory. This enables us to dynamically allocate types in the kernel.

By importing the alloc crate, we can now allocate Box, Vec, and Arc native rust types. This a big step forward.

Stack

Now that we have paging, we have page guards !

We moved from having our stack in a global array that lives in the .bss to a real, allocated type KernelStack, which puts a page guard at the end of every stack so we can immediately crash upon stack overflows. Later we will be able to catch this condition in the page fault handler.

Address space switching

Paving the way for our future scheduler, we already implement switching from a page table hierarchy A to a hierarchy B.

While the 3 GiB of user memory should be different, the 1 GiB of kernel memory should stay the same, otherwise the kernel would panic as soon as it accesses any of its own allocated structures. To do this, before doing the switch, we memcpy the entries of directory A that map to kernel memory (the last ¼ of the directory) to directory B, and then tell the MMU to perform the switch. When the MMU is done, we still have the same view of the kernel memory from hierarchy B. Note that this process only copies entries, not the pages themselves nor the tables. The pages and page tables of the kernel are shared between hierarchy A and B.

This implies we need to copy from the ActivePageDirectory (A) to an InactivePageDirectory (B) that we temporarily map in A. Once again our shared paging api is extremely helpful for that.

VGA & GIF

VGA text mode is fun, but it lacks the endless possibilities for displaying memes that graphical VGA has. So let's use it !

We enable VGA, map the frame buffer into kernel memory, and draw to it.

As a proof of concept we use the image-gif crate to decode a gif frame by frame into a heap-allocated buffer, and then blit this buffer to the vga frame-buffer. This was a good test for our paging and heap allocators (see above).

PIT

Displaying GIFs introduces the need to track time between each frame. So we wrote a minimalist driver for the Programmable Interrupt Timer.

Because we don't have support for interrupts yet, we can't use it in the traditional periodic mode. Instead we program it to use the "one shot" mode, and tell it to wait for a given amount of time. This will make the PIT do the equivalent of a countdown, and our sleep function will poll the PIT until the countdown hits 0, blocking execution while doing so. When the amount of time is higher than the max countdown the PIT can do, which is almost always the case for GIFs, which deals with fractions of a second between frames, we do multiple consecutive countdowns.

Serial

Now that we configured the screen to use VGA and display memes, we now longer have our VGA text mode to log to.

So we implemented a basic rs232 logger.

Since we don't yet handle interrupts, it's really minimalist as it's polling all the time to know if it can write a character, and blocking while doing so.

The api of logging was made so the logging could be multiplexed to several backing devices (e.g. both serial and a file, or serial and a ring buffer, whatever you desire) once we have such devices in the future.

Also we now use the log crate to implement log levels (error/warn/info/debug).

Documentation

We added the cargo make doc and cargo make doc-full rules to the makefile. These will invoke rustdoc to generate an html documentation for the project from our /// doc comments above functions in the code.

How to build

This release is pretty outdated, so we provide a roadmap on how to build it.

reveal build instructions First, make sure to to checkout the `kfs3-fixed` branch, as it contains a fix to pin the version of some dependencies: 
git switch --detach origin/kfs3-fix

Cargo-xbuild

You need to downgrade your version of cargo-xbuild to v0.4.9. You must do this outside of your repo as the rust-toolchain file would try to compile it with a non-working rustc.

cd ~
cargo install cargo-xbuild --force --version 0.4.9
cd -

Cargo.lock

Because sunrise developers were so young and innocent at that time, they didn't fully realize the crucial importance of a Cargo.lock in published project. So they didn't think about including it, and it remained that way for quite a while, until they once tried to checkout an old tag and spent half a day just trying to compile it.

Unfortunately history only goes in one way, and commit logs too ( 👀 ), so we provide a Cargo.lock as DLC in the release files.

Copy it at the root of your repository, and cargo will now fetch the right versions of our dependencies.

grub-mkrescue

Because sunrise developers were lazy-ass monkeys, busy being high on smoking rolled-up intel-specs pretending it's some kind of psychotrope, they used to let all the hard work of creating an iso image be done by grub-mkrescue, while taking all the credit.

This means that depending on your distro, you might need to install:

  • grub
  • mtools

This went on until Thog joined them and created mkisofs-rs, and effectively rescued them, before they fall victim to 8042 withdrawal, and OD trying to snort an A20 line, while attempting to relieve the crave.

Building

Now you should finally be able to build and create the boot iso:

cargo make qemu-release
Assets 4
Loading