layout: Support running the firmware as a BIOS image #24

josephlr · 2019-11-12T11:53:10Z

Depends on #37
Fixes #5

Now you can do the following:

cargo xbuild --target target.json --release --features=rom
sstrip target/target/release/hypervisor-fw
qemu-system-x86_64 \
    -machine type=q35,accel=kvm \
    -cpu host -smp 1 -m 1024 \
    -display none -serial stdio \
    -bios target/target/release/hypervisor-fw \
    -drive id=boot,file=clear-28660-kvm.img,format=raw,if=none \
    -device virtio-blk-pci,drive=boot,disable-legacy=on

And the firmware will go from the real-mode reset vector all the way to Rust.

josephlr

@rbradford this code is ready for review (but we may want to merge #23 first). I had some questions (commented inline) that we should resolve before merging this.

Cargo.toml

josephlr · 2019-11-13T10:59:17Z

src/asm/rom.s

+reset_vector: # 0xffff_fff0
+    # This code must be 16 bytes or less, so be careful when adding anyting.


Is there an easy way to check in this file that the reset_vector code is exactly 16 bytes? I had to do objdump to make sure everything was setup correctly.

I suggest placing only the jmp here, and moving the CR0 write after the jump. The interrupt flag is already 0, so CLI is not needed either.

I rewrote this to just have jmp rom16 at the reset vector, and then the rom16 code just follows the exact steps in the SDM to enter protected mode.

I kept the CLI, as the SDM says to call it before entering protected mode, but I might be misreading it (or reading the wrong section).

josephlr · 2019-11-13T11:01:59Z

src/asm/ram32.s

+    movw $0x3f8, %dx
+    movb $'R', %al
+    outb %al, %dx


Should we keep these characters being printed to serial output? I found it useful when debugging/writing the assembly, but it is a little weird to see RPL printed right before the firmware starts up.

Fine for now, but maybe we could look into splitting out output between 0x402 (QEMU firmware I/O debug port) and the serial port. This could be a candidate for that.

I agree, it would be nice to have all of the log! statements go to debug output and only have SimpleTextOutputProtocol log to the standard serial port.

I removed all the serial logs from the assembly. I just have them in a separate branch for local debugging.

src/asm/ram32.s

src/asm/rom.s

rbradford · 2019-11-13T13:56:04Z

And the firmware will go from the real-mode reset vector all the way to Rust. Driver bug is preventing boot due to VIRTIO feature negotiation.

Looking good.

The virtio issue is really that the PCI bar address is zero but that zero bar is still the one that the virtio device is still saying to be used. I think understanding why that PCI bar is zero is the next step forward.

bonzini

I suggest only leaving a jmp at the reset vector.

bonzini · 2019-11-13T15:43:20Z

src/asm/rom.s

+reset_vector: # 0xffff_fff0
+    # This code must be 16 bytes or less, so be careful when adding anyting.


I suggest placing only the jmp here, and moving the CR0 write after the jump. The interrupt flag is already 0, so CLI is not needed either.

Cargo.toml

josephlr

I made some changes to simplify and better document the code.

josephlr · 2019-11-13T21:44:01Z

src/asm/ram32.s

+    movw $0x3f8, %dx
+    movb $'R', %al
+    outb %al, %dx


I agree, it would be nice to have all of the log! statements go to debug output and only have SimpleTextOutputProtocol log to the standard serial port.

Cargo.toml

josephlr · 2019-11-13T23:21:08Z

src/asm/rom.s

+reset_vector: # 0xffff_fff0
+    # This code must be 16 bytes or less, so be careful when adding anyting.


I rewrote this to just have jmp rom16 at the reset vector, and then the rom16 code just follows the exact steps in the SDM to enter protected mode.

I kept the CLI, as the SDM says to call it before entering protected mode, but I might be misreading it (or reading the wrong section).

src/asm/ram32.s

josephlr · 2019-11-15T11:21:53Z

@rbradford @bonzini do the fixes and comments look good? Is there anything else you want me to do here? I could include a high-level doc file explain the multistage boot process (rom16 -> rom32 -> ram32 -> ram64 -> Rust), if you think it's necessary.

rbradford · 2019-11-15T11:28:27Z

@josephlr I think what you've got here is great. I do think it's a good idea to boot through to userspace before merging this as there is obviously something missing. We have a good data point now with the fact that PVH mode with Seabios does show up the PCI BAR address.

josephlr · 2019-11-15T11:32:12Z

@josephlr I think what you've got here is great. I do think it's a good idea to boot through to userspace before merging this as there is obviously something missing. We have a good data point now with the fact that PVH mode with Seabios does show up the PCI BAR address.

Seems reasonable, I'll focus on #26, and try to get that booting to user-space first, and then come back to this (once we know why it's broken).

rbradford · 2019-11-15T12:10:53Z

@josephlr I was able to dump the registers for QEMU and for CH just before jumping to the kernel. They should be in the same state:

https://gist.github.com/rbradford/7ccc15bc2c55d6423840896a42491ac0
vs
https://gist.github.com/rbradford/b452d5d5b8e3bddb39040996dede9516

I've not done a deep analysis yet but one obvious thing stands out and that is that the CS and CR0 look different.

josephlr · 2019-11-15T12:29:54Z

I've not done a deep analysis yet but one obvious thing stands out and that is that the CS and CR0 look different.

I think the CS segments being different is OK (as most of the bits are ignored), CR0 difference is also interesting. However, manually setting the CR0 and CS bits to match CH didn't seem to help (kernel stopped in the same place).

rbradford · 2019-11-15T12:31:21Z

In case you want to reproduce:

Modifying the main.rs just after "Jumping to kernel" to write the 0x80 I/O port. CH has a function that prints out debug messages when that is written to so it wasn't hard to print the KVM registers. For QEMU i modified the firmware to boot infinitely after the same message and used "-serial mon:stdio" and used the monitor (ctrl-a c) to get the registers ("info registers")

bonzini · 2019-11-15T12:39:00Z

Clearing CR0.CD and CR0.NW is a good idea anyway for the firmware, but it's only a matter of tidiness (disabling the cache or writeback doesn't work in VMs).

josephlr · 2019-11-15T12:57:24Z

So setting earlyprintk=serial,keep on the command line is great, you get printing from super early in boot, before Linux even decompresses itself.

So the problem is a kernel panic due to it running out of memory.
Normal Boot with CH
Paniced boot with QEMU

josephlr · 2019-11-15T13:04:50Z

Oh duh, we don't get an e820 map passed in with PVH, so we have to make it ourselves. Right now we're passing in a map with zero memory, and this makes Linux quite unhappy.

rbradford · 2019-11-15T13:09:59Z

@josephlr Use the CMOS to get the limits and make one? Although CH currently has a CMOS implementation it might not be sticking around as nothing is currently using it so best use an E820 where available and fall back to CMOS where it's not.

josephlr · 2019-11-15T13:28:59Z

It looks like pvh gives us an e820 map, so we should be able to just use that.

josephlr · 2019-11-16T02:27:20Z

So it looks like regardless of how we run the firmware (QEMU, PVH, CH, Firecracker, etc..) we eventually need to get two pieces of information from the host:

the address of the RSDP
the e820 table

All the other info we get from the host is optional.

We can get the info in the following way:

Boot Method	RSDP	e820 table
CH/Firecracker	`acpi_rsdp_addr` in `boot_params`	`e820_table` in `boot_params`
PVH	`rsdp_paddr` in `hvm_start_info`	`memmap_paddr` in `hvm_start_info`
QEMU BIOS	`etc/e820` fw_cfg file	tables passed by QEMU

rbradford · 2019-11-16T09:11:59Z

For Q35 rsdp_in_ram is set so I think the RSDP should be findable by scanning the EBDA region for "RSDP" as the spec suggests. For PC however it's only available via QEMU FW_CFG,

As we have a very simple memory model we might be able to get away with hardcoded ranges + the memory output details from the CMOS which might be easier than implementing QEMU FW_CFG.

Our 32-bit GDT just has 1 code and 1 data segment descriptor. They are both flat (spanning the entire 4G address space) and are placed in normal static memory (for now). Signed-off-by: Joe Richey <[email protected]>

The new assembly files handle: - reset.s: Jumping from reset - rom16.s: Transitioning to 32-bit mode - rom32.s: Copying data from ROM to RAM, jumping to PVH entry point To place this code correctly, we add a new Program Header for the code and data that expect to be in ROM. See the comments in layout.ld for more information. We also place the 32-bit GDT in the ROM. This is mostly for convenience, as it lets us use the GDT directly from the ROM code without having to do any complex offset calculations. As laying out the code for a ROM makes the binary ~45% bigger, we gate building as a ROM behind an optional feature. Signed-off-by: Joe Richey <[email protected]>

Without this change, QEMU/CloudHV will attempt to load the ROM into the memory region below 4GiB. But something (i.e. SeaBIOS) is already there. We just pick an arbitrary address. It doesn't actually matter where it gets loaded, as the ROM code isn't used when doing PVH boot. Signed-off-by: Joe Richey <[email protected]>

josephlr force-pushed the bios branch 4 times, most recently from ecb4507 to dc97d6b Compare November 13, 2019 10:51

josephlr commented Nov 13, 2019

View reviewed changes

bonzini suggested changes Nov 13, 2019

View reviewed changes

josephlr force-pushed the bios branch from dc97d6b to 1b2291a Compare November 13, 2019 21:34

josephlr commented Nov 13, 2019

View reviewed changes

josephlr mentioned this pull request Nov 14, 2019

pvh: Support booting via PVH ELFNOTE #26

Merged

josephlr mentioned this pull request Feb 29, 2020

Cleanup code and remove unsupported dependencies #31

Merged

josephlr force-pushed the bios branch 3 times, most recently from 89ff693 to fd40b6e Compare March 30, 2020 11:11

josephlr force-pushed the bios branch from fd40b6e to 99abf1d Compare April 9, 2020 08:58

josephlr mentioned this pull request Apr 9, 2020

Move Note and GDT definitions into Rust #37

Merged

josephlr force-pushed the bios branch from 99abf1d to b2662fa Compare April 9, 2020 09:16

josephlr force-pushed the bios branch 2 times, most recently from cb38019 to 4d22a46 Compare April 13, 2020 04:24

josephlr force-pushed the bios branch 6 times, most recently from 9217c0a to 3c821b0 Compare May 6, 2020 03:57

josephlr force-pushed the bios branch from 3c821b0 to 48cd8bb Compare May 18, 2020 01:10

josephlr force-pushed the bios branch from 48cd8bb to b4bc1c3 Compare June 14, 2020 06:23

josephlr added 3 commits September 10, 2020 01:22

gdt: Add definitions for 32-bit GDT

a91b400

Our 32-bit GDT just has 1 code and 1 data segment descriptor. They are both flat (spanning the entire 4G address space) and are placed in normal static memory (for now). Signed-off-by: Joe Richey <[email protected]>

josephlr force-pushed the bios branch from b4bc1c3 to 8961efc Compare September 10, 2020 08:22

retrage mentioned this pull request Nov 6, 2020

coreboot support #74

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

layout: Support running the firmware as a BIOS image #24

layout: Support running the firmware as a BIOS image #24

josephlr commented Nov 12, 2019 •

edited

Loading

josephlr left a comment

josephlr Nov 13, 2019

bonzini Nov 13, 2019

josephlr Nov 13, 2019

josephlr Nov 13, 2019

rbradford Nov 13, 2019

josephlr Nov 13, 2019

josephlr Mar 30, 2020

rbradford commented Nov 13, 2019

bonzini left a comment

bonzini Nov 13, 2019

josephlr left a comment

josephlr Nov 13, 2019

josephlr Nov 13, 2019

josephlr commented Nov 15, 2019

rbradford commented Nov 15, 2019

josephlr commented Nov 15, 2019

rbradford commented Nov 15, 2019

josephlr commented Nov 15, 2019

rbradford commented Nov 15, 2019

bonzini commented Nov 15, 2019

josephlr commented Nov 15, 2019

josephlr commented Nov 15, 2019

rbradford commented Nov 15, 2019

josephlr commented Nov 15, 2019

josephlr commented Nov 16, 2019

rbradford commented Nov 16, 2019

		reset_vector: # 0xffff_fff0
		# This code must be 16 bytes or less, so be careful when adding anyting.

layout: Support running the firmware as a BIOS image #24

Are you sure you want to change the base?

layout: Support running the firmware as a BIOS image #24

Conversation

josephlr commented Nov 12, 2019 • edited Loading

josephlr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rbradford commented Nov 13, 2019

bonzini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josephlr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josephlr commented Nov 15, 2019

rbradford commented Nov 15, 2019

josephlr commented Nov 15, 2019

rbradford commented Nov 15, 2019

josephlr commented Nov 15, 2019

rbradford commented Nov 15, 2019

bonzini commented Nov 15, 2019

josephlr commented Nov 15, 2019

josephlr commented Nov 15, 2019

rbradford commented Nov 15, 2019

josephlr commented Nov 15, 2019

josephlr commented Nov 16, 2019

rbradford commented Nov 16, 2019

josephlr commented Nov 12, 2019 •

edited

Loading