-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Minimal collection for running on Xen/ARM64 #1358
base: main
Are you sure you want to change the base?
Conversation
Commit cecb5a1 was tested and confirmed to work. |
Before being rebased, 54bf574 was tested and confirmed to work. The delta is simply removing the stray clock interrupt fix and pulling 2 of the intermediates. Mostly there are 3 problems in need of fixes. First, since Xen interrupts are very dynamic, release needs to work #1281. Second, some sort of per-processor hook is needed; Lastly, INTRNG's interface is appropriate for many PICs. Problem is due to being an unusual bus, this is well suited to a lower-level interface, #1285. Trying to work with the existing interface would be quite expensive. Then simply becomes an issue of figuring out how much squashing should be done (while @jgrall did the original implementation, this has rather a lot of changes to get it into shape). |
So instead of #1280, #1363 also functions to provide access to the appropriate hooks. For
In
After the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just looked at intrng, not very much at Xen-specific bits yet.
I'm expecting the per-processor startup hook to change in light of other recent commits. Two issues with INTRNG remain though:
|
d5a5aff
to
f514ba9
Compare
fe6bd96
to
682243c
Compare
There is minimal benefit in delaying event allocation. Worse, this reduces opportunities for merging architectural interrupt structures together. Since the event is now created before being added to the interrupt table, there is no longer any need for locking. A few spots also no longer need to check for ->isrc_event being NULL, clean those up. Differential Revision: https://reviews.freebsd.org/D40166
SSIA. If the event is still valid when deregister is called, it needs to be destroyed. Differential Revision: https://reviews.freebsd.org/D38599
intr_describe_irq() was mixing two operations together. There was the mapping step and the underlying intr_event_describe_handler() call. Split these two steps apart to match other portions of the interface. Differential Revision: https://reviews.freebsd.org/D39333
Specialized peripheral PIC drivers may need to handle most interrupt setup steps themselves without touching newbus. Two cases are intr_add_handler() and intr_describe() which are internally used by `intr_setup_irq()`/`intr_describe_irq()`. Exposing these allow for alternative use case. In fact the BUS interface for INTRNG is arguably a layering violation. Those 6 functions could just as well be in nexus.c or a kernel library. Differential Revision: https://reviews.freebsd.org/D39333
Match the i386/AMD64 headers in doing inline functions for the Xen hypervisor calls. This has been heavily inspired by work done by Julien Grall and Stefano Stabellini. Differential Revision: https://reviews.freebsd.org/D30996
There is minimal variation between architectures for the hypercall wrappers. Most common is word size, which effects a few wrappers, for these check for ILP32 (which identifies all 32-bit architectures). Also notable is HYPERVISOR_set_trap_table() which appears x86-only. Due to the minimal difference, move most of the upper level wrappers to a common header, leaving only the lower level wrappers with the architecture. SPDX tags were added. Add comment documenting _hypercall#() interface with an eye towards future development. The amd64 definition of HYPERVISOR_set_callbacks() was wrong and the i386 was used instead (last argument is ignored by Xen on amd64, but the definition was still wrong).
xencons_cnprobe() is an ideal point to probe for the presence of the Xen hypervisor. This is the first place which MUST know whether Xen is present (unless the console is disabled). For x86 there are other earlier spots where Xen's presence is currently probed, but for other architectures this is ideal. No functional changes intended. Reviewed by: royger Differential Revision: https://reviews.freebsd.org/D30816
The main source file originated as the files "sys/arm64/xen/xen-dt.c" and "sys/arm64/xen/xen_arch_intr.c". They're being merged. These originally resided in sys/arm64/, but they've been moved to sys/dev/xen/bus/intrng-* since they can likely be shared between INTRNG architectures. Julien Grall's original commits lacked the copyright notice, but Julien Grall has agreed to BSD license.
This distributes the event channels among processors instead of placing all of them on vCPU#0. Normal interrupt sources are balanced once, at the end of the boot process. Since new event channels can be created any time, they need to be dynamically balanced. Differential Revision: https://reviews.freebsd.org/D31690
The isrc allocation has been moved to architecture code where the decision to allocate or not can be made. This means setting of several fields needs to be handled by architecture code. Modify the prototype of xen_arch_intr_alloc(), start passing the variables needed, and finally set them on aarch64. This also means releasing of isrcs also needs to be moved to architecture code. Differential Revision: https://reviews.freebsd.org/D31063
Fallout from abc7a4a. Apparently nowhere else in FreeBSD was the PAGE_MASK_4K macro used. Though perhaps few places were using the explicitly sized PAGE_* macros.
Simple cleanup.
New implementation of the check. While Julien Grall's implementation had the check, it was completely removed during bring-up for expedience. This has been rebased on top to recognize the original also had the test even though its action no longer works.
Nominally the function should work for this case too, just it hasn't been tested in this situation.
With the variables now in xen_common.c, this file no longer needs to define them. Nuke this remnant.
Some previous builds appeared to potentially need the simplebus binding, but this has been shown to be unnecessary. Only use ofwbus since that is how Xen/ARM passes the hypervisor information. This is known to work once based on top of INTRNG, does this actually work when on top of the kernel events?
Initial implementation. Best to replace the existing file Julien Grall originally created. I /think/ this is what these are supposed to do...
Almost certainly needed.
This is early enough for the setup. No barrier is needed, we merely need invocation on every processor during bring-up.
Alas, ARM declared xen_ulong_t to be 64-bits long, unlike i386 where it matches the word size. As a result, compatibility wrappers are needed for Xen atomic operations.
The step of enabling the setup in general is better a separate commit. This was broken off of "xen/arm64: add xen platform". Submitted by: Elliott Mitchell <[email protected]> Original implementation: Julien Grall <[email protected]>, 2014-01-13 17:40:58 Differential Revision: https://reviews.freebsd.org/D30950
Presently the probing mechanism for Xen requires device-trees, but no other portion requires FDT. As a result marking everything as depending upon FDT would be wrong, but this requires excluding Xen from LINT-ACPI. Add a similar section to LINT-FDT since it will likely be needed in the future.
Initial implementation. Best to replace the existing file Julien Grall originally created. I /think/ this is what these are supposed to do...
Should handle the interface for rebinding interrupts. May well produce error returns though. Based on GICv3.
This /looks/ plausible based on examples.
Seems using the proper interface adds a colon for us. Adjust to match what we should look like.
When this is called there /should/ be a source, right? (hmm, that may be backwards...)
The interface considers was previously willing to call bind without the event having been set. This is now impossible, but keep this as an assert. This fixes the previous commit. Appears subsequent adjustments omitted handling this.
Several functions have turned back into extern, now onto implementation.
Rather than looking at the event, INTRNG chooses to duplicate this data. As such need to update the information here too.
The use of these has completely disappeared, so nuke them.
Hand regions provided by Xen for grant-table and other allocations. This was originally intended for grant-table mappings, but Xen allows them to be used for anything. Xen guarantees these regions will be empty, and the device-tree code prevents them from being used for other purposes.
Despite the documentation being unclear, this *does* need to be initialized.
Now that we can add the handler and describe via INTRNG, do so. This might also fix balancing during start.
Appears INTRNG wants the PIC to choose the processor, not suggesting a processor? Seems odd. Try this.
Many thanks to Julien Grall who did the initial work in 2014-2015. Now we can have FreeBSD on Xen on ARM64 machines. Want accelerated graphics on a Raspberry PI? Differential Revision: https://reviews.freebsd.org/D31956
ARM64 has been completed. A great deal of the recent work was done by royger.
For people who didn't figure out the end goal of all the pull requests, here is a proof of concept with the pieces gathered together. Large amounts of other cleanup was found during implementation, hopefully those get in too.
The presently tested and known working bootloader is the ArmVirtXen build of Tianocore/EDK2. Due to a change in Xen at present one patch is required. Hopefully a similar patch will be integrated soon, but presently that extra is needed.
Configure the VM with
kernel = "XEN_EFI.fi"
and a FreeBSD mini-memstick build as a disk. Then everything will boot as expected.Likely a bunch of squashing is needed, this though is a bit tricky to ensure Julien Grall gets appropriate credit. Though the Xen implementation was heavily rewritten.
Issues:
pic_init_secondary()
.Possibly add pass for devices not in PIC hierarchy Intrng: adjusting pic_init_secondary for further use cases #1280.Possibly declare hypervisor PIC root.Mostly advice on which direction to use.intr_isrc_deregister()
to release the event. Presently fix INTRNG interrupt release #1281.