Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimal collection for running on Xen/ARM64 #1358

Draft
wants to merge 45 commits into
base: main
Choose a base branch
from
Draft

Conversation

ehem
Copy link
Contributor

@ehem ehem commented Aug 2, 2024

For people who didn't figure out the end goal of all the pull requests, here is a proof of concept with the pieces gathered together. Large amounts of other cleanup was found during implementation, hopefully those get in too.

The presently tested and known working bootloader is the ArmVirtXen build of Tianocore/EDK2. Due to a change in Xen at present one patch is required. Hopefully a similar patch will be integrated soon, but presently that extra is needed.

Configure the VM with kernel = "XEN_EFI.fi" and a FreeBSD mini-memstick build as a disk. Then everything will boot as expected.

Likely a bunch of squashing is needed, this though is a bit tricky to ensure Julien Grall gets appropriate credit. Though the Xen implementation was heavily rewritten.

Issues:

sys/kern/kern_clock.c Outdated Show resolved Hide resolved
@ehem
Copy link
Contributor Author

ehem commented Aug 3, 2024

Commit cecb5a1 was tested and confirmed to work.

sys/kern/kern_clock.c Outdated Show resolved Hide resolved
sys/kern/kern_clock.c Outdated Show resolved Hide resolved
@ehem
Copy link
Contributor Author

ehem commented Aug 5, 2024

Before being rebased, 54bf574 was tested and confirmed to work. The delta is simply removing the stray clock interrupt fix and pulling 2 of the intermediates.

Mostly there are 3 problems in need of fixes. First, since Xen interrupts are very dynamic, release needs to work #1281. Second, some sort of per-processor hook is needed; pic_init_secondary() is basically ideal, but the current implementation is problematic, #1280.

Lastly, INTRNG's interface is appropriate for many PICs. Problem is due to being an unusual bus, this is well suited to a lower-level interface, #1285. Trying to work with the existing interface would be quite expensive.

Then simply becomes an issue of figuring out how much squashing should be done (while @jgrall did the original implementation, this has rather a lot of changes to get it into shape).

@ehem
Copy link
Contributor Author

ehem commented Aug 8, 2024

So instead of #1280, #1363 also functions to provide access to the appropriate hooks. For sys/arm64/include/intr.h use:

+#ifdef XENHVM
+#define INTR_ROOT_XEN	2
+#define INTR_ROOT_NUM	3
+#else
+
 #define INTR_ROOT_NUM	2
+#endif

In sys/dev/xen/intrng/xen_arch_intr.c add:

+	i = intr_pic_claim_root_num(dev, ofw_bus_get_node(dev),
+	    (intr_irq_filter_t *)panic, NULL, INTR_ROOT_XEN);
+	if (i != 0)
+		return (i);

After the intr_pic_register() call. This is somewhat hack-ish as we're barely using INTRNG, but it does seem kind of appropriate as it does amount to a disconnected PIC.

Copy link
Member

@markjdb markjdb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just looked at intrng, not very much at Xen-specific bits yet.

sys/kern/subr_intr.c Show resolved Hide resolved
sys/kern/subr_intr.c Outdated Show resolved Hide resolved
sys/kern/subr_intr.c Outdated Show resolved Hide resolved
sys/kern/subr_intr.c Outdated Show resolved Hide resolved
sys/kern/subr_intr.c Outdated Show resolved Hide resolved
@ehem
Copy link
Contributor Author

ehem commented Sep 23, 2024

I'm expecting the per-processor startup hook to change in light of other recent commits. Two issues with INTRNG remain though:

  1. Since Xen events are rather more dynamic than hardware interrupts, released events need some sort of handling. The obvious approach is intr_isrc_deregister(), problem is this hadn't previously been tested and it leaks the struct intr_event.
  2. Since they're a specialized type of interrupt, not all of INTRNG's services are needed. In particular the lower level interface of intr_add_handler() (rename isrc_add_handler()) and intr_describe() (break intr_describe_irq() apart) is rather superior.

@ehem ehem force-pushed the whole branch 2 times, most recently from d5a5aff to f514ba9 Compare November 6, 2024 20:45
@ehem ehem force-pushed the whole branch 6 times, most recently from fe6bd96 to 682243c Compare December 19, 2024 00:33
ehem added 8 commits December 19, 2024 13:19
There is minimal benefit in delaying event allocation.  Worse, this
reduces opportunities for merging architectural interrupt structures
together.

Since the event is now created before being added to the interrupt
table, there is no longer any need for locking.

A few spots also no longer need to check for ->isrc_event being
NULL, clean those up.

Differential Revision: https://reviews.freebsd.org/D40166
SSIA.  If the event is still valid when deregister is called, it needs
to be destroyed.

Differential Revision: https://reviews.freebsd.org/D38599
intr_describe_irq() was mixing two operations together.  There was the
mapping step and the underlying intr_event_describe_handler() call.
Split these two steps apart to match other portions of the interface.

Differential Revision: https://reviews.freebsd.org/D39333
Specialized peripheral PIC drivers may need to handle most interrupt
setup steps themselves without touching newbus.  Two cases are
intr_add_handler() and intr_describe() which are internally used by
`intr_setup_irq()`/`intr_describe_irq()`.  Exposing these allow for
alternative use case.

In fact the BUS interface for INTRNG is arguably a layering violation.
Those 6 functions could just as well be in nexus.c or a kernel library.

Differential Revision: https://reviews.freebsd.org/D39333
Match the i386/AMD64 headers in doing inline functions for the Xen
hypervisor calls.  This has been heavily inspired by work done by
Julien Grall and Stefano Stabellini.

Differential Revision: https://reviews.freebsd.org/D30996
There is minimal variation between architectures for the hypercall
wrappers.  Most common is word size, which effects a few wrappers, for
these check for ILP32 (which identifies all 32-bit architectures).  Also
notable is HYPERVISOR_set_trap_table() which appears x86-only.  Due to
the minimal difference, move most of the upper level wrappers to a
common header, leaving only the lower level wrappers with the
architecture.

SPDX tags were added.

Add comment documenting _hypercall#() interface with an eye towards
future development.

The amd64 definition of HYPERVISOR_set_callbacks() was wrong and the
i386 was used instead (last argument is ignored by Xen on amd64, but the
definition was still wrong).
xencons_cnprobe() is an ideal point to probe for the presence of the Xen
hypervisor.  This is the first place which MUST know whether Xen is
present (unless the console is disabled).

For x86 there are other earlier spots where Xen's presence is currently
probed, but for other architectures this is ideal.

No functional changes intended.

Reviewed by: royger
Differential Revision: https://reviews.freebsd.org/D30816
The main source file originated as the files "sys/arm64/xen/xen-dt.c"
and "sys/arm64/xen/xen_arch_intr.c".  They're being merged.

These originally resided in sys/arm64/, but they've been moved to
sys/dev/xen/bus/intrng-* since they can likely be shared between INTRNG
architectures.

Julien Grall's original commits lacked the copyright notice, but Julien
Grall has agreed to BSD license.
ehem and others added 29 commits December 19, 2024 13:19
This distributes the event channels among processors instead of placing
all of them on vCPU#0.  Normal interrupt sources are balanced once, at
the end of the boot process.  Since new event channels can be created
any time, they need to be dynamically balanced.

Differential Revision: https://reviews.freebsd.org/D31690
The isrc allocation has been moved to architecture code where
the decision to allocate or not can be made.

This means setting of several fields needs to be handled by architecture
code.  Modify the prototype of xen_arch_intr_alloc(), start passing the
variables needed, and finally set them on aarch64.

This also means releasing of isrcs also needs to be moved to
architecture code.

Differential Revision: https://reviews.freebsd.org/D31063
Fallout from abc7a4a.  Apparently nowhere else in FreeBSD was the
PAGE_MASK_4K macro used.  Though perhaps few places were using the
explicitly sized PAGE_* macros.
New implementation of the check.  While Julien Grall's implementation
had the check, it was completely removed during bring-up for expedience.
This has been rebased on top to recognize the original also had the test
even though its action no longer works.
Nominally the function should work for this case too, just it hasn't
been tested in this situation.
With the variables now in xen_common.c, this file no longer needs to
define them.  Nuke this remnant.
Some previous builds appeared to potentially need the simplebus binding,
but this has been shown to be unnecessary.  Only use ofwbus since that is
how Xen/ARM passes the hypervisor information.

This is known to work once based on top of INTRNG, does this actually
work when on top of the kernel events?
Initial implementation.  Best to replace the existing file Julien Grall
originally created.  I /think/ this is what these are supposed to do...
This is early enough for the setup.  No barrier is needed, we merely
need invocation on every processor during bring-up.
Alas, ARM declared xen_ulong_t to be 64-bits long, unlike i386 where
it matches the word size.  As a result, compatibility wrappers are
needed for Xen atomic operations.
The step of enabling the setup in general is better a separate commit.

This was broken off of "xen/arm64: add xen platform".

Submitted by: Elliott Mitchell <[email protected]>
Original implementation: Julien Grall <[email protected]>, 2014-01-13 17:40:58
Differential Revision: https://reviews.freebsd.org/D30950
Presently the probing mechanism for Xen requires device-trees, but no
other portion requires FDT.  As a result marking everything as depending
upon FDT would be wrong, but this requires excluding Xen from LINT-ACPI.

Add a similar section to LINT-FDT since it will likely be needed in the
future.
Initial implementation.  Best to replace the existing file Julien Grall
originally created.  I /think/ this is what these are supposed to do...
Should handle the interface for rebinding interrupts.  May well produce
error returns though.  Based on GICv3.
This /looks/ plausible based on examples.
Seems using the proper interface adds a colon for us.  Adjust to
match what we should look like.
When this is called there /should/ be a source, right?  (hmm, that may
be backwards...)
The interface considers was previously willing to call bind without
the event having been set.  This is now impossible, but keep this as
an assert.

This fixes the previous commit.  Appears subsequent adjustments omitted
handling this.
Several functions have turned back into extern, now onto implementation.
Rather than looking at the event, INTRNG chooses to duplicate this data.
As such need to update the information here too.
The use of these has completely disappeared, so nuke them.
Hand regions provided by Xen for grant-table and other allocations.
This was originally intended for grant-table mappings, but Xen allows
them to be used for anything.  Xen guarantees these regions will be
empty, and the device-tree code prevents them from being used for other
purposes.
Despite the documentation being unclear, this *does* need to be
initialized.
Now that we can add the handler and describe via INTRNG, do so.  This
might also fix balancing during start.
Appears INTRNG wants the PIC to choose the processor, not suggesting
a processor?  Seems odd.  Try this.
Many thanks to Julien Grall who did the initial work in 2014-2015.  Now
we can have FreeBSD on Xen on ARM64 machines.  Want accelerated graphics
on a Raspberry PI?

Differential Revision: https://reviews.freebsd.org/D31956
ARM64 has been completed.  A great deal of the recent work was done by
royger.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants