x/sys/unix, x/sys/cpu: use the RISC-V Hardware Probing Interface on Linux

[markdryan]

Overview

The RISC-V port of Go is currently restricted to the RV64G instruction set. It does not take advantage of any other RISC-V extensions that could be used to boost performance or reduce code size, for example the Vector or Bit manipulation extensions. This is in stark contrast to other architectures supported by Go where SIMD and bit manipulation instructions are used when they are available. For example, amd64 builds of Go can use AVX2 optimised versions of certain critical functions, e.g., compare and memmove, to improve performance. Until recently, one issue preventing the use of RISC-V extensions beyond RV64G in Go was the lack of a mechanism to detect these extensions. This issue was resolved in Linux 6.4.

Version 6.4 of the Linux kernel introduces a new syscall, sys_riscv_hwprobe that can be used to determine interesting pieces of information about the underlying RISC-V CPUs on which a user space program runs. In particular, it exposes the RISC-V extensions supported by the CPUs in addition to other performance information such as whether and how efficiently the CPUs support unaligned memory accesses.

What about HWCAP?

Where available sys_riscv_hwprobe is preferable to using HWCAP as:

1. It's extensible (there's enough bits for all future extensions).
2. It can be used to query for extensions with multi-letter names, e.g., Zba, Zbb, which are likely to be useful in Go going forward.
3. The hwprobe extension bits are versioned and their meaning is concretely defined.
4. It can be used to determine additional information about the CPU. One potentially interesting example is its ability to return information about how well the CPU handles unaligned accesses. The current optimised RISC-V primitives for Go, e.g., memequal go out of their way to avoid unaligned accesses, i.e., they do 8 individual byte loads and stores instead of two unaligned 8 byte memory accesses. The current implementations might not be optimal on CPUs that support fast unaligned accesses. Hwprobe provides a way to detect fast unaligned accesses and an opportunity to switch to implementations that use them.

sys_riscv_hwprobe should be sufficient by itself for detecting the presence of V, Zba, Zbb, Zbs in vanilla kernels. However, there may be cases where extensions have been back ported to an earlier kernel but hw_probe has not. Such situations could be handled by first trying sys_hw_probe and then falling back to HWCAP if hw_probe is not available. This is the strategy used in OpenJDK.

Update 26/09/2023 However, it is probably not safe to use hwcap to detect the Vector extension as some RISC-V boards, e.g., the SiPeed LicheePi, indicate support for Vector 0.7 ( which is not compatible with Vector 1.0) through the hwcap 'V' bit.

Note that there's already a patch pending review that detects some extensions using HWCAP. Here we propose calling sys_riscv_hwprobe first and if that fails falling back to the code in this HWCAP patch. We would not fall back for Vector.

What extensions can sys_riscv_hwprobe detect?

It can detect FD and C in Linux 6.4

Support for detecting V, Zba, Zbb and Zbs are merged and are available in Linux 6.5.

What are we proposing?

The proposal is then to

(Updated on 26th of September 2023.)

1. Add support for the sys_riscv_hwprobe to golang.org/x/sys/unix (510795). This would add a new syscall, a new type and some new constants to riscv64 builds.
2. Update golang.org/x/sys/unix when Linux 6.5 is released to add support for the new extensions it supports. (530895)
3. Update golang.org/x/sys/cpu to invoke the new syscall and record which extensions are supported. We'd fall back to HWCAP where sys_riscv_hwprobe is not available.
4. Add support for the sys_riscv_hwprobe to internal/syscall/unix runtime (522995).
5. Update internal/cpu to invoke the new syscall in internal/syscall/unix and record which extensions are supported. We'd fall back to HWCAP (not for Vector) where sys_riscv_hwprobe is not available (522995).

Proposal #62238 introduces the golang/x/sys/cpu.RISCV64 structure with one member, HasV. Here we propose adding the following new members to this structure.

        IsMisalignedFast bool // Fast misaligned accesses 
        HasC bool              // The C extension
        HasZba bool            // The Zba address generation extension, version 1.0 or greater
        HasZbb bool            // The Zbb extension, version 1.0 or greater
        HasZbs bool            // The Zbs extension, version 1.0 or greater

[ianlancetaylor]

The proposal committee is concerned primarily with user-visible API changes. Switching to use the Hardware Probing Interface doesn't sound like a user-visible change, so that is fine. And adding new system calls and types to x/sys/unix is fine. So the only question is: what new API would be added to x/sys/cpu? If you can describe that, that would be helpful. Thanks.

[gopherbot]

Change https://go.dev/cl/510795 mentions this issue: unix/linux: Add sys_riscv_hwprobe for riscv64

[markdryan]

So the only question is: what new API would be added to x/sys/cpu? If you can describe that, that would be helpful. Thanks.

Patch 508676 already proposes some changes to golang/x/sys/cpu. It adds a new struct, RISCV64. The intention would be to expand this structure with some additional fields.

        HasMisalignedFast bool // Fast misaligned accesses 
        HasC bool              // The C extension
        HasZBA bool            // The Zba address generation extension
        HasZBB bool            // The Zbb extension
        HasZBS bool            // The Zbs extension

The sys_riscv_hwprobe syscall can detect different types of CPU support for misaligned accesses (unknown, emulated, slow, fast, unsupported). My feeling here is that we only care if fast misaligned access are supported and, if they're not, misaligned accesses should be avoided. One boolean field should then suffice.

HasZBA, HasZBB and HasZBS, would require golang.org/x/sys/unix/linux/Dockerfile to be updated to use LInux 6.5 before they could be set to any value other than false. More fields for new extensions would be added over time as they become supported by hwprobe and the kernel itself.

One issue I did not consider in the initial proposal above is that there appears to be some parallel cpu feature detection code in internal/cpu. IIUC we'd need to duplicate the RISCV64 structure in internal/cpu if we wanted to inspect any of these fields in Go itself, from internal/bytealg for example. To populate the fields in internal/cpu we'd need to make a syscall. How would this work? Would we need to manually add the new syscall to the syscall package in the main Go repo?

[ianlancetaylor]

You might have to add the new syscall to internal/syscall/unix in the standard library.

[gopherbot]

Change https://go.dev/cl/522995 mentions this issue: cpu/internal: provide runtime detection of RISC-V extensions on Linux

[4a6f656c]

So the only question is: what new API would be added to x/sys/cpu? If you can describe that, that would be helpful. Thanks.

Patch 508676 already proposes some changes to golang/x/sys/cpu. It adds a new struct, RISCV64. The intention would be to expand this structure with some additional fields.

        HasMisalignedFast bool // Fast misaligned accesses 
        HasC bool              // The C extension
        HasZBA bool            // The Zba address generation extension
        HasZBB bool            // The Zbb extension
        HasZBS bool            // The Zbs extension

HasMisalignedFast seems awkward - HasFastMisaligned at least matches the comment, or we potentially drop the Has and use FastMisaligned or MisalignedIsFast...

Re HasZB*, given that they're usually referered to as mixed case (Zba and not ZBA), it would probably be better to stick with that here and use HasZb[abs].

The sys_riscv_hwprobe syscall can detect different types of CPU support for misaligned accesses (unknown, emulated, slow, fast, unsupported). My feeling here is that we only care if fast misaligned access are supported and, if they're not, misaligned accesses should be avoided. One boolean field should then suffice.

HasZBA, HasZBB and HasZBS, would require golang.org/x/sys/unix/linux/Dockerfile to be updated to use LInux 6.5 before they could be set to any value other than false. More fields for new extensions would be added over time as they become supported by hwprobe and the kernel itself.

Not necessarily - I suspect there is prior art for adding hard coded constants, which can be replaced with autogenerated ones once available.

One issue I did not consider in the initial proposal above is that there appears to be some parallel cpu feature detection code in internal/cpu. IIUC we'd need to duplicate the RISCV64 structure in internal/cpu if we wanted to inspect any of these fields in Go itself, from internal/bytealg for example. To populate the fields in internal/cpu we'd need to make a syscall. How would this work? Would we need to manually add the new syscall to the syscall package in the main Go repo?

The necessary code should be copied from x/sys/cpu to internal/cpu - re making a system call, there is prior art for this in the form of sysctl - see cpu_arm64_darwin.go or cpu_arm64_openbsd.go for example.

[markdryan]

HasMisalignedFast seems awkward - HasFastMisaligned at least matches the comment, or we potentially drop the Has and use FastMisaligned or MisalignedIsFast...

How about IsMisalignedFast? The fields in the existing structures in src/internal/cpu.go all start with either 'Is' or 'Has' so this would be consistent with what's already there for the other architectures and hopefully, a bit more readable.

Re HasZB*, given that they're usually referered to as mixed case (Zba and not ZBA), it would probably be better to stick with that here and use HasZb[abs].

Agreed. This is better. I'll change this.

The necessary code should be copied from x/sys/cpu to internal/cpu - re making a system call, there is prior art for this in the form of sysctl - see cpu_arm64_darwin.go or cpu_arm64_openbsd.go for example.

I used cpu_arm64_openbsd.go as a model when creating https://go.dev/cl/522995

[gopherbot]

Change https://go.dev/cl/530895 mentions this issue: unix: update riscv_hwprobe constants

[gopherbot]

Change https://go.dev/cl/605815 mentions this issue: cpu: add support for detecting RISC-V extensions

[4a6f656c]

Since this does not need to be a proposal (confirmed by #61416 (comment)), I'm going to turn it into a regular tracking issue.