Overclocking partially broken with DVFS firmware

[profi200]

Describe the bug
Overclocking is broken and causes freezes before even the rainbow splash appears.

To reproduce
Update kernel to 4.19.83 and the matching firmware.
Add
arm_freq=1800 gpu_freq=600 over_voltage=2
to config.txt and reboot.

Expected behaviour
It should boot like it did before the update (this may not work for every board because silicon lottery but it was rock solid for this one).

Actual behaviour
Blinks the green LED a few times (not steady blinking but SD card access blinking) and hangs without any sign of life. No rainbow splash either.

System

• Official Raspberry Pi USB-C power supply
• SandDisk Ultra A1 64 GB microSD card
• Raspberry Pi4 B 4 GB RAM
• Manjaro ARM 19.10
• vcgencmd doesn't work on this Arch Linux based 64 bit distro so not sure which firmware version. If you know a way to read this out via other means please let me know.
• Linux raspi4 4.19.83-1-MANJARO-ARM #1 SMP PREEMPT Tue Nov 12 15:23:51 UTC 2019 aarch64 GNU/Linux
• Latest pieeprom and vl805 firmware (WAKE_ON_GPIO=0 POWER_OFF_ON_HALT=1) downloaded from the download section on raspberrypi.org and flashed via the recovery method. This was also broken with the stock bootloader the Pi4 arrived with (the very first one).

Logs
Not applicable.

Additional context
I know thirdparty OSs are not supported but this uses the official kernel (rebranded) and firmware.

config.txt:

# See /boot/overlays/README for all available options

gpu_mem=64
initramfs initramfs-linux.img followkernel
kernel=kernel8.img
arm_64bit=1
enable_gic=1
disable_overscan=1

#enable sound
dtparam=audio=on
#hdmi_drive=2

#enable vc4
dtoverlay=vc4-fkms-v3d
max_framebuffers=2

#arm_freq=2000
arm_freq=1800
gpu_freq=600
#over_voltage=4
over_voltage=2

cmdline.txt:
root=/dev/mmcblk0p2 rootflags=defaults,noatime,discard,data=ordered rw rootwait console=ttyAMA0,115200 console=tty1 selinux=0 plymouth.enable=0 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=ttyAMA0,115200 elevator=noop snd-bcm2835.enable_compat_alsa=0

[tvollmann]

Same on standard raspbian with latest updates.

After mounting the card on another computer and commenting out the overclocking options it booted just fine

[gilesn]

I found that gpu_freq was preventing Raspbian booting, commenting it out and keeping all other overclocking settings worked OK.

[popcornmix]

There has been a substantial rewrite of code that manages clocks and power. See:
https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=257394

Remove gpu_freq overclock options for now.
We'll try to work out how to get gpu side overclocking working,
but that will be lower priority than bugs affecting non-overclock use cases.

[profi200]

Thank you for taking this seriously even though this a thirdparty distribution (with original, untouched firmware included).

I can confirm commenting out gpu_freq=600 works for now. Causes a significant drop in performance for GPU heavy applications like Dolphin (emulator) though which is to be expected.

[popcornmix]

Does it work with gpu_freq=600 and core_freq=500 ?
I suspect you care more about v3d frequency than core.

[profi200]

Just tried this and indeed this does boot and is stable.

What benefits would increasing core_freq bring? In the forums i read something about faster L2 cache awhile ago but maybe i'm mixing this up.

[popcornmix]

Faster L2 only affects Pi0/1 where the ARM doesn't have it's own L2 and uses GPU's.

Faster core helps the AXI bus which can allow higher sdram bandwidth, but you probably will only see a measurable benefit with very heavy sdram usage (e.g. 4K video decode with multiple reference frames).

Presumably you can very core_freq (with the previous firmware) and determine if it benefits emulation (e.g. in fps displayed). My guess would be arm frequency is most important, v3d frequency is secondary with games that use GL. Core is probably only of minor importance.

[profi200]

Makes sense, thanks. core will probably improve compiling time a bit since this is very memory heavy.

It actually varies per game. But generally from what i have seen the GPU is the bottleneck most of the time. But the CPU has a big impact aswell as can be seen by reducing the clock of the emulated CPU (it's a setting in Dolphin). Most games will get 5-15 more fps this way but some start dropping frames which is not surprising if you don't give them enough time to render. Overal speed varies between 40-80%. Overclocking the GPU gives another 10-20%. Probably only Vulkan can give more fps. Anyway, this is a bit offtopic. I will play with the settings more later.

[popcornmix]

Latest rpi-update firmware should fix the issue with being unable to boot with core_freq overclocked.
Note: you will get the highest core frequency that is possible with the current PLL settings.
That will be 550 when hdmi_enable_4kp60=1 and 500 otherwise.

[profi200]

Question regarding this update:
How was it handled before DVFS got introduced? Was 600 MHz always broken and worked by luck? Or was that actually working before?

[roslof]

Faster L2 only affects Pi0/1 where the ARM doesn't have it's own L2 and uses GPU's.

Faster core helps the AXI bus which can allow higher sdram bandwidth, but you probably will only see a measurable benefit with very heavy sdram usage (e.g. 4K video decode with multiple reference frames).

Presumably you can very core_freq (with the previous firmware) and determine if it benefits emulation (e.g. in fps displayed). My guess would be arm frequency is most important, v3d frequency is secondary with games that use GL. Core is probably only of minor importance.

Overclocking the GPU significantly improves emulated 3D games. Without overclocking, some emulated games (like Dreamcast/Soul Calibur and N64/...anything) run significantly slower and don't hit framerate -- even on a RaspPi4B.

Bumping GPU to 600 MHz or 750 MHz on the old kernel made these games run like butter. Without the ability to overclock the GPU, the games all stutter and can't reliably hit framerate.

[roslof]

Question regarding this update:
How was it handled before DVFS got introduced? Was 600 MHz always broken and worked by luck? Or was that actually working before?

Worked fine for all previous kernels until November. Now that the DVFS kernel update is part of the standard Raspbian update, I suspect more and more people will update and find their systems frozen on boot (if they had the GPU OCed in config.txt).

[rhunjadi]

Faster L2 only affects Pi0/1 where the ARM doesn't have it's own L2 and uses GPU's.
Faster core helps the AXI bus which can allow higher sdram bandwidth, but you probably will only see a measurable benefit with very heavy sdram usage (e.g. 4K video decode with multiple reference frames).
Presumably you can very core_freq (with the previous firmware) and determine if it benefits emulation (e.g. in fps displayed). My guess would be arm frequency is most important, v3d frequency is secondary with games that use GL. Core is probably only of minor importance.

Overclocking the GPU significantly improves emulated 3D games. Without overclocking, some emulated games (like Dreamcast/Soul Calibur and N64/...anything) run significantly slower and don't hit framerate -- even on a RaspPi4B.

Bumping GPU to 600 MHz or 750 MHz on the old kernel made these games run like butter. Without the ability to overclock the GPU, the games all stutter and can't reliably hit framerate.

Can confirm that. Hope we'll get back the ability to overclock again.

[timothybrown]

Question regarding this update:
How was it handled before DVFS got introduced? Was 600 MHz always broken and worked by luck? Or was that actually working before?

Worked fine for all previous kernels until November. Now that the DVFS kernel update is part of the standard Raspbian update, I suspect more and more people will update and find their systems frozen on boot (if they had the GPU OCed in config.txt).

Can confirm this is exactly what happened to me. Really disappointed this was pushed live. (Seriously, the ability to overclock the GPU was crucial to a section of the emulation scene. We're seeing a 20 FPS drop in frame rates since loosing the overclock on certain systems.)

[profi200]

What if you try "v3d_freq=600"? Still slower than previously?

[timothybrown]

Using v3d_freq=600 also locks up the system on boot for me.

[JamesH65]

Interesting. Not much else can be done at the moment then. With regard to your previous comment,. there have never been any guarantee that overclocking will work, or continue to work in the future. We'll try and sort something out, but it won't be straight away. This release was tested thoroughly with standard settings and has been available for external beta testing for a month. We don't test non-standard setting to the same level, because, well, the phrase non-standard is a giveaway.

[JamesH65]

Just out of interest, what specific part of the GPU overclock are you relying on? https://www.raspberrypi.org/documentation/configuration/config-txt/overclocking.md shows there are 4 options encompassed by 'gpu_freq', two appear to be irrelevant (H264, ISP), so it's either v3d or core_freq or both.

[timothybrown]

Interesting. Not much else can be done at the moment then. With regard to your previous comment,. there have never been any guarantee that overclocking will work, or continue to work in the future. We'll try and sort something out, but it won't be straight away. This release was tested thoroughly with standard settings and has been available for external beta testing for a month. We don't test non-standard setting to the same level, because, well, the phrase non-standard is a giveaway.

I understand there are no guarantees with overclocking. (In fact, hardware wise, it’s basically a lottery as to how far you can push any particular board.)

Like you said, this release has been available for beta testing for some time, and likewise the issue has been known since at least November. Which is my very point: Why push out firmware that knowingly breaks something that was already working? Wouldn’t it have been possible to add an option that disables the new voltage and frequency scaling if any of the GPU clocks were set at higher than standard values? In theory this seems like a simple solution to me, but then again I haven’t actually looked at the code itself, so I don’t really know.

I don’t want it to seem like I’m complaining, because that’s not my intention. I’m an embedded developer myself, so I totally get it and I do appreciate the hard work everyone puts in.

Just out of interest, what specific part of the GPU overclock are you relying on? https://www.raspberrypi.org/documentation/configuration/config-txt/overclocking.md shows there are 4 options encompassed by 'gpu_freq', two appear to be irrelevant (H264, ISP), so it's either v3d or core_freq or both.

Yes, sorry, I should have been more specific. It’s the V3D portion that’s of primary concern.

I should also add that I had some time to do more testing today and I don’t think the loss of the V3D overclock is solely responsible for the entire drop in frame-rate we’re seeing. (I believe part of the issue is related to the new Mesa drivers.)

I’d say, at the end of the day, the V3D overclock gives about a 10 FPS boost in emulators like Mupen64. That’s still fairly substantial, but not as bad of a hit as originally thought.

Another thing to keep in mind is that all these drivers and emulators are still far from optimized for the Pi 4. This is all still bleeding edge. Once all that happens it might be fast enough that overclocking the GPU won’t be needed.

Finally, as I mentioned before I’m an embedded developer (with bare metal assembly and C experience on the ARM platform), so if there’s anything I can do to help (regarding this or any other issue), please let me know!

[JamesH65]

Having spoken to the experts on this, there does not appear to be a way to allow GPU overclocking with the DVFS firmware. AIUI (and this is me paraphrasing) it's a limitation of the hardware in that to do the DVFS we need to reallocate clocks and PLLS which means that its no longer possible to overclock the GPU as there is no longer the available dividers on the clocks/PLLS to give variable values. Making the code switchable is also not possible due to its complexity.

So we pushed this firmware out in the knowledge the GPU overclocking would be disabled, because we knew it would not be possible to enable it in the future.

Sorry, not what you want to hear, but nothing can be done. Looks like optimisation is the remaining option. We are continually improving the MESA drivers, which may gain something. Emulators are down to their maintainers.

[timothybrown]

Thanks @JamesH65, I appreciate you looking into this. You’re right, not what I wanted to hear, but hey that’s life, right? To be honest, I suspected the answer would be along these lines.

At the end of the day the DVFS firmware will be very beneficial to the majority of Pi users. Compare that with the tiny number that would actually need to overclock the GPU and the choice is pretty clear. :)

Someone should update the relevant section in the config.txt -> Overclocking section of the documentation. Perhaps something along the lines of: “The DVFS firmware update (January 2020) for the Pi 4B has permanently removed support for GPU overclocking on this platform.”

Just so we can have an official place to point people to.

[popcornmix]

Overclocking core and gpu frequencies has never been supported on pi4 (even before dvfs changes). It just happened that 600 was a magic number that had an integer divider from the channel frequency of the PLL. Even on older firmware, 610 would never have worked (however fast your Pi4 was).

The PLL/clock configuration is subject to change if we find settings that improve power consumption, performance or stability. And settings used may affect whether an overclock still works.

As @JamesH65 says, switching off dvfs is not feasible as the clocking strategy is now substantially different.

Currently v3d clock comes from PLLA=3GHz with a channel divisor of 6, and a clock divider of 2.
Requesting v3d_freq=600 will result in a fractional divider of 1.666 which means some clock pulses are effectively at 1GHz - no Pi4 will work like that. Rounding up the fractional divider should make this safe again, but you'd only have a v3d_freq of 500MHz.

The difficulty is that each PLL and channel divisor is shared by multiple clocks (e.g. v3d, isp, h264, hevc share PLLA, and many clocks share PLLC with core).

I'll try to have a look if we can do anything better. It should be possible to detect if an overclock requested has a non-integer divider that effectively exceeds the requested frequency and reduce the overclock so it doesn't. That will stop the crashing/failing to boot when things change.

I'll see if some limited gpu overclock could be supported. e.g. if v3d, isp, h264, hevc are all overclocked the same then increasing PLLA but leaving the dividers the same may be possible (no guarantees it will work on any specific Pi4, but it's something that can be tried).

Overclocking core is probably not possible - things like wifi, sdcard, composite, hdmi rely on PLLC and tend to fail when run with clocks that aren't as expected.

[JamesH65]

Not going to get a better explanation than that from@popcornmix - thanks! I'll update the docs tomorrow.

[profi200]

That's unfortunate. I guess the only option is downgrading for the extra 50 MHz (if that doesn't make it worse because of compatibility with recent kernels).

[JamesH65]

@profi200 Best to simply get used to not being able to overclock the GPU, which was never a guaranteed feature anyway. Using old firmware will soon mean ARM side updates will fail to work correctly.

ARM overclocking is the major source of improvement anyway.

EDIT: Just been informed that there may be a possibility that some levels of overclocks for things like H264/3D might be possible - currently testing. Core frequencies probably not. However, there is no guarantee that overclocks will work on any specific Pi4, and once again, future firmware changes may mean the facilities to overclock GPU blocks will reduce.

[profi200]

Yes, but the GPU is bottlenecking a bit for more demanding emulators. This can and will improve with driver updates but a wonder can't be expected (it is what it is but Vulkan may be a game changer when it's ready in the future). That's why the retro gaming people are so desperate for these 50 MHz.

To be honest this was what i expected considering you can't cram a NVIDIA tegra level of GPU in a SoC of a $35 computer. It doesn't work at all at this price point. It's fine for the price.

[popcornmix]

I have a test firmware that should better support gpu overclock here

core_freq still has usual restrictions. v3d/h264/hevc/isp can be overclocked.
Try gpu_freq=600 and if that works try setting it higher (will likely need over_voltage for higher frequencies)

[pelwell]

Note that only a limited set of overclock frequencies can be requested (550, 600, 750, 800 and 850), and there are no guarantees that any of the higher frequencies are achievable.

[popcornmix]

Arbitrary frequencies are possible:

pi@pi4d:~ $ grep gpu_freq /boot/config.txt 
gpu_freq=787
pi@pi4d:~ $ vcgencmd measure_clock v3d
frequency(46)=787007808

[quicksilver7837]

With test firmware I was able to boot successfully with gpu_freq=700. As noted above the core frequency is still locked to 500mhz but v3d frequency is being reported as 700mhz. Nice!

However in testing mupen64plus-next (N64) and flycast (dreamcast) emulators I can find no appreciable FPS increase by overclocking v3d alone. Im not convinced that v3d is a bottleneck on the pi 4 at least for these emulators. On the pi 3 overclocking v3d also did not yield any appreciable performance gains with the above emulators but overclocking the core frequency did. I would be curious to see some examples where v3d alone improves performance.

[popcornmix]

Note: you can get core_freq=550MHz with hdmi_enable_4kp60=1 (on a Pi4).
Use "vcgencmd measure_clock v3d" to be sure the v3d is clocked as requested.

[timothybrown]

With test firmware I was able to boot successfully with gpu_freq=700. As noted above the core frequency is still locked to 500mhz but v3d frequency is being reported as 700mhz. Nice!

However in testing mupen64plus-next (N64) and flycast (dreamcast) emulators I can find no appreciable FPS increase by overclocking v3d alone. Im not convinced that v3d is a bottleneck on the pi 4 at least for these emulators. On the pi 3 overclocking v3d also did not yield any appreciable performance gains with the above emulators but overclocking the core frequency did. I would be curious to see some examples where v3d alone improves performance.

@quicksilver7837
While I don’t think this is the place to discuss specifics, I will say lr-n64-parallel and lr-flycast both show up to an 8FPS and 5FPS bump in certain games with V3D overclocked for me.

I have a test firmware that should better support gpu overclock here

core_freq still has usual restrictions. v3d/h264/hevc/isp can be overclocked.
Try gpu_freq=600 and if that works try setting it higher (will likely need over_voltage for higher frequencies)

@popcornmix
Thanks for taking the time to work on this! I’ll continue to test the firmware and report back with any issues.

[roslof]

The Soul Calibur full intro yields solid improvement over stock. Using the same OC settings from the pre-DVFS kernel to the test kernel, the test kernel is in the ball-park.

Love this!

Thank you @popcornmix ! Wasn't expecting this and it's a welcome surprise!

[profi200]

From my short/limited testing of the uploaded firmware files i can say pre-DVFS performance is back! I can even go a little higher (but the improvement is pretty low above 600 MHz). With over_voltage=2 the gpu_freq can go up to 650. 700 makes the GPU wonky and there are random hangs for 1-2 seconds (probably needs more voltage).
If anyone is testing performance don't forget to set the CPU scaling governor to "performance" because "ondemand" doesn't work very well for emulation. And keep an eye on the temperature if you do so!

The most important part is no freeze at boot anymore with gpu_freq >550 (of course it will still break if set too high).

I will later do more testing with higher over_voltage. So far this is looking very good and i appreciate the attention and work from the RPi Foundation for this issue!

[JamesH65]

Good news! Note, it's Raspberry Pi Trading, not Foundation, who do this stuff.

[rhunjadi]

@popcornmix U'r the greatest! :)

[profi200]

Ok, later is a bit earlier than expected.
With over_voltage=4 i can go to gpu_freq=760. 775 is already wonky under load so 750 is probably the safe sweet spot (it of course depends on silicon lottery).

Raising the voltage even more doesn't make sense i think because the PMIC is already at the limit at over_voltage=2. Higher is not 100% stable as the Gimp unsharpen filter discovery on the forums shows.

And sorry, i of course meant Raspberry Pi Trading. It's easy to confuse them :)

edit:
Using v3d_freq instead of gpu_freq gives the exact same boost. So this is probably enough for all our needs.

edit2:
Just for documentations sake: over_voltage=6 + v3d_freq=830 seems stable under load. Dolphin with Super Mario Sunshine reaches 93-95% on title screen now. Previously it was at around 75-80%.

[Brunnis]

However in testing mupen64plus-next (N64) and flycast (dreamcast) emulators I can find no appreciable FPS increase by overclocking v3d alone. Im not convinced that v3d is a bottleneck on the pi 4 at least for these emulators.

Yeah, I think it's just not GPU limited. I did a quick test with this latest firmware, running lr-mupen64-next and GoldenEye (beginning of first level). I set the emulator FPS option to "Fullspeed" and disabled both vsync and audio sync to remove any artificial FPS blockers:

Default: 46 FPS
GPU@600: 48 FPS
GPU@600 + CPU@1800: 55 FPS

[profi200]

Yeah, not all emus will benefit from this. I have tested with Dolphin as mentioned above which is a mixed bag. Some games are CPU intensive and others GPU. I clearly saw a big improvement on the GPU side.

Also a reminder that the ondemand governor is not fast enough to keep up with fast load changes (which is exactly what you get with emulation). It caused lots of frame drops in my test. Change it to performance and see if you get better results. You can either use the cpupower tool or
# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

I see the new firmware has been pushed. Again, thank you to Raspberry Pi Trading for the work. I must admit the Pi's still have the best support of all SBCs. Other SBC makers probably would not care about this. This issue went from not much hope to better than ever before :)

[pelwell]

I must admit the Pi's still have the best support of all SBCs. Other SBC makers probably would not care about this.

Thanks - tell your friends.

Can we close this issue now?

[profi200]

I have not found any issues with this firmware. And looks like it works for other people aswell. So i'm going ahead and close this.

[Brunnis]

Also a reminder that the ondemand governor is not fast enough to keep up with fast load changes (which is exactly what you get with emulation). It caused lots of frame drops in my test. Change it to performance and see if you get better results. You can either use the cpupower tool or
# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

My tests were actually with performance governor enabled.

[profi200]

Ok. That should give consistent results then.

[francocalvo]

Sorry, I might not have understood since English isn't my first language. Do I need to install the firmware that's linked here in order to be able to overclock the GPU or is it possible with the new firmware? My Pi boots but it hangs after a minute or two.

[popcornmix]

Any improvements in firmware linked here will be in recent official firmware (check "vcgencmd version" has a date newer than date of that the version was linked).

No overclock settings are guaranteed - hanging is the expected behaviour if you are trying to overclock higher than your specific board can cope with.