Disable CQE on mmc0 to fix I/O freezes on Yellow+CM5 #3705
Conversation
The I/O operations on the eMMC can sometimes fail and lock up completely, and disabling CQE on the sdio1 (mmc0) interface seems to solve the issue. While it is a known (and potentially resolved) issue [1] for SD cards in Raspberry Pi's Linux fork, it is not acknowledged neither resolved for CM5's eMMC. With CQE enabled, the device usually locks up within the first 10 first boots, when the swap file is being created. After disabling CQE, no error occurred after more that 100 cold boots (every time with swap file removed). [1] https://github.com/raspberrypi/linuxissues/6349
📝 WalkthroughWalkthroughThe pull request introduces a patch to the device tree source file for the Broadcom BCM2712 Raspberry Pi Compute Module 5 (CM5). It disables the Command Queue Engine (CQE) feature on the eMMC interface due to issues causing random hangs during I/O operations, particularly at boot time. The specific change involves removing the Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (1)
buildroot-external/board/raspberrypi/yellow/patches/linux/0019-ARM-dts-bcm2712-yellow-Disable-CQE-on-eMMC-interface.patch (1)
30-32: Consider broader hardware impact and upstream reporting.Since similar issues exist with SD cards but haven't been acknowledged for eMMC, consider:
- Investigating if this issue affects other BCM27xx boards with eMMC
- Reporting this issue upstream to the Raspberry Pi Linux repository with the collected evidence
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
buildroot-external/board/raspberrypi/yellow/patches/linux/0019-ARM-dts-bcm2712-yellow-Disable-CQE-on-eMMC-interface.patch(1 hunks)
🔇 Additional comments (2)
buildroot-external/board/raspberrypi/yellow/patches/linux/0019-ARM-dts-bcm2712-yellow-Disable-CQE-on-eMMC-interface.patch (2)
9-32: Well-documented problem and solution!
The commit message effectively documents the issue with concrete evidence from system logs, showing multiple tasks being blocked. The correlation with known SD card issues and the validation of the fix through extensive testing (>100 cold boots) provides strong justification for this change.
43-49: LGTM! Consider monitoring performance impact.
The removal of CQE support is a clean, minimal change that maintains other essential eMMC features. While CQE typically improves performance through command queuing, the stability improvement justifies its removal.
Consider monitoring eMMC performance metrics before and after this change:
✅ Verification successful
Let me generate additional scripts to search for any existing performance monitoring or testing infrastructure related to eMMC/storage.
Let me try one final search focusing specifically on the device tree and kernel configurations related to eMMC and CQE.
Verified: CQE removal is justified due to stability issues
The patch commit message and related changes in the codebase confirm that CQE was causing random I/O hangs, and its removal was a deliberate stability improvement. While performance monitoring would be beneficial, the critical stability issues outweigh potential performance benefits. The change is minimal and maintains other essential eMMC features.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Search for any performance-related configurations or tests in the codebase
# Look for eMMC/SD performance test files
fd -t f -e sh -e py . | rg -i "test|bench|perf" | rg -i "mmc|sd|emmc"
# Look for related performance documentation
rg -i "performance|benchmark" -g "*.md" -A 5 -B 5
Length of output: 1226
Script:
#!/bin/bash
# Search for storage/eMMC related monitoring or testing code
rg -i "mmc|emmc|storage" -g "*.sh" -g "*.py" -g "*.c" -A 3 -B 3
# Look for any existing monitoring or diagnostic tools
fd -t f monitoring diagnostics tests
# Search for storage performance related configurations
rg -i "mmc|emmc" -g "*.conf" -g "*.yaml" -g "*.yml"
Length of output: 1368
Script:
#!/bin/bash
# Search for CQE and eMMC related configurations in device trees and kernel configs
rg -i "cqe|command.?queue" -g "*.dts" -g "*.dtsi" -A 3 -B 3
# Look for kernel config options related to eMMC CQE
rg -i "MMC.*CQE|CQE.*MMC" -g "defconfig" -g "*.config"
# Search for any performance-related comments in eMMC patches
fd -t f -e patch . | xargs rg -i "cqe|command.?queue|performance"
Length of output: 3130
The I/O operations on the eMMC can sometimes fail and lock up completely, and disabling CQE on the sdio1 (mmc0) interface seems to solve the issue. While it is a known (and potentially resolved) issue [1] for SD cards in Raspberry Pi's Linux fork, it is not acknowledged neither resolved for CM5's eMMC. With CQE enabled, the device usually locks up within the first 10 first boots, when the swap file is being created. After disabling CQE, no error occurred after more that 100 cold boots (every time with swap file removed). [1] https://github.com/raspberrypi/linuxissues/6349 (cherry picked from commit c514d6b)
The I/O operations on the eMMC can sometimes fail and lock up completely, and disabling CQE on the sdio1 (mmc0) interface seems to solve the issue. While it is a known (and potentially resolved) issue [1] for SD cards in Raspberry Pi's Linux fork, it is not acknowledged neither resolved for CM5's eMMC. With CQE enabled, the device usually locks up within the first 10 first boots, when the swap file is being created. After disabling CQE, no error occurred after more that 100 cold boots (every time with swap file removed).
[1] https://github.com/raspberrypi/linuxissues/6349
Summary by CodeRabbit