[Bug]: Enabling PMU section in k230.dtsi causes boot to hang after initializing GPIO

[ardangelo]

What happened

Evaluating suspend mode on CanMV K230. Starting by rebuilding image with PMU support.
Boot starts but hangs after output lines:

[    1.665207] gpio gpiochip6: (9140c000.gpio53): added GPIO chardev (254:6)
[    1.672057] gpio gpiochip6: registered GPIOs 53 to 53 on 9140c000.gpio53

No errors related to the PMU are reported here.
Upstream DTSI with commented-out PMU section boots successfully.

Reproduction steps

• Start Docker build container
• Edit the DTSI src/little/linux/arch/riscv/boot/dts/kendryte/k230.dtsi
• Change status from disabled to okay
• Uncomment the commented-out PMU sections
• Build the image with make CONF=k230_canmv_defconfig build-image
• Flash and boot

Hardware board

CanMV K230

Software version

bee8707

Bug frequency

Every boot

Anything else

Boot log:

OpenSBI v0.9
[    0.000000] Linux version 5.10.4 (root@53bbbc2e84e9) (riscv64-unknown-linux-gnu-gcc (Xuantie-900 linux-5.10.4 glibc gcc Toolchain V2.6.0 B-20220715) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 SMP Mon Nov 18 08:38:34 CST 2024
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Initial ramdisk at: 0x(____ptrval____) (23576576 bytes)
[    0.000000] cma: Reserved 52 MiB at 0x000000000c800000
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000008200000-0x000000000fffefff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000008200000-0x000000000fffefff]
[    0.000000] Initmem setup node 0 [mem 0x0000000008200000-0x000000000fffefff]
[    0.000000] On node 0 totalpages: 32255
[    0.000000]   DMA32 zone: 441 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 32255 pages, LIFO batch:7
[    0.000000] SBI specification v0.3 detected
[    0.000000] SBI implementation ID=0x1 Version=0x9
[    0.000000] SBI v0.2 TIME extension detected
[    0.000000] SBI v0.2 IPI extension detected
[    0.000000] SBI v0.2 RFENCE extension detected
[    0.000000] SBI v0.2 HSM extension detected
[    0.000000] riscv: ISA extensions acdefhimtv
[    0.000000] riscv: ELF capabilities acdfimv
[    0.000000] percpu: Embedded 25 pages/cpu s65112 r8192 d29096 u102400
[    0.000000] pcpu-alloc: s65112 r8192 d29096 u102400 alloc=25*4096
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 31814
[    0.000000] Kernel command line: root=/dev/mmcblk1p3 loglevel=8 rw rootdelay=4 rootfstype=ext4 console=ttyS0,115200 crashkernel=256M-:128M earlycon=sbi
[    0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000000] Sorting __ex_table...
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 29336K/129020K available (9098K kernel code, 4867K rwdata, 4096K rodata, 271K init, 370K bss, 46436K reserved, 53248K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] plic: interrupt-controller@f00000000: mapped 200 interrupts with 1 handlers for 2 contexts.
[    0.000000] random: get_random_bytes called from start_kernel+0x388/0x4fe with crng_init=0
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x63a1e71a3, max_idle_ns: 440795203123 ns
[    0.000006] sched_clock: 64 bits at 27MHz, resolution 37ns, wraps every 4398046511093ns
[    0.008551] Console: colour dummy device 80x25
[    0.012984] Calibrating delay loop (skipped), value calculated using timer frequency.. 54.00 BogoMIPS (lpj=108000)
[    0.023348] pid_max: default: 32768 minimum: 301
[    0.028165] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.035334] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.044486] ASID allocator initialised with 65536 entries
[    0.050020] rcu: Hierarchical SRCU implementation.
[    0.055110] EFI services will not be available.
[    0.059758] smp: Bringing up secondary CPUs ...
[    0.064244] smp: Brought up 1 node, 1 CPU
[    0.068763] devtmpfs: initialized
[    0.091718] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.101433] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.109933] NET: Registered protocol family 16
[    0.115292] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.122381] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.130791] thermal_sys: Registered thermal governor 'step_wise'
[    0.329589] [K230_RESET]:sysctl reset phy addr 0x91101000
[    0.335564] [K230_RESET]:k230_reset_probe ok!
[    0.368064] k230-powerdomain 91103000.sysctl_power: powerdomain init ok
[    0.379624] SCSI subsystem initialized
[    0.383619] usbcore: registered new interface driver usbfs
[    0.389130] usbcore: registered new interface driver hub
[    0.394478] usbcore: registered new device driver usb
[    0.399660] mc: Linux media interface: v0.10
[    0.403934] videodev: Linux video capture interface: v2.00
[    0.410249] Canaan Hard Lock Driver init.
[    0.415717] clocksource: Switched to clocksource riscv_clocksource
[    1.380725] NET: Registered protocol family 2
[    1.385738] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    1.394101] TCP established hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    1.401782] TCP bind hash table entries: 1024 (order: 2, 16384 bytes, linear)
[    1.408994] TCP: Hash tables configured (established 1024 bind 1024)
[    1.415496] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    1.422022] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    1.429204] NET: Registered protocol family 1
[    1.434265] RPC: Registered named UNIX socket transport module.
[    1.440168] RPC: Registered udp transport module.
[    1.444901] RPC: Registered tcp transport module.
[    1.449669] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.457379] Unpacking initramfs...
[    1.460777] Initramfs unpacking failed: invalid magic at start of compressed archive
[    1.477394] Freeing initrd memory: 23020K
[    1.482453] Initialise system trusted keyrings
[    1.487150] workingset: timestamp_bits=62 max_order=15 bucket_order=0
[    1.502029] NFS: Registering the id_resolver key type
[    1.507092] Key type id_resolver registered
[    1.511294] Key type id_legacy registered
[    1.515500] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.522193] ntfs: driver 2.1.32 [Flags: R/W].
[    1.526935] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[    1.533743] fuse: init (API version 7.32)
[    1.538165] 9p: Installing v9fs 9p2000 file system support
[    1.544065] NET: Registered protocol family 38
[    1.548480] Key type asymmetric registered
[    1.552615] Asymmetric key parser 'x509' registered
[    1.557611] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    1.565034] io scheduler mq-deadline registered
[    1.569622] io scheduler kyber registered
[    1.575366] k230-gpio 9140b000.gpio1: request hardlock 0 success!
[    1.581697] gpio gpiochip0: (9140b000.gpio1): added GPIO chardev (254:0)
[    1.588485] gpio gpiochip0: registered GPIOs 1 to 1 on 9140b000.gpio1
[    1.595340] gpio gpiochip1: (9140b000.gpio23): added GPIO chardev (254:1)
[    1.602199] gpio gpiochip1: registered GPIOs 23 to 23 on 9140b000.gpio23
[    1.609345] gpio gpiochip2: (9140b000.gpio27): added GPIO chardev (254:2)
[    1.616208] gpio gpiochip2: registered GPIOs 27 to 27 on 9140b000.gpio27
[    1.623306] gpio gpiochip3: (9140c000.gpio33): added GPIO chardev (254:3)
[    1.630168] gpio gpiochip3: registered GPIOs 33 to 33 on 9140c000.gpio33
[    1.637265] gpio gpiochip4: (9140c000.gpio34): added GPIO chardev (254:4)
[    1.644119] gpio gpiochip4: registered GPIOs 34 to 34 on 9140c000.gpio34
[    1.651235] gpio gpiochip5: (9140c000.gpio52): added GPIO chardev (254:5)
[    1.658085] gpio gpiochip5: registered GPIOs 52 to 52 on 9140c000.gpio52
[    1.665207] gpio gpiochip6: (9140c000.gpio53): added GPIO chardev (254:6)
[    1.672057] gpio gpiochip6: registered GPIOs 53 to 53 on 9140c000.gpio53

[ardangelo]

Driver at src/little/linux/drivers/soc/kendryte/k230-pmu.c is hanging at this line:

pmu_status = readl(pmu->regs + PMU_STATUS);

[ardangelo]

I also tried suspending on the 6.6.36 dev branch of https://github.com/kendryte/k230_linux_sdk
Adding to the kernel config

CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_PM_SLEEP=y

SD card controller fails to suspend.

[root@canaan ~ ]#echo mem | tee /sys/power/state
mem[  175.899938] PM: suspend entry (s2idle)

[  175.948482] Filesystems sync: 0.044 seconds
[  175.953659] Freezing user space processes
[  175.958839] Freezing user space processes completed (elapsed 0.001 seconds)
[  175.965839] OOM killer disabled.
[  175.969083] Freezing remaining freezable tasks
[  175.974683] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[  175.982121] printk: Suspending console(s) (use no_console_suspend to debug)
[  175.990200] [dhd] bcmsdh_sdmmc_suspend Enter func->num=2
[  175.990213] [dhd] dhdsdio_suspend Enter
[  175.990217] [dhd] bcmsdh_sdmmc_suspend: can't keep power while host is suspended
[  175.990231] bcmsdh_sdmmc mmc0:0001:2: PM: dpm_run_callback(): pm_generic_suspend+0x0/0x26 returns -22
[  175.990265] bcmsdh_sdmmc mmc0:0001:2: PM: failed to suspend async: error -22
[  176.038043] PM: Some devices failed to suspend, or early wake event detected
[  176.098962] OOM killer enabled.
[  176.102112] Restarting tasks ... 
[  176.103364] r8152-cfgselector 2-1: USB disconnect, device number 2
[  176.113370] [dhd] sdioh_remove: Enter
[  176.117116] done.
[  176.119087] random: crng reseeded on system resumption
[  176.124860] [dhd] dhdsdio_disconnect : no mutex held
[  176.129858] [dhd] dhdsdio_disconnect : set mutex lock
[  176.134950] PM: suspend exit