Workqueue: kmmcd mmc_rescan crash with Raspbian kernel 4.4.38-v7

[fkbreitl]

My Raspberry Pi 3 runs unstable and crashes every few days. Before the crash /var/log/messages shows the following error repeatedly with a separation of a few minutes:

Jan 25 18:36:22 raspberrypi kernel: [1103404.218839] kworker/u8:2 D 805b770c 0 20119 2 0x00000000
Jan 25 18:36:22 raspberrypi kernel: [1103404.218886] Workqueue: kmmcd mmc_rescan
Jan 25 18:36:22 raspberrypi kernel: [1103404.218923] [<805b770c>] (__schedule) from [<805b7c84>] (schedule+0x50/0xa8)
Jan 25 18:36:22 raspberrypi kernel: [1103404.218945] [<805b7c84>] (schedule) from [<8046d568>] (__mmc_claim_host+0xb8/0x1cc)
Jan 25 18:36:22 raspberrypi kernel: [1103404.218974] [<8046d568>] (__mmc_claim_host) from [<8046d6ac>] (mmc_get_card+0x30/0x34)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219006] [<8046d6ac>] (mmc_get_card) from [<80475780>] (mmc_sd_detect+0x2c/0x80)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219030] [<80475780>] (mmc_sd_detect) from [<8046fcbc>] (mmc_rescan+0xc8/0x324)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219057] [<8046fcbc>] (mmc_rescan) from [<8003c82c>] (process_one_work+0x154/0x458)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219080] [<8003c82c>] (process_one_work) from [<8003cb84>] (worker_thread+0x54/0x500)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219100] [<8003cb84>] (worker_thread) from [<80042860>] (kthread+0xec/0x104)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219122] [<80042860>] (kthread) from [<8000fbc8>] (ret_from_fork+0x14/0x2c)
Jan 25 18:36:23 raspberrypi rsyslogd-2007: action 'action 17' suspended, next retry is Wed Jan 25 18:37:53 2017 [try http://www.rsyslog.com/e/2007 ]

I am running Raspbian 8.0 (Jessie) with kernel

Linux raspberrypi 4.4.38-v7+ #938 SMP Thu Dec 15 15:22:21 GMT 2016 armv7l GNU/Linux

Could this be a firmware problem?

I am also asking this at

http://raspberrypi.stackexchange.com/questions/61050/workqueue-kmmcd-mmc-rescan-crash-with-raspbian-kernel-4-4-38-v7 .

[JamesH65]

@fkbreitl Is this still an issue with the latest kernels?

[sunshineco]

I'm not the OP, but my Pi 3 experiences this same problem every few days, as well, with a much more recent kernel:

Linux pi.local 4.9.59-v7+ #1047 SMP Sun Oct 29 12:19:23 GMT 2017 armv7l GNU/Linux

[JamesH65]

Please update to the latest kernel which may contain a fix for this issue, and report back if there has been any change/improvement.

[sunshineco]

I've updated to the latest kernel available via apt:

Linux pi.local 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux

For the last few months, I have had the machine perform a scheduled reboot every three days, which seems to be an effective workaround for the mmc_rescan problem; it hasn't manifested since. It is, of course, just a workaround.

I'm a bit hesitant to disable the scheduled reboot since this is a production device, however, I will do so in order to test if this new kernel makes a difference. Unfortunately, though, there is no reliable way to make the problem manifest (that I've discovered); it happened randomly. At first, it only happened every few weeks, then more frequently, until it was happening every 3 or 4 days (or even after just 5 minutes uptime, once). So, I may or may not have anything to report in the near term.

Thanks for keeping an eye on this issue.

[sunshineco]

Status report: The device had uptime of 7 days (exactly) without an mmc_rescan failure. A newer kernel, however, forced a reboot, so the uptime datapoint starts anew.

Linux pi.local 4.14.52-v7+ #1123 SMP Wed Jun 27 17:35:49 BST 2018 armv7l GNU/Linux

[sunshineco]

Status report: 4 weeks uptime and counting.

[JamesH65]

If you consider this issue to be fixed, please indicate whether you woudl be happy for it to be closed.

[sunshineco]

Although, it eventually got to the point where the problem was manifesting every few days (without a scheduled reboot), originally it only manifested every 4 or 4.5 weeks or so. Consequently, it's probably a bit premature to consider closing this issue. I'll continue with the status reports. If we get to 6 or 8 weeks continuous uptime, I'd feel more confident considering it fixed.

Thanks.

[sunshineco]

Status report: 6 weeks uptime (and counting).

It probably wouldn't hurt to close this issue as fixed at this point. If it manifests again, I'll leave a comment here and the issue can be re-opened.

[titiviking]

I'm having same issues, however no fix/workaround available for me... My Pi is crashing 4 times a day !!!!!

Syslog:

Jan  4 15:43:28 raspberrypi kernel: [ 4036.445856] Bluetooth: hci0: advertising data len corrected 30 -> 3
Jan  4 15:44:00 raspberrypi kernel: [ 4298.740680] INFO: task kworker/1:2:13023 blocked for more than 122 seconds.
Jan  4 15:44:31 raspberrypi kernel: [ 4298.740706]       Tainted: G         C        5.10.3-v7+ #1386
Jan  4 15:45:05 raspberrypi kernel: [ 4298.740718] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 15:45:38 raspberrypi kernel: [ 4298.740732] task:kworker/1:2     state:D stack:    0 pid:13023 ppid:     2 flags:0x00000000
Jan  4 15:46:03 raspberrypi kernel: [ 4298.740778] Workqueue: events_freezable mmc_rescan
Jan  4 15:46:36 raspberrypi kernel: [ 4298.740800] Backtrace:

kern.log:

Jan  4 15:09:06 raspberrypi kernel: [ 2240.848751] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.594965] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595133] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595214] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595324] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197103] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197284] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197332] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197451] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:43:28 raspberrypi kernel: [ 4036.445856] Bluetooth: hci0: advertising data len corrected 30 -> 3
Jan  4 15:44:00 raspberrypi kernel: [ 4298.740680] INFO: task kworker/1:2:13023 blocked for more than 122 seconds.
Jan  4 15:44:31 raspberrypi kernel: [ 4298.740706]       Tainted: G         C        5.10.3-v7+ #1386
Jan  4 15:45:05 raspberrypi kernel: [ 4298.740718] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 15:45:38 raspberrypi kernel: [ 4298.740732] task:kworker/1:2     state:D stack:    0 pid:13023 ppid:     2 flags:0x00000000
Jan  4 15:46:03 raspberrypi kernel: [ 4298.740778] Workqueue: events_freezable mmc_rescan
Jan  4 15:46:36 raspberrypi kernel: [ 4298.740800] Backtrace:
Jan  4 15:40:12 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0
Jan  4 15:40:12 raspberrypi kernel: [    0.000000] Linux version 5.10.3-v7+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1386 SMP Sun Dec 27 14:01:57 GMT 2020

My Pi seems to be fully up-to-date as I've performed following already:

• rpi-update
• apt full-upgrade (with apt update before of course)

I'm completely desperate in this case...
My only option I'm evaluating now is to step away from the Pi and go with a full-fledged server :-(

[pelwell]

Your post will be ignored because your symptoms are unrelated to the 2 year old issue you've replied to.