Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pi freezing up 4 times a day - Workqueue: events_freezable mmc_rescan #1522

Open
titiviking opened this issue Jan 4, 2021 · 11 comments
Open

Pi freezing up 4 times a day - Workqueue: events_freezable mmc_rescan #1522

titiviking opened this issue Jan 4, 2021 · 11 comments

Comments

@titiviking
Copy link

titiviking commented Jan 4, 2021 •
edited
Loading

I'm having some issues where my Pi freezes completely.
It is crashing 4 times a day !!!!! Only thing I can do is to unplug/replug to reboot it.

The Pi is still responding to ping commands, but all the rest is unresponsive (no SSH either, so I can't look at logs/processes when it actually happens).

Syslog:

Jan  4 15:43:28 raspberrypi kernel: [ 4036.445856] Bluetooth: hci0: advertising data len corrected 30 -> 3
Jan  4 15:44:00 raspberrypi kernel: [ 4298.740680] INFO: task kworker/1:2:13023 blocked for more than 122 seconds.
Jan  4 15:44:31 raspberrypi kernel: [ 4298.740706]       Tainted: G         C        5.10.3-v7+ #1386
Jan  4 15:45:05 raspberrypi kernel: [ 4298.740718] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 15:45:38 raspberrypi kernel: [ 4298.740732] task:kworker/1:2     state:D stack:    0 pid:13023 ppid:     2 flags:0x00000000
Jan  4 15:46:03 raspberrypi kernel: [ 4298.740778] Workqueue: events_freezable mmc_rescan
Jan  4 15:46:36 raspberrypi kernel: [ 4298.740800] Backtrace:

kern.log:

Jan  4 15:09:06 raspberrypi kernel: [ 2240.848751] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.594965] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595133] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595214] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:22:49 raspberrypi kernel: [ 3063.595324] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197103] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197284] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197332] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:38:21 raspberrypi kernel: [ 3996.197451] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 15:43:28 raspberrypi kernel: [ 4036.445856] Bluetooth: hci0: advertising data len corrected 30 -> 3
Jan  4 15:44:00 raspberrypi kernel: [ 4298.740680] INFO: task kworker/1:2:13023 blocked for more than 122 seconds.
Jan  4 15:44:31 raspberrypi kernel: [ 4298.740706]       Tainted: G         C        5.10.3-v7+ #1386
Jan  4 15:45:05 raspberrypi kernel: [ 4298.740718] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 15:45:38 raspberrypi kernel: [ 4298.740732] task:kworker/1:2     state:D stack:    0 pid:13023 ppid:     2 flags:0x00000000
Jan  4 15:46:03 raspberrypi kernel: [ 4298.740778] Workqueue: events_freezable mmc_rescan
Jan  4 15:46:36 raspberrypi kernel: [ 4298.740800] Backtrace:
Jan  4 15:40:12 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0
Jan  4 15:40:12 raspberrypi kernel: [    0.000000] Linux version 5.10.3-v7+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1386 SMP Sun Dec 27 14:01:57 GMT 2020

My Pi seems to be fully up-to-date as I've performed following already:

  • rpi-update
  • apt full-upgrade (with apt update before of course)

I'm completely desperate in this case...
My only option I'm evaluating now is to step away from the Pi and go with a full-fledged server :-(
@pelwell
Copy link
Contributor

pelwell commented Jan 4, 2021

  1. Are you using Bluetooth, and if so for what?
  2. What's in your config.txt?
  3. What power supply are you using?
  4. What external equipment is attached?

@titiviking
Copy link
Author

titiviking commented Jan 4, 2021 •
edited
Loading

  1. Are you using Bluetooth, and if so for what?
    Yes I am. Used to fetch info from a Bluetooth sensor. Data is displayed in Home Assistant.

  2. What's in your config.txt?

root@raspberrypi:~ # cat /boot/config.txt
# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable infrared communication.
#dtoverlay=gpio-ir,gpio_pin=17
#dtoverlay=gpio-ir-tx,gpio_pin=18

#Disable WiFi
dtoverlay=disable-wifi


# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=2

[all]
#dtoverlay=vc4-fkms-v3d

  1. What power supply are you using?
    I'm using the USB output of my NAS or TPLink router.
    (Now I'm making a tast with real power plug to see if it offers any stability)

  2. What external equipment is attached?
    None.

Additional info on current running processes (after 10min of runtime):

root@raspberrypi:~ # ps axo comm | sort | uniq -c | sort -rn | head
     21 s6-supervise
     12 containerd-shim
     10 s6-svscan
      8 bash
      7 docker-proxy
      7 dbus
      4 mmal-vchiq
      3 udevd
      3 sshd
      2 systemd

image

@popcornmix
Copy link
Contributor

I'm using the USB output of my NAS or TPLink router.
(Now I'm making a tast with real power plug to see if it offers any stability)

This is likely to be the issue. Not guaranteed to get more than 500mA from a USB port which isn't enough for any model of Pi (which Pi model is it?)

@titiviking
Copy link
Author

It's a Raspberry Pi 3B

@JamesH65
Copy link
Contributor

JamesH65 commented Jan 4, 2021

For power requirements see this page. https://github.com/raspberrypi/documentation/tree/master/faqs

@titiviking
Copy link
Author

Issue reoccured now... Pi was properly powered through the provided adapter.

kern.log:

Jan  4 20:11:19 raspberrypi kernel: [ 9764.166094] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:17:03 raspberrypi kernel: [10108.186048] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:24:44 raspberrypi kernel: [10108.186587] Bluetooth: hci0: advertising data len corrected 17 -> 3
Jan  4 20:24:50 raspberrypi kernel: [10565.683087] INFO: task kworker/2:3:12050 blocked for more than 122 seconds.
Jan  4 20:24:52 raspberrypi kernel: [10565.683110]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:24:57 raspberrypi kernel: [10565.683120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:25:02 raspberrypi kernel: [10565.683133] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:25:08 raspberrypi kernel: [10565.683172] Workqueue: events_freezable mmc_rescan
Jan  4 20:25:15 raspberrypi kernel: [10565.683192] Backtrace:
Jan  4 20:25:24 raspberrypi kernel: [10565.683222] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:25:33 raspberrypi kernel: [10565.683242]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:25:42 raspberrypi kernel: [10565.683253]  r4:ffffe000
Jan  4 20:25:47 raspberrypi kernel: [10565.683272] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:25:53 raspberrypi kernel: [10565.683285]  r5:82211218 r4:00000002
Jan  4 20:25:58 raspberrypi kernel: [10565.683302] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:26:00 raspberrypi kernel: [10565.683320]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:26:14 raspberrypi kernel: [10565.683331]  r4:82211800
Jan  4 20:26:24 raspberrypi kernel: [10565.683353] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:26:31 raspberrypi kernel: [10565.683366]  r5:82211000 r4:82211000
Jan  4 20:26:41 raspberrypi kernel: [10565.683386] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:26:49 raspberrypi kernel: [10565.683399]  r5:82211000 r4:8221127c
Jan  4 20:26:55 raspberrypi kernel: [10565.683420] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:27:05 raspberrypi kernel: [10565.683438]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:27:17 raspberrypi kernel: [10565.683457] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:27:26 raspberrypi kernel: [10565.683475]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:27:37 raspberrypi kernel: [10565.683486]  r4:86abf900
Jan  4 20:27:48 raspberrypi kernel: [10565.683506] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:28:05 raspberrypi kernel: [10565.683524]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:28:35 raspberrypi kernel: [10565.683535]  r4:9e4771c0
Jan  4 20:28:56 raspberrypi kernel: [10565.683553] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:29:15 raspberrypi kernel: [10565.683566] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:29:29 raspberrypi kernel: [10565.683580] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:29:43 raspberrypi kernel: [10565.683596] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:29:47 raspberrypi kernel: [10565.683611] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:29:54 raspberrypi kernel: [10565.683629]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:29:57 raspberrypi kernel: [10565.683640]  r4:b6004780
Jan  4 20:29:59 raspberrypi kernel: [10688.564295] INFO: task kworker/2:3:12050 blocked for more than 245 seconds.
Jan  4 20:30:05 raspberrypi kernel: [10688.564311]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:30:13 raspberrypi kernel: [10688.564317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:30:15 raspberrypi kernel: [10688.564324] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:30:21 raspberrypi kernel: [10688.564351] Workqueue: events_freezable mmc_rescan
Jan  4 20:30:26 raspberrypi kernel: [10688.564363] Backtrace:
Jan  4 20:30:31 raspberrypi kernel: [10688.564381] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:30:35 raspberrypi kernel: [10688.564393]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:30:41 raspberrypi kernel: [10688.564400]  r4:ffffe000
Jan  4 20:30:45 raspberrypi kernel: [10688.564411] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:30:49 raspberrypi kernel: [10688.564419]  r5:82211218 r4:00000002
Jan  4 20:30:53 raspberrypi kernel: [10688.564429] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:30:57 raspberrypi kernel: [10688.564439]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:30:59 raspberrypi kernel: [10688.564446]  r4:82211800
Jan  4 20:31:06 raspberrypi kernel: [10688.564459] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:31:16 raspberrypi kernel: [10688.564467]  r5:82211000 r4:82211000
Jan  4 20:31:22 raspberrypi kernel: [10688.564478] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:31:25 raspberrypi kernel: [10688.564486]  r5:82211000 r4:8221127c
Jan  4 20:31:28 raspberrypi kernel: [10688.564499] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:31:29 raspberrypi kernel: [10688.564509]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:31:31 raspberrypi kernel: [10688.564521] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:31:34 raspberrypi kernel: [10688.564531]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:31:37 raspberrypi kernel: [10688.564538]  r4:86abf900
Jan  4 20:31:40 raspberrypi kernel: [10688.564550] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:31:43 raspberrypi kernel: [10688.564561]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:31:46 raspberrypi kernel: [10688.564567]  r4:9e4771c0
Jan  4 20:31:48 raspberrypi kernel: [10688.564578] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:31:51 raspberrypi kernel: [10688.564586] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:31:55 raspberrypi kernel: [10688.564594] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:31:57 raspberrypi kernel: [10688.564604] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:32:00 raspberrypi kernel: [10688.564612] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:32:06 raspberrypi kernel: [10688.564622]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:32:11 raspberrypi kernel: [10688.564629]  r4:b6004780
Jan  4 20:32:14 raspberrypi kernel: [10811.445604] INFO: task kworker/2:3:12050 blocked for more than 368 seconds.
Jan  4 20:32:19 raspberrypi kernel: [10811.445620]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:32:21 raspberrypi kernel: [10811.445626] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:32:27 raspberrypi kernel: [10811.445634] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:32:34 raspberrypi kernel: [10811.445660] Workqueue: events_freezable mmc_rescan
Jan  4 20:32:44 raspberrypi kernel: [10811.445672] Backtrace:
Jan  4 20:32:48 raspberrypi kernel: [10811.445690] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:32:52 raspberrypi kernel: [10811.445702]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:32:57 raspberrypi kernel: [10811.445709]  r4:ffffe000
Jan  4 20:33:03 raspberrypi kernel: [10811.445719] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:33:08 raspberrypi kernel: [10811.445727]  r5:82211218 r4:00000002
Jan  4 20:33:11 raspberrypi kernel: [10811.445737] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:33:19 raspberrypi kernel: [10811.445748]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:33:23 raspberrypi kernel: [10811.445755]  r4:82211800
Jan  4 20:33:24 raspberrypi kernel: [10811.445767] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:33:28 raspberrypi kernel: [10811.445775]  r5:82211000 r4:82211000
Jan  4 20:33:31 raspberrypi kernel: [10811.445786] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:33:33 raspberrypi kernel: [10811.445794]  r5:82211000 r4:8221127c
Jan  4 20:33:38 raspberrypi kernel: [10811.445807] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:33:43 raspberrypi kernel: [10811.445817]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:33:49 raspberrypi kernel: [10811.445828] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:33:52 raspberrypi kernel: [10811.445839]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:33:54 raspberrypi kernel: [10811.445845]  r4:86abf900
Jan  4 20:33:56 raspberrypi kernel: [10811.445857] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:33:58 raspberrypi kernel: [10811.445868]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:34:04 raspberrypi kernel: [10811.445875]  r4:9e4771c0
Jan  4 20:34:07 raspberrypi kernel: [10811.445886] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:34:11 raspberrypi kernel: [10811.445893] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:34:20 raspberrypi kernel: [10811.445901] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:34:28 raspberrypi kernel: [10811.445911] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:34:36 raspberrypi kernel: [10811.445920] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:34:46 raspberrypi kernel: [10811.445930]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:34:56 raspberrypi kernel: [10811.445936]  r4:b6004780
Jan  4 20:35:05 raspberrypi kernel: [10934.326923] INFO: task kworker/2:3:12050 blocked for more than 491 seconds.
Jan  4 20:35:16 raspberrypi kernel: [10934.326939]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:35:24 raspberrypi kernel: [10934.326945] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:35:33 raspberrypi kernel: [10934.326953] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:35:45 raspberrypi kernel: [10934.326981] Workqueue: events_freezable mmc_rescan
Jan  4 20:35:55 raspberrypi kernel: [10934.326993] Backtrace:
Jan  4 20:36:01 raspberrypi kernel: [10934.327012] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:36:09 raspberrypi kernel: [10934.327024]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:36:18 raspberrypi kernel: [10934.327031]  r4:ffffe000
Jan  4 20:36:24 raspberrypi kernel: [10934.327041] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:36:31 raspberrypi kernel: [10934.327049]  r5:82211218 r4:00000002
Jan  4 20:36:39 raspberrypi kernel: [10934.327059] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:36:53 raspberrypi kernel: [10934.327070]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:37:00 raspberrypi kernel: [10934.327076]  r4:82211800
Jan  4 20:37:07 raspberrypi kernel: [10934.327088] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:37:16 raspberrypi kernel: [10934.327096]  r5:82211000 r4:82211000
Jan  4 20:37:26 raspberrypi kernel: [10934.327108] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:37:33 raspberrypi kernel: [10934.327116]  r5:82211000 r4:8221127c
Jan  4 20:37:40 raspberrypi kernel: [10934.327128] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:37:46 raspberrypi kernel: [10934.327139]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:37:51 raspberrypi kernel: [10934.327150] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:37:57 raspberrypi kernel: [10934.327161]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:38:04 raspberrypi kernel: [10934.327168]  r4:86abf900
Jan  4 20:38:15 raspberrypi kernel: [10934.327180] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:38:23 raspberrypi kernel: [10934.327190]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:38:31 raspberrypi kernel: [10934.327197]  r4:9e4771c0
Jan  4 20:38:41 raspberrypi kernel: [10934.327208] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:38:50 raspberrypi kernel: [10934.327216] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:38:59 raspberrypi kernel: [10934.327224] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:39:09 raspberrypi kernel: [10934.327234] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:39:21 raspberrypi kernel: [10934.327243] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:39:29 raspberrypi kernel: [10934.327253]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:39:32 raspberrypi kernel: [10934.327259]  r4:b6004780
Jan  4 20:39:37 raspberrypi kernel: [11055.198724] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:39:47 raspberrypi kernel: [11055.198779] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:39:55 raspberrypi kernel: [11055.198864] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:40:01 raspberrypi kernel: [11055.198949] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:40:10 raspberrypi kernel: [11055.199036] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:40:17 raspberrypi kernel: [11055.199101] Bluetooth: hci0: Frame reassembly failed (-84)
Jan  4 20:40:25 raspberrypi kernel: [11057.208319] INFO: task kworker/2:3:12050 blocked for more than 614 seconds.
Jan  4 20:40:32 raspberrypi kernel: [11057.208343]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:40:39 raspberrypi kernel: [11057.208353] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:40:48 raspberrypi kernel: [11057.208365] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:40:59 raspberrypi kernel: [11057.208405] Workqueue: events_freezable mmc_rescan
Jan  4 20:41:02 raspberrypi kernel: [11057.208424] Backtrace:
Jan  4 20:41:09 raspberrypi kernel: [11057.208455] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:41:17 raspberrypi kernel: [11057.208474]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:41:27 raspberrypi kernel: [11057.208486]  r4:ffffe000
Jan  4 20:41:31 raspberrypi kernel: [11057.208504] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:41:41 raspberrypi kernel: [11057.208517]  r5:82211218 r4:00000002
Jan  4 20:41:53 raspberrypi kernel: [11057.208535] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:42:00 raspberrypi kernel: [11057.208553]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:42:12 raspberrypi kernel: [11057.208564]  r4:82211800
Jan  4 20:42:22 raspberrypi kernel: [11057.208586] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:42:28 raspberrypi kernel: [11057.208599]  r5:82211000 r4:82211000
Jan  4 20:42:35 raspberrypi kernel: [11057.208618] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:42:44 raspberrypi kernel: [11057.208632]  r5:82211000 r4:8221127c
Jan  4 20:42:56 raspberrypi kernel: [11057.208652] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:43:07 raspberrypi kernel: [11057.208670]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:43:17 raspberrypi kernel: [11057.208690] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:43:26 raspberrypi kernel: [11057.208708]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:43:33 raspberrypi kernel: [11057.208719]  r4:86abf900
Jan  4 20:43:41 raspberrypi kernel: [11057.208738] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:43:48 raspberrypi kernel: [11057.208757]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:43:55 raspberrypi kernel: [11057.208768]  r4:9e4771c0
Jan  4 20:44:03 raspberrypi kernel: [11057.208787] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:44:13 raspberrypi kernel: [11057.208799] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:44:23 raspberrypi kernel: [11057.208813] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:44:27 raspberrypi kernel: [11057.208829] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:44:35 raspberrypi kernel: [11057.208844] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:44:44 raspberrypi kernel: [11057.208861]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:44:53 raspberrypi kernel: [11057.208873]  r4:b6004780
Jan  4 20:45:11 raspberrypi kernel: [11180.089644] INFO: task kworker/2:3:12050 blocked for more than 737 seconds.
Jan  4 20:45:27 raspberrypi kernel: [11180.089667]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:45:36 raspberrypi kernel: [11180.089677] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:45:47 raspberrypi kernel: [11180.089689] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:45:56 raspberrypi kernel: [11180.089730] Workqueue: events_freezable mmc_rescan
Jan  4 20:46:08 raspberrypi kernel: [11180.089749] Backtrace:
Jan  4 20:46:19 raspberrypi kernel: [11180.089779] [<809e44e0>] (__schedule) from [<809e4eb0>] (schedule+0x68/0xe4)
Jan  4 20:46:39 raspberrypi kernel: [11180.089799]  r10:82211000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000113 r5:81733e00
Jan  4 20:46:54 raspberrypi kernel: [11180.089810]  r4:ffffe000
Jan  4 20:47:04 raspberrypi kernel: [11180.089829] [<809e4e48>] (schedule) from [<807f6a88>] (__mmc_claim_host+0xe0/0x238)
Jan  4 20:47:22 raspberrypi kernel: [11180.089842]  r5:82211218 r4:00000002
Jan  4 20:47:38 raspberrypi kernel: [11180.089859] [<807f69a8>] (__mmc_claim_host) from [<807f6c18>] (mmc_get_card+0x38/0x3c)
Jan  4 20:47:46 raspberrypi kernel: [11180.089877]  r10:b6b54640 r9:00000000 r8:b6b57c00 r7:00000080 r6:82211218 r5:00000000
Jan  4 20:47:51 raspberrypi kernel: [11180.089888]  r4:82211800
Jan  4 20:47:58 raspberrypi kernel: [11180.089909] [<807f6be0>] (mmc_get_card) from [<80800830>] (mmc_sd_detect+0x24/0x7c)
Jan  4 20:48:03 raspberrypi kernel: [11180.089922]  r5:82211000 r4:82211000
Jan  4 20:48:14 raspberrypi kernel: [11180.089942] [<8080080c>] (mmc_sd_detect) from [<807f9450>] (mmc_rescan+0xdc/0x3b0)
Jan  4 20:48:20 raspberrypi kernel: [11180.089954]  r5:82211000 r4:8221127c
Jan  4 20:48:29 raspberrypi kernel: [11180.089976] [<807f9374>] (mmc_rescan) from [<8013b78c>] (process_one_work+0x254/0x584)
Jan  4 20:48:43 raspberrypi kernel: [11180.089993]  r9:00000000 r8:b6b57c00 r7:00000080 r6:b6b54640 r5:86abf900 r4:8221127c
Jan  4 20:48:52 raspberrypi kernel: [11180.090013] [<8013b538>] (process_one_work) from [<8013bb1c>] (worker_thread+0x60/0x5c4)
Jan  4 20:49:01 raspberrypi kernel: [11180.090031]  r10:b6b54640 r9:80e03d00 r8:b6b54658 r7:00000008 r6:b6b54640 r5:86abf914
Jan  4 20:49:08 raspberrypi kernel: [11180.090042]  r4:86abf900
Jan  4 20:49:18 raspberrypi kernel: [11180.090061] [<8013babc>] (worker_thread) from [<801435a4>] (kthread+0x170/0x174)
Jan  4 20:49:30 raspberrypi kernel: [11180.090080]  r10:84acde74 r9:86abf900 r8:8013babc r7:87ca0000 r6:00000000 r5:b6004780
Jan  4 20:49:39 raspberrypi kernel: [11180.090091]  r4:9e4771c0
Jan  4 20:49:53 raspberrypi kernel: [11180.090110] [<80143434>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Jan  4 20:50:00 raspberrypi kernel: [11180.090122] Exception stack(0x87ca1fb0 to 0x87ca1ff8)
Jan  4 20:50:11 raspberrypi kernel: [11180.090136] 1fa0:                                     00000000 00000000 00000000 00000000
Jan  4 20:50:24 raspberrypi kernel: [11180.090153] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan  4 20:50:38 raspberrypi kernel: [11180.090167] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan  4 20:50:48 raspberrypi kernel: [11180.090185]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143434
Jan  4 20:51:03 raspberrypi kernel: [11180.090196]  r4:b6004780
Jan  4 20:51:20 raspberrypi kernel: [11302.970915] INFO: task kworker/2:3:12050 blocked for more than 860 seconds.
Jan  4 20:51:38 raspberrypi kernel: [11302.970938]       Tainted: G         C        5.10.4-v7+ #1387
Jan  4 20:52:01 raspberrypi kernel: [11302.970948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  4 20:52:15 raspberrypi kernel: [11302.970960] task:kworker/2:3     state:D stack:    0 pid:12050 ppid:     2 flags:0x00000000
Jan  4 20:17:06 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0

Any way we can find out what is causing this ?

@makermelissa makermelissa mentioned this issue Jan 8, 2021
Closed
@titiviking
Copy link
Author

I was able to solve the issues, but it required a small investment...

I did the following:

  1. Use the power adapter provided, it reduces slightly the occurences, but don't fully remove them... You can keep it active for slightly longer, but after a while errors occur again and you need to reboot it by power-cycling
  2. Get rid of SD-card. SD cards are slow, so I went for a USB 3.0... works a bit better, but the USB is still slow (especially in my case where Home Assistant writes to the DB quite a bit)
  3. Get rid of USB and go for SSD. This is magical!!! No more errors, no more latency, no more unavailability. Going SSD was THE solution for me. Raspberry is now fast and never got stuck again.
    In order to attach the SSD I got an adapter board from amazon
    .
  4. Next step for me if an upgrade to a Raspberry Pi 4 B that has USB3.1 ports to go even faster and really have seamless experience and reduced lag when it comes to reading large amounts of values from the HA DB.

@jvonau jvonau mentioned this issue Mar 4, 2021
Closed
@samyk samyk mentioned this issue Dec 5, 2021
Open
@All3xJ
Copy link

All3xJ commented Apr 24, 2022

I still have this bug with 5.15.30-v8+ 64 bit Raspberry Pi 3B+.... the system freezed and resumed after 3 hours of freeze.
Here are journalctl logs:
journalctl.txt

@pelwell
Copy link
Contributor

pelwell commented Apr 28, 2022 

Apr 23 16:35:06 raspberrypi kernel: hwmon hwmon1: Undervoltage detected!
Apr 23 16:35:06 raspberrypi kernel: hwmon hwmon1: Voltage normalised

Your system is struggling because there is insufficient power getting to the processor.

  1. Check the power supply - it should be capable of delivering at least 3A for a Pi 4. Note that a charger is not necessarily the same as a supply because chargers don't have to worry about voltage and current variations.
  2. Check the cable from the supply to the Pi - it's shocking how much power can be lost by in-line switches etc.
  3. If you have power-hungry USB peripherals attached, put them on a powered hub.

@All3xJ
Copy link

All3xJ commented Apr 29, 2022 •
edited
Loading 

Apr 23 16:35:06 raspberrypi kernel: hwmon hwmon1: Undervoltage detected!
Apr 23 16:35:06 raspberrypi kernel: hwmon hwmon1: Voltage normalised

Your system is struggling because there is insufficient power getting to the processor.

1. Check the power supply - it should be capable of delivering at least 3A for a Pi 4. Note that a charger is not necessarily the same as a supply because chargers don't have to worry about voltage and current variations.

2. Check the cable from the supply to the Pi - it's shocking how much power can be lost by in-line switches etc.

3. If you have power-hungry USB peripherals attached, put them on a powered hub.

I have these logs all the time, always.... and it never happened that it completely freezed for 3 hours, so I don't think that this is the real reason. I followed this workaround: https://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/

@sahib sahib mentioned this issue Sep 11, 2022
Open
@sresam89
Copy link

sresam89 commented Mar 3, 2024 •
edited
Loading

I can confirm the same occurs on newer firmware too. Started happening after I migrated from 32-bit to 64-bit OS

RPI4 2GBRAM & its original power supply
Using Brand new 64GB samsung evo sd card. (Switched to a new card since 64 bit OS crash started)
Default partition size done by RPiOS flash tool.
No Wifi, hardwired to network.
No USB accessories, running headless (not while debugging the hanging state)

pi@raspberrypi:~ $ docker-compose ps
NAME            IMAGE                                                     COMMAND                  SERVICE         CREATED      STATUS       PORTS
homeassistant   ghcr.io/home-assistant/home-assistant:stable              "/init"                  homeassistant   5 days ago   Up 6 hours
matterServer    ghcr.io/home-assistant-libs/python-matter-server:stable   "matter-server --sto…"   matterServer    6 days ago   Up 6 hours
mosquitto       eclipse-mosquitto:latest                                  "/docker-entrypoint.…"   mosquitto       6 days ago   Up 6 hours
neolink         quantumentangledandy/neolink                              "/entrypoint.sh /usr…"   neolink         4 days ago   Up 6 hours
pi-mariadb-1    yobasystems/alpine-mariadb:latest                         "/scripts/run.sh"        mariadb         6 days ago   Up 6 hours   3306/tcp
pi-nginx-1      jc21/nginx-proxy-manager:latest                           "/init"                  nginx           6 days ago   Up 6 hours   0.0.0.0:80-81->80-81/tcp, :::80-81->80-81/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp

Occurs every two to three hours, hangs up and waits for forced reboot via removing power. Console, network-ping, everything becomes unresponsive.

Tried the solutions recommended in this post

I've so far found some advice from https://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/ summary: sudo sysctl -w vm.dirty_ratio=10 sudo sysctl -w vm.dirty_background_ratio=5 sudo sysctl -p seems to do the trick so far, still testing (3 movies down) I'll comment if it occurs again. To make it permanent(after reboot) read the link

no luck there

pi@raspberrypi:~ $ free
               total        used        free      shared  buff/cache   available
Mem:         1892500      815472      196372        2212      951516     1077028
Swap:         102396           0      102396
pi@raspberrypi:~ $


uname -a
Linux raspberrypi 6.1.0-rpi7-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux

pi@raspberrypi:~ $ vcgencmd version
Oct 17 2023 15:39:16
Copyright (c) 2012 Broadcom
version 30f0c5e4d076da3ab4f341d88e7d505760b93ad7 (clean) (release) (start)

pi@raspberrypi:~ $ cat /etc/rpi-issue
Raspberry Pi reference 2023-12-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 2acf7afcba7d11500313a7b93bb55a2aae20b2d6, stage2

journalctl -b -1
Mar 02 23:23:03 raspberrypi kernel: INFO: task kworker/0:3:2023 blocked for more than 120 seconds.
Mar 02 23:25:37 raspberrypi kernel:       Tainted: G         C         6.1.0-rpi7-rpi-v8 #1 Debian 1:6.1.63-1+rpt1
Mar 02 23:26:42 raspberrypi kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 02 23:27:46 raspberrypi kernel: task:kworker/0:3     state:D stack:0     pid:2023  ppid:2      flags:0x00000008
Mar 02 23:29:30 raspberrypi kernel: Workqueue: events_freezable mmc_rescan
Mar 02 23:33:15 raspberrypi kernel: Call trace:
Mar 02 23:35:23 raspberrypi kernel:  __switch_to+0xe8/0x160
Mar 02 23:48:33 raspberrypi kernel:  __schedule+0x338/0x924
Mar 03 00:44:29 raspberrypi kernel:  schedule+0x64/0x10c
Mar 03 01:31:32 raspberrypi kernel:  __mmc_claim_host+0xe0/0x204
Mar 03 02:31:38 raspberrypi kernel:  mmc_get_card+0x3c/0x4c

journalctl -b -2
Mar 02 13:29:46 raspberrypi kernel: INFO: task kworker/0:0:2138 blocked for more than 120 seconds.
Mar 02 13:32:48 raspberrypi kernel:       Tainted: G         C         6.1.0-rpi7-rpi-v8 #1 Debian 1:6.1.63-1+rpt1
Mar 02 13:36:04 raspberrypi kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 02 13:39:09 raspberrypi kernel: task:kworker/0:0     state:D stack:0     pid:2138  ppid:2      flags:0x00000008
Mar 02 13:41:02 raspberrypi kernel: Workqueue: events_freezable mmc_rescan
Mar 02 13:43:13 raspberrypi kernel: Call trace:
Mar 02 13:44:47 raspberrypi kernel:  __switch_to+0xe8/0x160
Mar 02 13:45:52 raspberrypi kernel:  __schedule+0x338/0x924
Mar 02 13:46:58 raspberrypi kernel:  schedule+0x64/0x10c
Mar 02 13:48:54 raspberrypi kernel:  __mmc_claim_host+0xe0/0x204
Mar 02 13:50:56 raspberrypi kernel:  mmc_get_card+0x3c/0x4c
Mar 02 13:52:55 raspberrypi kernel:  mmc_sd_detect+0x28/0xa0
Mar 02 13:55:59 raspberrypi kernel:  mmc_rescan+0x94/0x2ec
Mar 02 13:59:29 raspberrypi kernel:  process_one_work+0x200/0x474
Mar 02 14:02:55 raspberrypi kernel:  worker_thread+0x74/0x43c
Mar 02 14:06:27 raspberrypi kernel:  kthread+0x110/0x114
Mar 02 14:09:28 raspberrypi kernel:  ret_from_fork+0x10/0x20

This was referenced Mar 3, 2024
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@popcornmix @JamesH65 @pelwell @All3xJ @sresam89 @titiviking