WARNING: CPU: 0 PID: 0 at net/core/skbuff.c:3995

[simonhiggs]

Hello. It's been suggested on the Arch Linux ARM forums that I post here.
I'm using a Pi 2 with a HiFiBerry Digi+ SPDIF interface. Occasionally I have a loud stuttering whilst streaming from a NAS, and have the following in the logs

[  298.902324] ------------[ cut here ]------------
[  298.907282] WARNING: CPU: 0 PID: 0 at net/core/skbuff.c:3995 skb_try_coalesce+0x320/0x378()
[  298.915988] Modules linked in: snd_soc_bcm2708_i2s snd_soc_hifiberry_digi snd_soc_wm8804 regmap_mmio snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm regmap_spi snd_timer regmap_i2c snd i2c_bcm2708 leds_gpio led_class uio_pdrv_genirq uio sch_fq_codel bcm2708_rng rng_core ipv6
[  298.942115] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.18.7-4-ARCH #1
[  298.948937] [<c0015630>] (unwind_backtrace) from [<c0011f2c>] (show_stack+0x10/0x14)
[  298.957014] [<c0011f2c>] (show_stack) from [<c0734b44>] (dump_stack+0x84/0xc4)
[  298.964421] [<c0734b44>] (dump_stack) from [<c002ca84>] (warn_slowpath_common+0x74/0x90)
[  298.972751] [<c002ca84>] (warn_slowpath_common) from [<c002cb3c>] (warn_slowpath_null+0x1c/0x24)
[  298.981794] [<c002cb3c>] (warn_slowpath_null) from [<c065db4c>] (skb_try_coalesce+0x320/0x378)
[  298.990631] [<c065db4c>] (skb_try_coalesce) from [<c06b2108>] (tcp_try_coalesce.part.38+0x28/0xbc)
[  298.999841] [<c06b2108>] (tcp_try_coalesce.part.38) from [<c06b38dc>] (tcp_data_queue+0x7a8/0xca8)
[  299.009019] [<c06b38dc>] (tcp_data_queue) from [<c06b55d0>] (tcp_rcv_established+0x150/0x63c)
[  299.034790] [<c06b55d0>] (tcp_rcv_established) from [<c06bd998>] (tcp_v4_do_rcv+0x180/0x394)
[  299.078781] [<c06bd998>] (tcp_v4_do_rcv) from [<c06c01a4>] (tcp_v4_rcv+0x8c4/0x8cc)
[  299.121648] [<c06c01a4>] (tcp_v4_rcv) from [<c069c830>] (ip_local_deliver_finish+0xe0/0x2c8)
[  299.161879] [<c069c830>] (ip_local_deliver_finish) from [<c0667f9c>] (__netif_receive_skb_core+0x664/0x85c)
[  299.203365] [<c0667f9c>] (__netif_receive_skb_core) from [<c066b8d8>] (process_backlog+0x8c/0x138)
[  299.244130] [<c066b8d8>] (process_backlog) from [<c066b71c>] (net_rx_action+0x13c/0x26c)
[  299.284239] [<c066b71c>] (net_rx_action) from [<c00303f4>] (__do_softirq+0x128/0x320)
[  299.324355] [<c00303f4>] (__do_softirq) from [<c0030880>] (irq_exit+0xb8/0xf4)
[  299.347914] [<c0030880>] (irq_exit) from [<c0073508>] (__handle_domain_irq+0xa8/0x104)
[  299.388094] [<c0073508>] (__handle_domain_irq) from [<c073a830>] (__irq_svc+0x30/0x144)
[  299.428484] [<c073a830>] (__irq_svc) from [<c000f13c>] (arch_cpu_idle+0x38/0x3c)
[  299.468195] [<c000f13c>] (arch_cpu_idle) from [<c0064528>] (cpu_startup_entry+0x358/0x3e4)
[  299.508846] [<c0064528>] (cpu_startup_entry) from [<c09f7c64>] (start_kernel+0x3a0/0x3ac)
[  299.549293] ---[ end trace 90c67f02a1d32a34 ]---

I've no idea about what this means, let alone how to fix. Any help most welcome.

Simon.

Linux raspi 3.18.7-4-ARCH #1 SMP PREEMPT Mon Feb 16 11:39:41 MST 2015 armv7l GNU/Linux

[popcornmix]

Some similar reports:
http://lists.infradead.org/pipermail/ath10k/2014-May/001900.html
http://comments.gmane.org/gmane.linux.network/236358
http://thr3ads.net/xen-devel/2012/12/2279679-PATCH-xen-netfront-improve-truesize-tracking

suggest it may be an issue with skb->truesize in the network driver.
I've pinged the author of smsc95xx driver, hopefully he might have a suggestion.

[popcornmix]

Are you able to rebuild kernel? I have a patch from Steve that may help:
http://pastebin.com/raw.php?i=xm5SnJmB

[simonhiggs]

popcornmix, thank-you very much for this. Patch applied and kernel rebuilt overnight. Streaming without any stuttering or warnings all day today.

Cheers!

[0xC0FFEE]

I've got the same problem with my raspberry pi 2. Will this patch be merged into master or do i have to start building my own kernel? Thanks.

[popcornmix]

@steveglen thought the linked patch might have some performance impact and he'd think about an alternative. Hopefully he can suggest something.

[f1vefour]

Getting the same thing on the XBian kernel. I'm going to test the patch as this bug is a showstopper.

[lucazen]

Hi There!
I've the same issue, below the log if you need (even if it is pretty the same) :

[61967.998329] ------------[ cut here ]------------
[61967.998411] WARNING: CPU: 0 PID: 0 at net/core/skbuff.c:3995 skb_try_coalesce+0x314/0x39c()
[61967.998424] Modules linked in: xt_multiport iptable_filter ip_tables x_tables snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd evdev uio_pdrv_genirq uio
[61967.998513] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.18.7-v7+ #755
[61967.998556] [<80016d14>] (unwind_backtrace) from [<80012c40>] (show_stack+0x20/0x24)
[61967.998586] [<80012c40>] (show_stack) from [<8052efe8>] (dump_stack+0x98/0xd8)
[61967.998617] [<8052efe8>] (dump_stack) from [<800251e0>] (warn_slowpath_common+0x7c/0xa0)
[61967.998647] [<800251e0>] (warn_slowpath_common) from [<800252c0>] (warn_slowpath_null+0x2c/0x34)
[61967.998673] [<800252c0>] (warn_slowpath_null) from [<8044c968>] (skb_try_coalesce+0x314/0x39c)
[61967.998701] [<8044c968>] (skb_try_coalesce) from [<804a5dd8>] (tcp_try_coalesce.part.38+0x38/0xc0)
[61967.998727] [<804a5dd8>] (tcp_try_coalesce.part.38) from [<804a6d4c>] (tcp_data_queue+0x7d0/0xca0)
[61967.998753] [<804a6d4c>] (tcp_data_queue) from [<804a92cc>] (tcp_rcv_established+0x148/0x610)
[61967.998781] [<804a92cc>] (tcp_rcv_established) from [<804b1864>] (tcp_v4_do_rcv+0x14c/0x368)
[61967.998809] [<804b1864>] (tcp_v4_do_rcv) from [<804b412c>] (tcp_v4_rcv+0x864/0x878)
[61967.998838] [<804b412c>] (tcp_v4_rcv) from [<8048fb88>] (ip_local_deliver_finish+0xec/0x2d8)
[61967.998866] [<8048fb88>] (ip_local_deliver_finish) from [<804902dc>] (ip_local_deliver+0x4c/0xac)
[61967.998893] [<804902dc>] (ip_local_deliver) from [<8048fed4>] (ip_rcv_finish+0x160/0x3ec)
[61967.998918] [<8048fed4>] (ip_rcv_finish) from [<80490628>] (ip_rcv+0x2ec/0x4a4)
[61967.998947] [<80490628>] (ip_rcv) from [<804599dc>] (__netif_receive_skb_core+0x600/0x824)
[61967.998976] [<804599dc>] (__netif_receive_skb_core) from [<80459c20>] (__netif_receive_skb+0x20/0x70)
[61967.999002] [<80459c20>] (__netif_receive_skb) from [<8045b474>] (process_backlog+0xa4/0x170)
[61967.999024] [<8045b474>] (process_backlog) from [<8045b278>] (net_rx_action+0x13c/0x294)
[61967.999047] [<8045b278>] (net_rx_action) from [<80028ed8>] (__do_softirq+0x12c/0x384)
[61967.999071] [<80028ed8>] (__do_softirq) from [<800294b4>] (irq_exit+0xc0/0x110)
[61967.999097] [<800294b4>] (irq_exit) from [<8006938c>] (__handle_domain_irq+0x8c/0xdc)
[61967.999123] [<8006938c>] (__handle_domain_irq) from [<80008564>] (asm_do_IRQ+0x2c/0x30)
[61967.999146] [<80008564>] (asm_do_IRQ) from [<805349b4>] (__irq_svc+0x34/0x14c)
[61967.999161] Exception stack(0x807cbf08 to 0x807cbf50)
[61967.999181] bf00:                   807e9ccc 00000000 ffffffed 00000000 807ca020 807e8dd4
[61967.999201] bf20: 00000000 00000000 807ca000 807e8d70 808238fc 807cbf5c 807cbf50 807cbf50
[61967.999218] bf40: 8000f900 8000f904 60000013 ffffffff
[61967.999243] [<805349b4>] (__irq_svc) from [<8000f904>] (arch_cpu_idle+0x30/0x4c)
[61967.999272] [<8000f904>] (arch_cpu_idle) from [<8005c5cc>] (cpu_startup_entry+0x314/0x3d4)
[61967.999299] [<8005c5cc>] (cpu_startup_entry) from [<8052a5dc>] (rest_init+0x80/0x98)
[61967.999326] [<8052a5dc>] (rest_init) from [<80767cb4>] (start_kernel+0x370/0x3d8)
[61967.999340] ---[ end trace 03902c275340a551 ]---

on

Linux raspzen 3.18.7-v7+ #755 SMP PREEMPT Thu Feb 12 17:20:48 GMT 2015 armv7l GNU/Linux

regards,
Luca

[popcornmix]

Does smsc95xx.turbo_mode=N added to cmdline.txt help?
Any noticeable performance impact?

[simonhiggs]

Thanks popcornmix. I'm away from home so can't test smsc95xx.turbo_mode=N fix for a few weeks. As I'm only streaming audio (1411kbs) so probably not notice any performance problems, unless severe.

regards, Simon.

[analyser-]

Hello!

I'm using raspberry 2 with raspbian as file server with syncthing and sometimes got this nasty error in my logs.

$uname -a
Linux xyz 3.18.7+ #755 PREEMPT Thu Feb 12 17:14:31 GMT 2015 armv6l GNU/Linux

The error:

[153347.580638] ------------[ cut here ]------------
[153347.580713] WARNING: CPU: 0 PID: 6171 at net/core/skbuff.c:3995 skb_try_coalesce+0x330/0x3c0()
[153347.580728] Modules linked in: snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd raid0 md_mod sg uas uio_pdrv_genirq uio
[153347.580801] CPU: 0 PID: 6171 Comm: syncthing Not tainted 3.18.7+ #755
[153347.580868] [<c00151fc>] (unwind_backtrace) from [<c0012710>] (show_stack+0x20/0x24)
[153347.580911] [<c0012710>] (show_stack) from [<c0555b30>] (dump_stack+0x20/0x28)
[153347.580953] [<c0555b30>] (dump_stack) from [<c0022ef4>] (warn_slowpath_common+0x7c/0x9c)
[153347.580985] [<c0022ef4>] (warn_slowpath_common) from [<c0022fd0>] (warn_slowpath_null+0x2c/0x34)
[153347.581014] [<c0022fd0>] (warn_slowpath_null) from [<c04700c8>] (skb_try_coalesce+0x330/0x3c0)
[153347.581050] [<c04700c8>] (skb_try_coalesce) from [<c04c8e30>] (tcp_try_coalesce+0x5c/0xd8)
[153347.581083] [<c04c8e30>] (tcp_try_coalesce) from [<c04c994c>] (tcp_data_queue+0x3a0/0xc58)
[153347.581113] [<c04c994c>] (tcp_data_queue) from [<c04cc2e0>] (tcp_rcv_established+0x138/0x5f4)
[153347.581149] [<c04cc2e0>] (tcp_rcv_established) from [<c04d4964>] (tcp_v4_do_rcv+0x114/0x298)
[153347.581182] [<c04d4964>] (tcp_v4_do_rcv) from [<c04d7228>] (tcp_v4_rcv+0x87c/0x890)
[153347.581224] [<c04d7228>] (tcp_v4_rcv) from [<c04b2cd8>] (ip_local_deliver_finish+0xdc/0x250)
[153347.581255] [<c04b2cd8>] (ip_local_deliver_finish) from [<c04b3318>] (ip_local_deliver+0x48/0xa8)
[153347.581283] [<c04b3318>] (ip_local_deliver) from [<c04b2f84>] (ip_rcv_finish+0x138/0x358)
[153347.581313] [<c04b2f84>] (ip_rcv_finish) from [<c04b35c0>] (ip_rcv+0x248/0x38c)
[153347.581358] [<c04b35c0>] (ip_rcv) from [<c047c378>] (__netif_receive_skb_core+0x628/0x850)
[153347.581392] [<c047c378>] (__netif_receive_skb_core) from [<c047c5c0>] (__netif_receive_skb+0x20/0x70)
[153347.581421] [<c047c5c0>] (__netif_receive_skb) from [<c047d32c>] (process_backlog+0x7c/0x11c)
[153347.581448] [<c047d32c>] (process_backlog) from [<c047ca60>] (net_rx_action+0x144/0x2a4)
[153347.581482] [<c047ca60>] (net_rx_action) from [<c00265e8>] (__do_softirq+0x118/0x36c)
[153347.581513] [<c00265e8>] (__do_softirq) from [<c0026ba0>] (irq_exit+0xbc/0x110)
[153347.581550] [<c0026ba0>] (irq_exit) from [<c005ef68>] (__handle_domain_irq+0x88/0xd4)
[153347.581582] [<c005ef68>] (__handle_domain_irq) from [<c000853c>] (asm_do_IRQ+0x2c/0x30)
[153347.581622] [<c000853c>] (asm_do_IRQ) from [<c055b248>] (__irq_usr+0x48/0xc0)
[153347.581638] Exception stack(0xd77dbfb0 to 0xd77dbff8)
[153347.581659] bfa0:                                     880fa651 4c26266a 00003ae9 000000e9
[153347.581684] bfc0: 00000007 00000020 4d48bb8e b641cc4f 3ae9b788 00000000 1199eb40 007dad64
[153347.581705] bfe0: 23410549 124f850c 002dafd0 002d9468 80000010 ffffffff
[153347.581721] ---[ end trace 85c2aa94f92d0153 ]---

And yes, I'm using smsc95xx.turbo_mode=N in cmdline.txt, otherwise I get thousands of other errors.

$ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p6 rootfstype=ext4 elevator=deadline rootwait smsc95xx.turbo_mode=N

I did not tried the patch afore mentioned patch yet.

Regards,
Leonardo.

[Razican]

Also happens using Linux 3.18.11-v7+ #777 SMP PREEMPT Sat Apr 11 17:30:37 BST 2015 armv7l GNU/Linux

On a Raspberry Pi 2 B using latest firmware from rpi-update. Here is the dmsg dump:

[  530.323701] ------------[ cut here ]------------
[  530.328514] WARNING: CPU: 0 PID: 0 at net/core/skbuff.c:3996 skb_try_coalesce+0x314/0x39c()
[  530.336988] Modules linked in: nfsd ipv6 fuse sg snd_bcm2835 snd_pcm snd_timer snd uio_pdrv_genirq uio
[  530.346807] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.18.11-v7+ #777
[  530.353489] [<800172bc>] (unwind_backtrace) from [<80012f7c>] (show_stack+0x20/0x24)
[  530.361386] [<80012f7c>] (show_stack) from [<80520980>] (dump_stack+0x98/0xd8)
[  530.368781] [<80520980>] (dump_stack) from [<80025618>] (warn_slowpath_common+0x7c/0xa0)
[  530.377038] [<80025618>] (warn_slowpath_common) from [<800256f8>] (warn_slowpath_null+0x2c/0x34)
[  530.385981] [<800256f8>] (warn_slowpath_null) from [<8043e928>] (skb_try_coalesce+0x314/0x39c)
[  530.394757] [<8043e928>] (skb_try_coalesce) from [<80497d80>] (tcp_try_coalesce.part.38+0x38/0xc0)
[  530.403875] [<80497d80>] (tcp_try_coalesce.part.38) from [<80498cf4>] (tcp_data_queue+0x7d0/0xca0)
[  530.412987] [<80498cf4>] (tcp_data_queue) from [<8049b274>] (tcp_rcv_established+0x148/0x610)
[  530.421681] [<8049b274>] (tcp_rcv_established) from [<804a353c>] (tcp_v4_do_rcv+0x14c/0x368)
[  530.432097] [<804a353c>] (tcp_v4_do_rcv) from [<804a5f5c>] (tcp_v4_rcv+0x864/0x878)
[  530.443525] [<804a5f5c>] (tcp_v4_rcv) from [<80481b94>] (ip_local_deliver_finish+0xec/0x2d8)
[  530.455839] [<80481b94>] (ip_local_deliver_finish) from [<804822e8>] (ip_local_deliver+0x4c/0xac)
[  530.470515] [<804822e8>] (ip_local_deliver) from [<80481ee0>] (ip_rcv_finish+0x160/0x3ec)
[  530.484499] [<80481ee0>] (ip_rcv_finish) from [<80482634>] (ip_rcv+0x2ec/0x4a4)
[  530.494793] [<80482634>] (ip_rcv) from [<8044b990>] (__netif_receive_skb_core+0x600/0x824)
[  530.508855] [<8044b990>] (__netif_receive_skb_core) from [<8044bbd4>] (__netif_receive_skb+0x20/0x70)
[  530.523942] [<8044bbd4>] (__netif_receive_skb) from [<8044d428>] (process_backlog+0xa4/0x170)
[  530.538341] [<8044d428>] (process_backlog) from [<8044d22c>] (net_rx_action+0x13c/0x294)
[  530.552266] [<8044d22c>] (net_rx_action) from [<80028be8>] (__do_softirq+0x12c/0x384)
[  530.565919] [<80028be8>] (__do_softirq) from [<800291c4>] (irq_exit+0xc0/0x110)
[  530.576236] [<800291c4>] (irq_exit) from [<80066be8>] (__handle_domain_irq+0x8c/0xdc)
[  530.589883] [<80066be8>] (__handle_domain_irq) from [<80008564>] (asm_do_IRQ+0x2c/0x30)
[  530.603690] [<80008564>] (asm_do_IRQ) from [<805261b4>] (__irq_svc+0x34/0x14c)
[  530.613931] Exception stack(0x807b5f20 to 0x807b5f68)
[  530.621898] 5f20: 807d9ccc 00000000 00000000 00000000 807b4000 808174c0 807d8dd4 ffffffff
[  530.633890] 5f40: 808128dc 807d8d74 8052b828 807b5f74 807b5f68 807b5f68 8000fc78 8000fc7c
[  530.645886] 5f60: 60000013 ffffffff
[  530.651277] [<805261b4>] (__irq_svc) from [<8000fc7c>] (arch_cpu_idle+0x30/0x4c)
[  530.662404] [<8000fc7c>] (arch_cpu_idle) from [<8005b690>] (cpu_startup_entry+0x1e4/0x264)
[  530.674430] [<8005b690>] (cpu_startup_entry) from [<8051c434>] (rest_init+0x80/0x98)
[  530.685951] [<8051c434>] (rest_init) from [<80754cb4>] (start_kernel+0x370/0x3d8)
[  530.697204] ---[ end trace 7f7d5ec0232a9b0c ]---

[lucazen]

Hi,
after a week of test in which i streamed once a day, i haven't had evidence of new errors.

This is my settings:

pi@raspzen ~ $ cat /boot/cmdline.txt 
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait smsc95xx.turbo_mode=N

and /etc/samba/smb.conf

 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65535 SO_SNDBUF=65535 SO_KEEPALIVE
        write cache size = 2097152 # 2MB - you should test some values it's pretty memory intesive
        use sendfile = yes

Thanks,
Luca

[popcornmix]

Any difference in performance with "smsc95xx.turbo_mode=N"?
Our belief is that this doesn't make much difference, but it's good to have it confirmed.

[lucazen]

Well, I've no concrete evidences or measurement about performances, but I can't feel sensible degradation of the performance. Probably depends also on the load of the raspberry net interfaces.

So, i think you can consider it as a confirmation of your belief.

[Razican]

I had to put smsc95xx.turbo_mode=N becouse I was having other type of errors (keyvent drops) but I wasn't able to check if it was changing the performance.

[popcornmix]

Latest firmware includes the patch from Steve and disables smsc95xx.turbo_mode by default (can be enabled with smsc95xx.turbo_mode=Y on command line).

Please rpi-update and report if you have any errors in dmesg log, and if performance seems okay.

[Razican]

I cannot tell about performance (I currently have issues with #777), but this error has disappeared from dmesg.

[popcornmix]

@simonhiggs can you test?

[analyser-]

I've updated to kernel with rpi-update:
$ uname -a
Linux xyz 3.18.11+ #780 PREEMPT Mon Apr 20 14:46:05 BST 2015 armv6l GNU/Linux

And my raspberry 2 model b, running syncthing, is now up:
$ uptime
21:28:39 up 2 days, 8:04, 1 user, load average: 1.95, 1.89, 1.79

And so far I did not get this error. This bug seems to be fixed till now.