PrusaLink with camera goes offline

[ingbrzy]

Hello..

My setup:

• Prusa MK3S+
• RPi Zero 2 W
• RPi camera 3 module
• PrusaLink 0.7.1 RC1
• Connected to Wi-Fi

I have an issue that PrusaLink gets disconnected from Wi-Fi after some time when using camera...

logs in attachment..

syslog.txt
daemon.log

[ingbrzy]

Maybe related with wlan power saving..

#811

[ingbrzy]

As I see in the log this [wlan](wlan0: carrier lost)

Oct 4 08:16:37 prusa01 dhcpcd[561]: wlan0: carrier lost Oct 4 08:16:37 prusa01 dhcpcd[561]: wlan0: deleting address fe80::51b2:14f:c16c:b2c7 Oct 4 08:16:37 prusa01 avahi-daemon[347]: Withdrawing address record for fe80::51b2:14f:c16c:b2c7 on wlan0. Oct 4 08:16:37 prusa01 avahi-daemon[347]: Leaving mDNS multicast group on interface wlan0.IPv6 with address fe80::51b2:14f:c16c:b2c7. Oct 4 08:16:37 prusa01 avahi-daemon[347]: Interface wlan0.IPv6 no longer relevant for mDNS. Oct 4 08:16:38 prusa01 avahi-daemon[347]: Withdrawing address record for 10.192.35.20 on wlan0. Oct 4 08:16:38 prusa01 avahi-daemon[347]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 10.192.35.20. Oct 4 08:16:38 prusa01 avahi-daemon[347]: Interface wlan0.IPv4 no longer relevant for mDNS. Oct 4 08:16:38 prusa01 dhcpcd[561]: wlan0: deleting route to 10.192.35.0/24 Oct 4 08:16:38 prusa01 dhcpcd[561]: wlan0: deleting default route via 10.192.35.254 Oct 4 08:16:38 prusa01 avahi-daemon[347]: Got SIGHUP, reloading. Oct 4 08:16:38 prusa01 avahi-daemon[347]: No service file found in /etc/avahi/services. Oct 4 08:16:41 prusa01 prusa.link.printer_adapter.ip_updater[567]: WARNING: Failed getting the local IP, are we connected to LAN? {update_ip():130} Oct 4 08:16:41 prusa01 prusa.link.printer_adapter.lcd_printer[567]: WARNING: Displaying an error message No LAN access {_check_errors():310}

according this post

https://dontpressthat.wordpress.com/2017/11/03/prevent-raspberry-pi-dropping-wifi/

wlan0 set power_save off should help..

@TojikCZ could you build a test img with this fix to confirm?

Thank you

[ingbrzy]

more interesting info from log related to camera..

Oct 4 07:54:41 prusa01 camera_controller[567]: WARNING: Camera: RaspberryPi Camera: imx708 id: quz390woShCB looks stuck. Disconnecting {disconnect_stuck_cameras():93} Oct 4 07:54:41 prusa01 interesting_logger[567]: WARNING: Interesting log triggered by a stuck instruction {check_status():640} Oct 4 07:54:41 prusa01 interesting_logger[567]: WARNING: Was[DEBUG]: SDK queue looks stuck -> no telemetry {check_status():640} Oct 4 07:54:41 prusa01 camera_controller[567]: WARNING: Skipping camera RaspberryPi Camera: imx708 because it's busy {trigger_pile():149}

Oct 4 07:58:30 prusa01 camera_controller[567]: ERROR: Unexpected exception caught in SDK snapshot loop! {snapshot_loop():186}#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn#012 conn = connection.create_connection(#012 File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection#012 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):#012 File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo#012 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):#012socket.gaierror: [Errno -3] Temporary failure in name resolution#012#012During handling of the above exception, another exception occurred:#012#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen#012 httplib_response = self._make_request(#012 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request#012 self._validate_conn(conn)#012 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn#012 conn.connect()#012 File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect#012 conn = self._new_conn()#012 File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn#012 raise NewConnectionError(#012urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x716190b8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution#012#012During handling of the above exception, another exception occurred:#012#012Traceback (most recent call last):#012 File "/home/jo/.local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send#012 resp = conn.urlopen(#012 File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen#012 retries = retries.increment(#012 File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment#012 raise MaxRetryError(_pool, url, error or ResponseError(cause))#012urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='connect.prusa3d.com', port=443): Max retries exceeded with url: /c/snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x716190b8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))#012#012During handling of the above exception, another exception occurred:#012#012Traceback (most recent call last):#012 File "/home/jo/.local/lib/python3.9/site-packages/prusa/connect/printer/camera_controller.py", line 174, in snapshot_loop#012 res = item.send(self.session, self.server)#012 File "/home/jo/.local/lib/python3.9/site-packages/prusa/connect/printer/camera.py", line 70, in send#012 res = conn.request(method=self.method,#012 File "/home/jo/.local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request#012 resp = self.send(prep, **send_kwargs)#012 File "/home/jo/.local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send#012 r = adapter.send(request, **kwargs)#012 File "/home/jo/.local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send#012 raise ConnectionError(e, request=request)#012requests.exceptions.ConnectionError: HTTPSConnectionPool(host='connect.prusa3d.com', port=443): Max retries exceeded with url: /c/snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x716190b8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

[TojikCZ]

If there is no internet, the snapshot upload will fail. And on top of that, we were moving Connect today. Changing where the DNS entries point to, so there really was a window, where the name resolution should have been failing. If you catch it doing that any other day, while connected to the internet, i will be very surprised, just not today 😬

[atommoyer]

I am also seeing this happen.

I am using a RPi0W from prusa with the 0.7.2 prusalink/up to date software and firmware which had been 100% successful for weeks. Then I installed the new wide angle NoIR picamera v3 tonight. Everything was going well.

A print abruptly terminated without error message or home-ing. I was no longer connected over prusa-link. I had to restart the printer to reset everything. It took about 30 minutes to fail. The camera had been feeding successfully at full resolution and 10s interval. Chalked it up to a bug on first install, but then it happened again when I started a print.

Now, I have put the resolution down to as low as it can go and the interval is on layer change. I have not had a disconnect yet after a couple prints.

Ill update after some experimenting.

[ingbrzy]

I am also seeing this happen.

I am using a RPi0W from prusa with the 0.7.2 prusalink/up to date software and firmware which had been 100% successful for weeks. Then I installed the new wide angle NoIR picamera v3 tonight. Everything was going well.

A print abruptly terminated without error message or home-ing. I was no longer connected over prusa-link. I had to restart the printer to reset everything. It took about 30 minutes to fail. The camera had been feeding successfully at full resolution and 10s interval. Chalked it up to a bug on first install, but then it happened again when I started a print.

Now, I have put the resolution down to as low as it can go and the interval is on layer change. I have not had a disconnect yet after a couple prints.

Ill update after some experimenting.

thank you for your feedback, will check it on mine as well and let you know if it is caused due to resolution for me as well.. then we can consider it as a bug..

[atommoyer]

I have had it happen 3 more times (5 total).

Happened with the lowest resolution/lowest frequency photo updates. I also have tried with and without the "prusa connect" toggle button on the camera settings page.

Seems like I will just have to disconnect the camera. I will update if I still see the prusalink disconnection issue without the camera.

I am also sending the print directly from prusa slicer which I only started doing recently, so it is remotely possible that is the problem too.

[ingbrzy]

Got some news..

According this thread,
https://forums.raspberrypi.com/viewtopic.php?t=338451&sid=cd5cd77cf4764c660d033ba22d97df78

if you see in the log this line, it may be a HW issue of RPi...

Oct 4 08:17:06 prusa01 kernel: [ 11.250157] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43436s-sdio.clm_blob failed with error -2
Oct 4 08:17:06 prusa01 kernel: [ 11.424199] brcmfmac: brcmf_c_process_clm_blob: no clm_blob available (err=-2), device may have limited channels available

[ingbrzy]

linking these issue threads here for more investigation

raspberrypi/linux#1342

https://gist.github.com/mkb/40bf48bc401ffa0cc4d3

[ingbrzy]

seems this workaround works for some users..
https://gist.github.com/mkb/40bf48bc401ffa0cc4d3#file-gistfile2-md

[ingbrzy]

Got some news..

According this thread, https://forums.raspberrypi.com/viewtopic.php?t=338451&sid=cd5cd77cf4764c660d033ba22d97df78

if you see in the log this line, it may be a HW issue of RPi...

Oct 4 08:17:06 prusa01 kernel: [ 11.250157] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43436s-sdio.clm_blob failed with error -2 Oct 4 08:17:06 prusa01 kernel: [ 11.424199] brcmfmac: brcmf_c_process_clm_blob: no clm_blob available (err=-2), device may have limited channels available

Actually this error message seems not related with this issue.. I have 3 MK3S+Camera and only one has camera issue but this message is in logs from all 3 printers..

[atommoyer]

Interesting. Have you tested if it is specific to a printer/einsy, a raspberry pi, or a camera between your various setups?

I havent had issue with a v2.1 camera, but I would prefer to use the v3.

[TojikCZ]

I have turned the power saving off in the newer 0.7.2 image, so you can try to flash it (OTA just updates Link, not the image), but i have not done anything with any hooks that might have been present. Also the next update will use NetworkManager (as that's what's present in the base image from now on), so we might get a completely different set of bugs. Yaaaay

[ingbrzy]

I have turned the power saving off in the newer 0.7.2 image, so you can try to flash it (OTA just updates Link, not the image), but i have not done anything with any hooks that might have been present. Also the next update will use NetworkManager (as that's what's present in the base image from now on), so we might get a completely different set of bugs. Yaaaay

Today I have reinstalled all 3 MK3S+ Prusalink to 0.7.2 by full image as you pointed out that Wi-Fi power saving is in the system and not in PrusaLink.. Also reduced PiCam image resolution to 1920x1080 to avoid overheating due to 12MPx image processing.. for now all 3 are online all day..

[ingbrzy]

Mysteriously solved by itself since 25.01.2024