Use http instead of https to avoid memory errors

[helgibbons]

No description provided.

[jonbhanson]

@helgibbons My Pico W with pimoroni's uf2 will successfully fetch data one time from my Firebase Cloud Function api, which is accessed through HTTPS. However, subsequent fetches are failing with the error -29312, 'MBEDTLS_ERR_SSL_CONN_EOF'. I see that in this example you moved from https to http for memory errors. Can you describe what was going on? Were some fetches in this example successful and then subsequent fetches failed with an error like in my case? Is this error coming from a bug in the pimoroni uf2 and are there any plans to fix it?

[Gadgetoid]

@jonbhanson unfortunately it's got nothing to do with our custom .uf2 and everything to do with SSL just needing more memory than we can reasonably expect from a 262k microcontroller that has ~200k allocated to MicroPython heap storage and flags.

If you run the following minimal example:

import gc
import rp2
import time
import network
import urequests

SSID = "your-ssid"
PSK = "your-psk"
TEST_URL = "https://api.open-meteo.com/v1/forecast?latitude=53.38609085276884&longitude=-1.4239983439328177&current_weather=true&timezone=auto"

rp2.country("GB")
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, PSK)

while not wlan.isconnected():
    print("Connecting...")
    time.sleep(1.0)

rcount = 1

while True:
    print(f"Requests: {rcount}")
    r = urequests.get(TEST_URL)
    del r
    gc.collect()
    time.sleep(1.0)
    rcount += 1

It will fail at ~5 attempts on both our .uf2 and the Pico W nightly.

⚠️ This is because it doesn't close the socket. Calling r.content, r.json() or r.text will implicitly close the underlying raw socket, Calling r.close() will explicitly close it.

Though I'll usually see something like:

OSError: (-17040, 'MBEDTLS_ERR_RSA_PUBLIC_FAILED+MBEDTLS_ERR_MPI_ALLOC_FAILED')

Or:

OSError: [Errno 12] ENOMEM

It looks like the underlying SSL wrapper is not being particularly conscientious about freeing memory and we're either chewing through the heap remaining to C (invisible and unmanageable to the user) or just fragmenting it until there's not a sufficient portion of RAM left.

Also on Pico W nightly, using urequest.urlopen I seem to get the same error as you:

OSError: (-29312, 'MBEDTLS_ERR_SSL_CONN_EOF')

But only intermittently, with the above errors being more prominent.

It's possible something weird with the socket management is going on there, IE: a failure to explicitly close() might - and I am just throwing ideas at the wall here - mean the networking hardware is still talking via an open socket to the server and getting an old response back to a new request.

I tend to prefer urequest.urlopen because it returns a socket you can then explicitly close and garbage collect. The following code:

⚠️ again, I'm wrong above- the socket is implicitly closed with urequests.get() as long as you grab the response content. The main difference with `urequest.urlopen is that you must explicitly close the socket.

ℹ️ The easy availability of readinto() (see below) may be a good reason to prefer urllib.urequest.urlopen() over urequests.get() since it allows for better up-front memory management, but you could probably do r.raw.readinto(buffer) using the latter anyway, since the raw socket property is right there: https://github.com/micropython/micropython-lib/blob/a5ef231e7d854bab0e6db95f1676173e2281ee3b/python-ecosys/urequests/urequests.py#L6

import gc
import rp2
import time
import network
from urllib import urequest

SSID = "your-ssid"
PSK = "your-psk"
TEST_URL = "https://api.open-meteo.com/v1/forecast?latitude=53.38609085276884&longitude=-1.4239983439328177&current_weather=true&timezone=auto"

rp2.country("GB")
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, PSK)

while not wlan.isconnected():
    print("Connecting...")
    time.sleep(1.0)

rcount = 1

while True:
    print(f"Requests: {rcount}")
    r = urequest.urlopen(TEST_URL)
    r.close()
    del r
    gc.collect()
    time.sleep(1.0)
    rcount += 1

You can get urequest (urllib.urequest in particular) (and not to be confused with the built in urequests) from here: https://github.com/micropython/micropython-lib/blob/master/micropython/urllib.urequest/urllib/urequest.py

Or if you have a WiFi connection on your Pico you can do:

import mip
mip.install('urllib.urequest')

I managed up to 25 requests using the above code with urllib.urequest so maybe that's your answer!

[jonbhanson]

@Gadgetoid thank you for taking the time to write this informative and helpful response! I will give your suggestion a try and see how it goes.

[jonbhanson]

BTW, here is the project I'm working on, just in case you're interested: Boiler monitor.

Now I'm working on the next revision of this project, which is a Pimoroni Display 2.0 screen to display my boiler's status:

[Gadgetoid]

That little monitor setup looks awesome! Thank you for sharing.

I spent a long time bashing my head against this stuff and needed a good excuse to formalise my reasons for preferring urllib 😆

A fairer test on the latter case might be:

import gc
import rp2
import time
import network
import urequest

SSID = "your-ssid"
PSK = "your-psk"
TEST_URL = "https://api.open-meteo.com/v1/forecast?latitude=53.38609085276884&longitude=-1.4239983439328177&current_weather=true&timezone=auto"

rp2.country("GB")
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, PSK)

while not wlan.isconnected():
    print("Connecting...")
    time.sleep(1.0)

rcount = 1

while True:
    print(f"Requests: {rcount}")
    r = urequest.urlopen(TEST_URL)
    data = r.read(1000)
    print(data[0:10])
    r.close()
    del data
    del r
    gc.collect()
    time.sleep(1.0)
    rcount += 1

Since this actually reads back and prints data from the socket.

If your API is pretty stable and has a known limit in the size of its responses then you can pre-allocate a bytearray and use readinto() to avoid the memory fragmentation potential of constantly allocating string buffers.

Here's the above example modified to do that:

import gc
import rp2
import time
import network
import urequest

SSID = "your-ssid"
PSK = "your-psk"
TEST_URL = "https://api.open-meteo.com/v1/forecast?latitude=53.38609085276884&longitude=-1.4239983439328177&current_weather=true&timezone=auto"

rp2.country("GB")
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, PSK)

while not wlan.isconnected():
    print("Connecting...")
    time.sleep(1.0)

rcount = 1
data = bytearray(2048)

while True:
    print(f"Requests: {rcount}")
    r = urequest.urlopen(TEST_URL)
    r.readinto(data)
    print(data[0:10])
    r.close()
    del r
    gc.collect()
    time.sleep(1.0)
    rcount += 1

It's normally pretty good practice to pre-allocate buffers on micro-controllers, but it feels very weird doing it in MicroPython. Nonetheless it can head off memory fragmentation and prevent unexpected server responses from chomping all your RAM since - IIRC - readinfo() observes the size of its target buffer..

[jonbhanson]

Once again, very helpful, thanks! My api response is pretty stable and under my control, so I may be able to make that work. 👍

[Gadgetoid]

In fact (I'm reminded via the MicroPython Discord) urequests.get() returns a Response() object wraps the raw socket and has an explicit close() method, but if you grab the .content() it should close for you. Again if I explicitly close, it'll keep running.

Edit: I have added some ⚠️ Warning notes into my above posts to try and tiptoe around my own misinformation.

[jonbhanson]

I tried your final example with my api endpoint (reserving a 6k byte array) and it worked fine. However, when I added the line to initialize the display

# set up the hardware
display = PicoGraphics(display=DISPLAY_PICO_DISPLAY_2, rotate=0)

it resumed failing again with the same error OSError: (-29312, 'MBEDTLS_ERR_SSL_CONN_EOF').

So maybe there is not enough memory onboard for the display code and the SSL overhead at the same time?

[Gadgetoid]

Pico Display 2 is, by default, RGB232 at 320x240 or 75k. Definitely not small. You could try the 4-bit palette mode to get the size down to 37.5k using:

display = PicoGraphics(display=DISPLAY_PICO_DISPLAY_2, pen_type=PEN_P4, rotate=0)

But that will give you just 16 colours to play with.

How big is your API payload?

[jonbhanson]

Thanks for the palette suggestion, that could really help when I get to the bottom of this, but at the moment there seems to be a different problem:

I created a "hello world" endpoint to see if the problem is payload size related.

If I use your example above, adding the display declaration to consume the associated memory, and poll your test url, the example runs fine. However, if I switch the URL to my "hello world" endpoint that has a payload of 20 bytes, I get the OSError: (-29312, 'MBEDTLS_ERR_SSL_CONN_EOF') error.

So, it seems something about my server response is causing this problem. I figured maybe it was related to encryption type, but it looks like both endpoints are using TLS 1.3. Do you have any thought on anything that could be changed in the server response to make it digestible?

[jonbhanson]

@Gadgetoid @helgibbons do you have any advice given the note above about different endpoints?