Downloading large files to spiffs via https #5175

MarcFinns · 2018-09-27T06:51:07Z

I am observing at behaviour that I would like to confirm being as expected.
I need to download a file from an HTTPS service into SPIFFS. This file is fairly large and when downloading using HTTP it is not a problem. This is what I have done in other projects for example with the json streaming parser. It looks like HTTPS is instead loading the whole thing in RAM, and then it crashes.
Is this what others are experiencing/ the expected behaviour? Is there a way to do it without lording it in RAM? ?
Thank you.

d-a-v · 2018-09-27T10:25:03Z

Please more details, fill the issue template !
Are you using BearSSL or AxTLS ?

MarcFinns · 2018-09-27T18:45:29Z

Apologies. Here it is.

Basic Infos

This issue complies with the issue POLICY doc.
I have read the documentation at readthedocs and the issue is not addressed there.
I have tested that the issue is present in current master branch (aka latest git).
I have searched the issue tracker for a similar issue.
If there is a stack dump, I have decoded it.
I have filled out all fields below.

Platform

Hardware: [ESP-12]
Core Version: [2.4.2]
Development Env: [Arduino IDE]
Operating System: [MacOS]

Settings in IDE

Module: [Nodemcu 1.0]
Flash Mode: [dio]
Flash Size: [4MB]
lwip Variant: [v2 Lower Memory]
Reset Method: [nodemcu]
Flash Frequency: [20Mhz]
CPU Frequency: [80Mhz]
Upload Using: [SERIAL]
Upload Speed: [other] (serial upload only)

Problem Description

I need to download a file from an HTTPS service directly into SPIFFS, as I used to do successfully with plain HTTP with files larger than the available RAM (e.g. json files).
The sketch shows about 30K free heap before the download starts. The download fails if the file is above 8k or so. The sketch gets stuck, no stack dump. Enabling the serial debug shows a ram issue (ssl->need_bytes=10464 > 6859).
It seems the file is loaded first into RAM and does not go directly to SPIFFS. Is this the expected behaviour? Is there a way to download it without copying it into RAM?

MCVE Sketch

#define FS_NO_GLOBALS
#include <FS.h>

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include <Arduino.h>

char* myssid = "GuestNet";
char* mypass = "xxxxx";

WiFiClientSecure client;

void setup()
{

  // Use some ram to simulate the full sketch utilisation
  int kb = 6;
  int *myArr = ( int* ) malloc( sizeof( int ) * kb * 500 );
  myArr[ 500 * kb - 1 ] = 4;

  Serial.begin(115200);
  delay(1000);

  Serial.println("START - Free Heap: " + String(ESP.getFreeHeap()));
  SPIFFS.begin();
  //SPIFFS.format();

  // Connect to WiFi
  Serial.print("Connecting to ");
  Serial.println(myssid);

  WiFi.mode(WIFI_STA);
  WiFi.begin(myssid, mypass);

  // Wait for connection...
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(500);
    Serial.print(".");
  }
  Serial.println(".");
  Serial.println("Connected to network " + String(WiFi.SSID()) + " with address " + String(WiFi.localIP().toString()));
}


void loop()
{

  // Download file
  downloadImage("https://api.buienradar.nl/image/1.0/24hourforecastmapnl/jpg/?t=2200&w=240&h=240&type=rain",
                "A8 A5 14 C3 A5 87 E5 11 7E 93 4A D8 90 EF FE BB 9F EA 12 14",
                "/image1.jpg" );


  Serial.println("SPIFFS dir listing:");

  String fileName;
  fs::Dir dir = SPIFFS.openDir(F("/"));
  while (dir.next())
  {
    fileName = dir.fileName();

    Serial.println(fileName);
  }

  Serial.println( " -----  ALL DONE ----- ");
  while (true)
    delay(1000);
}


bool downloadImage(String url, String fingerprint, String filename)
{

  HTTPClient http;
  bool outcome = true;

  // configure server and url
  http.begin(url, fingerprint);

  // start connection and send HTTP header
  int httpCode = http.GET();

  Serial.println(("[HTTP] GET DONE with code " + String(httpCode)));

  if (httpCode > 0)
  {
    Serial.println(F("-- >> OPENING FILE..."));

    SPIFFS.remove(filename);
    fs::File f = SPIFFS.open(filename, "w+");
    if (!f)
    {
      Serial.println(F("file open failed"));
      return false;
    }

    // HTTP header has been sent and Server response header has been handled
    Serial.printf("-[HTTP] GET... code: %d\n", httpCode);
    Serial.println("Free Heap: " + String(ESP.getFreeHeap()));

    // file found at server
    if (httpCode == HTTP_CODE_OK)
    {
      // get lenght of document (is -1 when Server sends no Content-Length header)
      int total = http.getSize();
      int len = total;

      Serial.println("HTTP SIZE IS " + String(total));

      // create buffer for read
      uint8_t buff[128] = { 0 };

      // get tcp stream
      WiFiClient * stream = http.getStreamPtr();

      // read all data from server
      while (http.connected() && (len > 0 || len == -1))
      {
        // get available data size
        size_t size = stream->available();

        if (size)
        {
          // read up to 128 byte
          int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));

          // write it to
          // f.write(buff, c);

          if (len > 0)
          {
            len -= c;
          }
        }
        delay(1);
      }

      Serial.println("[HTTP] connection closed or file end.");
    }
    f.close();
  }
  else
  {
    Serial.println("[HTTP] GET... failed, error: " + http.errorToString(httpCode));
    outcome = false;
  }
  http.end();
  return outcome;
}

Debug Messages

connected with GuestNet, channel 5
dhcp client start...
wifi evt: 0
ip:10.0.2.207,mask:255.255.255.0,gw:10.0.2.1
wifi evt: 3
START - Free Heap: 30424
SPIFFSImpl: allocating 512+240+1400=2152 bytes
SPIFFSImpl: mounting fs @200000, size=1fb000, block=2000, page=100
SPIFFSImpl: mount rc=0
Connecting to GuestNet
sta config unchangedscandone
..
Connected to network GuestNet with address 10.0.2.207
[HTTP-Client][begin] url: https://api.buienradar.nl/image/1.0/24hourforecastmapnl/jpg/?t=2300&w=240&h=240&type=rain
[HTTP-Client][begin] host: api.buienradar.nl port: 443 url: /image/1.0/24hourforecastmapnl/jpg/?t=2300&w=240&h=240&type=rain
[HTTP-Client][begin] httpsFingerprint: A8 A5 14 C3 A5 87 E5 11 7E 93 4A D8 90 EF FE BB 9F EA 12 14
[hostByName] request IP for: api.buienradar.nl
[hostByName] Host: api.buienradar.nl IP: 80.67.74.65
:ref 1
:ref 2
State:	sending Client Hello (1)
:wr 98 98 0
:wrc 98 98 0
:ack 98
:rn 536
:rch 536, 536
:rch 1072, 336
:rch 1408, 536
:rd 5, 1944, 0
:rdi 536, 5
:rd 80, 1944, 5
:rdi 531, 80
State:	receiving Server Hello (2)
:rd 5, 1944, 85
:rdi 451, 5
:rd 1854, 1944, 90
:rdi 446, 446
:c 446, 536, 1944
:rdi 536, 536
:c 536, 536, 1408
:rdi 336, 336
:c 336, 336, 87:rch 872, 536
:rch 1408, 464
2
:rdi 536, 536
:c 536, 536, 1536
:rd 991, 1000, 0
:rdi 536, 536
:c 536, 536, 1000
:rdi 464, 455
State:	receiving Certificate (11)
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		buienradar.nl
Organization (O):		RTL Nederland B.V.
Organizational Unit (OU):	Weer en Verkeer
Location (L):			Hilversum
Country (C):			NL
Basic Constraints:		CA:FALSE, pathlen:10000
Key Usage:			critical, Digital Signature, Key Encipherment
Subject Alt Name:		buienradar.nl *.routeradar.nl routeradar.nl *.buienradar.be *.buienradar.nl buienradar.be 
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		DigiCert SHA2 Secure Server CA
Organization (O):		DigiCert Inc
Country (C):			US
Not Before:			Wed Dec 13 00:00:00 2017
Not After:			Thu Dec 13 12:00:00 2018
RSA bitsize:			2048
Sig Type:			SHA256
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		DigiCert SHA2 Secure Server CA
Organization (O):		DigiCert Inc
Country (C):			US
Basic Constraints:		critical, CA:TRUE, pathlen:0
Key Usage:			critical, Digital Signature, Key Cert Sign, CRL Sign
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		DigiCert Global Root CA
Organization (O):		DigiCert Inc
Organizational Unit (OU):	www.digicert.com
Country (C):			US
Not Before:			Fri Mar  8 12:00:00 2013
Not After:			Wed Mar  8 12:00:00 2023
RSA bitsize:			2048
Sig Type:			SHA256
:rd 5, 464, 455
:rdi 9, 5
:rd 4, 464, 460
:rdi 4, 4
:c0 4, 464
State:	receiving Server Hello Done (14)
State:	sending Client Key Exchange (16)
:wr 267 267 0
:wrc 256 267 0
:wrc 11 11 0
:wr 6 6 0
:wrc 6 6 0
State:	sending Finished (16)
:wr 85 85 0
:wrc 85 85 0
:ack 267
:ack 91
:rn 91
:rd 5, 91, 0
:rdi 91, 5
:rd 1, 91, 5
:rdi 86, 1
:rd 5, 91, 6
:rdi 85, 5
:rd 80, 91, 11
:rdi 80, 80
:c0 80, 91
State:	receiving Finished (16)
[HTTP-Client] connected to api.buienradar.nl:443
:ref 3
domain name: 'api.buienradar.nl'
SAN 0: 'buienradar.nl', no match
SAN 1: '*.routeradar.nl', no match
SAN 2: 'routeradar.nl', no match
SAN 3: '*.buienradar.be', no match
:ur 3
[HTTP-Client] sending request header
-----
GET /image/1.0/24hourforecastmapnl/jpg/?t=2300&w=240&h=240&type=rain HTTP/1.1
Host: api.buienradar.nl
User-Agent: ESP8266HTTPClient
Connection: close
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0

-----
:wr 261 261 0
:wrc 256 261 0
:wrc 5 5 0
:ack 261
:rn 536
:rch 536, 536
:rch 1072, 336
:rch 1408, 536
:rch 1944, 173
:rd 5, 2117, 0
:rdi 536, 5
:rd 2112, 2117, 5
:rdi 531, 531
:c 531, 536, 2117
:rdi 536, 536
:c 536, 536, 1581
:rdi 336, 336
:c 336, 336, 1045
:r:rch 709, 536
:rch 1245, 536
di 536, 536
:c 536, 536, 1781
:rdi 173, 173
:c 173, 173, :rch 1245, 536
1245
:wcs ra 2056
[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Type: image/jpeg'
[HTTP-Client][handleHeaderResponse] RX: 'Last-Modified: Thu, 27 Sep 2018 18:43:13 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Server: Microsoft-IIS/10.0'
[HTTP-Client][handleHeaderResponse] RX: 'X-AspNetMvc-Version: 5.2'
[HTTP-Client][handleHeaderResponse] RX: 'X-Server: br-web005'
[HTTP-Client][handleHeaderResponse] RX: 'X-AspNet-Version: 4.0.30319'
[HTTP-Client][handleHeaderResponse] RX: 'X-Powered-By: ASP.NET'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 11872'
[HTTP-Client][handleHeaderResponse] RX: 'Cache-Control: public, max-age=900'
[HTTP-Client][handleHeaderResponse] RX: 'Expires: Thu, 27 Sep 2018 18:58:13 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Date: Thu, 27 Sep 2018 18:43:13 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Connection: close'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Max-Age: 86400'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Credentials: false'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Methods: GET,POST,PUT'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Origin: *'
[HTTP-Client][handleHeaderResponse] RX: ''
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 11872
[HTTP] GET DONE with code 200
-- >> OPENING FILE...
-[HTTP] GET... code: 200
Free Heap: 10504
HTTP SIZE IS 11872
:rd 5, 1608, 0
:rdi 536, 5
ssl->need_bytes=10464 > 6859
:oom(11264)@?
failed to grow plain buffer
:rch 1608, 363
pm open,type:2 0

d-a-v · 2018-09-28T13:36:26Z

ssl->need_bytes=10464 > 6859
:oom(11264)@?

This is an AxTLS known bug. You should try BearSSL.

Check this.

MarcFinns · 2018-09-28T23:32:18Z

Hi d-a-v, thanks for the hint.
I did implement it and in fact i confirm it works!
HOWEVER i see a behaviour that i would like to confirm is as expected and not a memory leak.
When the line: BearSSL::WiFiClientSecure client; is executed, heap goes down by about 5K.
These 5K are never deallocated, even if the class that downloaded is long gone.
To me this seems not right.

devyte · 2018-09-29T03:07:43Z

CC @earlephilhower

earlephilhower · 2018-09-29T03:13:28Z

@MarcFinns that's a feature, not a bug. :)

~4.5K are taken from the heap to use as a 2nd stack as long as there is at least one BearSSL::WiFi{Client,Server}Secure. If you have a global WiFiClientSecure, it will obviously never go away so you'll end up with that chunk never freeing. I'd recommend a dynamically allocated object (i.e. new ...) or one that's stack-local inside a function that exits, to ensure it's deleted. The 4.5K is constant no matter how many objects you've got, so you could have, say, 20 BearSSL::WiFiClientSecures and it would still be 4.5K.

If you've got something like

BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure;
...
(do some work)
...
delete client;

and still seeing the heap used, that's a bug and I'll need a MCVE to reproduce and debug.

MarcFinns · 2018-09-29T07:37:38Z

It seems a bug. You get the behavior with both types of allocation and it can be reproduced with 2 lines of code - see below. Is this sufficient to report the bug?

#include <ESP8266WiFi.h>
void setup()
{
Serial.begin(115200);
delay(1000);
Serial.println("Setup - before instantiation - Free Heap: " + String(ESP.getFreeHeap()));
BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure;
client->stop();
Serial.println("Setup - After instantiation - Free Heap: " + String(ESP.getFreeHeap()));
delete client;
Serial.println("Setup - After delete - Free Heap: " + String(ESP.getFreeHeap()));
}

void loop()
{
Serial.println("Loop - Free residual Heap: " + String(ESP.getFreeHeap()));
while (1)
delay(1000);
}

earlephilhower · 2018-09-29T15:03:52Z

Thanks, that'll do. Appreciate the check! I'll look into it.

ZinggJM · 2018-10-22T07:58:35Z

@MarcFinns,

large file to Spiffs works for you with BearSSL? up to what size?

It does not work for me, issue #4814.

MarcFinns · 2018-10-22T11:49:29Z

I used it to download every half hour 24 images of a weather animation. In the web service I can specify the jpg size and I now use 240x200. This means images are 10-25k depending on the cloud patterns. It would not work with axTLS but it works nicely with BearSSL.
Please note that with bear you can specify the size of the receiving buffer and read from network/write to spiffs in small chunks.
How large are your files?

ZinggJM · 2018-10-22T14:43:09Z

Sorry, I found out in the meantime that my GxEPD2_32_Spiffs_Loader example works with BearSSL.
I still have an issue, e.g. with GxEPD_WiFi_Example, when I use it with an e-paper that needs a 15k graphics buffer. I have seen an issue that BearSSL takes up to 48k from heap.

The example bitmaps I download are up to 230k in size.

MarcFinns · 2018-10-22T19:54:33Z

BearSSL works with less than that. Have you tried reducing the buffers, not checking the fingerprint and reading the file in chunks?
see example below, hope it helps. let me know.

int getForecastImage(String host, String resource, String filename)
{

  int contentLength = -1;
  int httpCode;

  // HTTPS but dont verify certificates
  BearSSL::WiFiClientSecure client;
  client.setBufferSizes(1024, 256);
  client.setInsecure();

  // Connect
  client.connect(host, 443);

  // If not connected, return
  if (!client.connected())
  {
    client.stop();
    errLog("HTTPS: Can't connect");
    return -1;
  }

  // HTTP GET
  client.print(F("GET "));
  client.print(resource);
  client.print(F(" HTTP/1.1\r\nHost: "));
  client.print(host);
  client.print(F("\r\nUser-Agent: ESP8266\r\n"));
  client.print(F("\r\n"));

  // Handle headers
  while (client.connected())
  {
    String header = client.readStringUntil('\n');
    if (header.startsWith(F("HTTP/1.")))
    {
      httpCode = header.substring(9, 12).toInt();
      if (httpCode != 200)
      {
        errLog(String(F("HTTP GET code=")) + String(httpCode));
        client.stop();
        return -1;
      }
    }
    if (header.startsWith(F("Content-Length: ")))
    {
      contentLength = header.substring(15).toInt();
    }
    if (header == F("\r"))
    {
      break;
    }
  }

  if (!(contentLength > 0))
  {
    errLog(F("HTTP content length=0"));
    client.stop();
    return -1;
  }

  // Open file for write
  fs::File f = SPIFFS.open(filename, "w+");
  if (!f)
  {
    errLog( F("file open failed"));
    client.stop();
    return -1;
  }

  // Download file
  int remaining = contentLength;
  int received;
  uint8_t buff[512] = { 0 };

  // read all data from server
  while (client.available() && remaining > 0)
  {
    // read up to buffer size
    received = client.readBytes(buff, ((remaining > sizeof(buff)) ? sizeof(buff) : remaining));

    // write it to file
    f.write(buff, received);

    if (remaining > 0)
    {
      remaining -= received;
    }
    yield();
  }

  if (remaining != 0)
    errLog("[HTTP] Img truncated -" + String(remaining));

  // Close SPIFFS file
  f.close();

  // Stop client
  client.stop();

  return (remaining == 0 ? contentLength : -1);
}

ZinggJM · 2018-10-23T07:33:47Z

Thank you for your response.
The site I download from does not support MFLN, returns false,

I tried to update my MCVE sketch to use with BearSSL to show the issue. But my MCVE sketch works.
So I need to analyze my issue with download to draw on e-paper from start.

devyte · 2018-10-23T12:01:16Z

@ZinggJM this thread has gone in several directions already, and is getting a bit long. Given your previous comment, I'm closing it.
If after your analysis you think there is still a problem, please open a new issue, fill out the template, and provide a fresh explanation and MCVE.

ZinggJM · 2018-10-24T06:07:45Z

I had used this method (from class Client):
virtual int read(uint8_t *buf, size_t size) = 0;
If I use single byte reads instead, checking for available, it works.
virtual int available() = 0; virtual int read() = 0;

devyte added the waiting for feedback Waiting on additional info. If it's not received, the issue may be closed. label Sep 29, 2018

earlephilhower mentioned this issue Sep 29, 2018

Really free stack after last BearSSL obj destroyed #5185

Merged

d-a-v assigned d-a-v and earlephilhower and unassigned d-a-v Oct 22, 2018

devyte closed this as completed Oct 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Downloading large files to spiffs via https #5175

Downloading large files to spiffs via https #5175

MarcFinns commented Sep 27, 2018

d-a-v commented Sep 27, 2018

MarcFinns commented Sep 27, 2018 •

edited by d-a-v

Loading

d-a-v commented Sep 28, 2018

MarcFinns commented Sep 28, 2018

devyte commented Sep 29, 2018

earlephilhower commented Sep 29, 2018

MarcFinns commented Sep 29, 2018

earlephilhower commented Sep 29, 2018

ZinggJM commented Oct 22, 2018

MarcFinns commented Oct 22, 2018

ZinggJM commented Oct 22, 2018 •

edited

Loading

MarcFinns commented Oct 22, 2018 •

edited

Loading

ZinggJM commented Oct 23, 2018

devyte commented Oct 23, 2018

ZinggJM commented Oct 24, 2018

Downloading large files to spiffs via https #5175

Downloading large files to spiffs via https #5175

Comments

MarcFinns commented Sep 27, 2018

d-a-v commented Sep 27, 2018

MarcFinns commented Sep 27, 2018 • edited by d-a-v Loading

Basic Infos

Platform

Settings in IDE

Problem Description

MCVE Sketch

Debug Messages

d-a-v commented Sep 28, 2018

MarcFinns commented Sep 28, 2018

devyte commented Sep 29, 2018

earlephilhower commented Sep 29, 2018

MarcFinns commented Sep 29, 2018

earlephilhower commented Sep 29, 2018

ZinggJM commented Oct 22, 2018

MarcFinns commented Oct 22, 2018

ZinggJM commented Oct 22, 2018 • edited Loading

MarcFinns commented Oct 22, 2018 • edited Loading

ZinggJM commented Oct 23, 2018

devyte commented Oct 23, 2018

ZinggJM commented Oct 24, 2018

MarcFinns commented Sep 27, 2018 •

edited by d-a-v

Loading

ZinggJM commented Oct 22, 2018 •

edited

Loading

MarcFinns commented Oct 22, 2018 •

edited

Loading