StreamHttpsClient.ino crashes

[Jeroen88]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: NodeMCU
• Core Version: git pull upstream HEAD
• Development Env: Arduino IDE
• Operating System: Ubuntu

Settings in IDE

• Module: Generic ESP8266 Module
• Flash Mode: qio
• Flash Size: 4MB/1MB
• lwip Variant: v2 Lower Memory
• Reset Method: ck
• Flash Frequency: 80Mhz
• CPU Frequency: 160MHz
• Upload Using: SERIAL
• Upload Speed: 230400 (serial upload only)

Problem Description

Running StreamHttpsClient.ino crashes. Remarkable is that when I enable Debug Level: SSL+HTTP_CLIENT it runs fine. @earlephilhower as requested!

MCVE Sketch

   StreamHTTPClient.ino

    Created on: 24.05.2015

*/

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>

ESP8266WiFiMulti WiFiMulti;

void setup() {

  Serial.begin(115200);
  // Serial.setDebugOutput(true);

  Serial.println();
  Serial.println();
  Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP("SSID", "PASSWORD");

}

void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    HTTPClient http;

    BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure ;

    bool mfln = client->probeMaxFragmentLength("tls.mbed.org", 443, 1024);
    Serial.printf("\nConnecting to https://tls.mbed.org\n");
    Serial.printf("Maximum fragment Length negotiation supported: %s\n", mfln ? "yes" : "no");
    if (mfln) {
      client->setBufferSizes(1024, 1024);
    }

    Serial.print("[HTTPS] begin...\n");

    // configure server and url
    const uint8_t fingerprint[20] = {0xEB, 0xD9, 0xDF, 0x37, 0xC2, 0xCC, 0x84, 0x89, 0x00, 0xA0, 0x58, 0x52, 0x24, 0x04, 0xE4, 0x37, 0x3E, 0x2B, 0xF1, 0x41};
    client->setFingerprint(fingerprint);

    if (http.begin(*client, "https://tls.mbed.org/")) {

      Serial.print("[HTTPS] GET...\n");
      // start connection and send HTTP header
      int httpCode = http.GET();
      if (httpCode > 0) {
        // HTTP header has been send and Server response header has been handled
        Serial.printf("[HTTPS] GET... code: %d\n", httpCode);

        // file found at server
        if (httpCode == HTTP_CODE_OK) {

          // get lenght of document (is -1 when Server sends no Content-Length header)
          int len = http.getSize();

          // create buffer for read
          static uint8_t buff[128] = { 0 };

          // get tcp stream
          WiFiClient * stream = client;

          // read all data from server
          while (http.connected() && (len > 0 || len == -1)) {
            // get available data size
            size_t size = stream->available();

            if (size) {
              // read up to 128 byte
              int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));

              // write it to Serial
              Serial.write(buff, c);

              if (len > 0) {
                len -= c;
              }
            }
            delay(1);
          }

          Serial.println();
          Serial.print("[HTTPS] connection closed or file end.\n");

        }
      } else {
        Serial.printf("[HTTPS] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
      }

      http.end();
    } else {
      Serial.printf("Unable to connect\n");
    }

    delete client;
  }

  delay(10000);
}

#include <Arduino.h>

void setup() {

}

void loop() {

}

Debug Messages

09:39:40.098 -> [HTTPS] connection closed or file end.
09:39:40.099 -> 
09:39:40.099 -> Exception (9):
09:39:40.099 -> epc1=0x40206c19 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000137 depc=0x00000000
09:39:40.100 -> 
09:39:40.101 -> ctx: cont 
09:39:40.101 -> sp: 3ffffcd0 end: 3fffffc0 offset: 01a0
09:39:40.102 -> 
09:39:40.103 -> >>>stack>>>
09:39:40.103 -> 3ffffe70:  3fff29ac 3fff1644 3fff007c 40208540  
09:39:40.104 -> 3ffffe80:  ffffffff 00000000 3ffef188 402049b8  
09:39:40.104 -> 3ffffe90:  3ffef2f0 0000013b 0000013b 40100538  
09:39:40.129 -> 3ffffea0:  ffffffff 3ffef1ac 3fff0204 40100a04  
09:39:40.130 -> 3ffffeb0:  00000000 00000000 3ffffee0 40207ee8  
09:39:40.131 -> 3ffffec0:  402015e2 3ffef1ac 3fff0204 40204c14  
09:39:40.131 -> 3ffffed0:  ffffffff 3ffef1ac 3fff0204 402028db  
09:39:40.132 -> 3ffffee0:  00000000 00000000 3fff0204 3ffeff84  
09:39:40.163 -> 3ffffef0:  0000000f 0000000c fe0001bb 40001388  
09:39:40.164 -> 3fffff00:  3ffeff9c 0000000f 00000001 3ffeffb4  
09:39:40.165 -> 3fffff10:  0000000f 00000005 3fff000c 0000000f  
09:39:40.167 -> 3fffff20:  00000000 3fff0024 0000001f 00000011  
09:39:40.168 -> 3fffff30:  3fff004c 0000000f 00000000 00000000  
09:39:40.196 -> 3fffff40:  00000000 00000000 ffffffff 4020a600  
09:39:40.197 -> 3fffff50:  00000000 00000000 37dfd9eb 8984ccc2  
09:39:40.198 -> 3fffff60:  5258a000 37e40424 41f12b3e 00000000  
09:39:40.199 -> 3fffff70:  00000000 00000000 40208104 3fffefa0  
09:39:40.200 -> 3fffff80:  00000008 00000000 3ffef1ac 40202f90  
09:39:40.229 -> 3fffff90:  3fffdad0 00000000 3ffef1ac 3ffef278  
09:39:40.230 -> 3fffffa0:  3fffdad0 00000000 3ffef270 40208190  
09:39:40.231 -> 3fffffb0:  feefeffe feefeffe 3ffe8504 40100a61  
09:39:40.232 -> <<<stack<<<
09:39:40.261 -> 
09:39:40.262 ->  ets Jan  8 2013,rst cause:2, boot mode:(1,6)
09:39:40.262 -> 
09:39:48.327 -> 
09:39:48.328 ->  ets Jan  8 2013,rst cause:4, boot mode:(1,6)
09:39:48.328 -> 
09:39:48.329 -> wdt reset

Exception decoder

Decoding stack results
0x40208540: std::__shared_count(__gnu_cxx::_Lock_policy)0>::~__shared_count() at /home/jeroen/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/1.20.0-26-gb404fb9-2/xtensa-lx106-elf/include/c++/4.8.2/bits/shared_ptr_base.h line 547
0x402049b8: std::__shared_ptr ::operator=(std::__shared_ptr &&) at /home/jeroen/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/1.20.0-26-gb404fb9-2/xtensa-lx106-elf/include/c++/4.8.2/bits/shared_ptr_base.h line 862
0x40100538: _umm_free at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/cores/esp8266/umm_malloc/umm_malloc.c line 1299
0x40100a04: free at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/cores/esp8266/umm_malloc/umm_malloc.c line 1759
0x40207ee8: operator delete(void*) at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/cores/esp8266/abi.cpp line 54
0x402015e2: delay at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_wiring.c line 54
0x40204c14: BearSSL::WiFiClientSecure::~WiFiClientSecure() at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 123
0x402028db: loop() at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/libraries/ESP8266HTTPClient/examples/StreamHttpsClient/StreamHttpsClient.ino line 114
0x40208104: esp_yield() at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_main.cpp line 91
0x40202f90: ESP8266WiFiMulti::addAP(char const*, char const*) at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiMulti.cpp line 39
0x40208190: loop_wrapper() at /home/jeroen/Downloads/arduino-PR-beta1.9-BUILD-69/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_main.cpp line 125

[earlephilhower]

The crash is actually in HTTPClient.cpp:

Exception 9: LoadStoreAlignmentCause: Load or store to an unaligned address
PC: 0x40206c3d: HTTPClient::~HTTPClient() at /home/earle/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp line 126
EXCVADDR: 0x00000131

Decoding stack results
0x40208564: std::__shared_count(__gnu_cxx::_Lock_policy)0>::~__shared_count() at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/xtensa-lx106-elf/include/c++/4.8.2/bits/shared_ptr_base.h line 547
0x402049dc: std::__shared_ptr ::operator=(std::__shared_ptr &&) at /home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/xtensa-lx106-elf/include/c++/4.8.2/bits/shared_ptr_base.h line 862
0x4010048c: _umm_free at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/umm_malloc/umm_malloc.c line 1299
0x4010095c: free at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/umm_malloc/umm_malloc.c line 1759
0x40208014: operator delete(void*) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/abi.cpp line 54
0x4020153e: delay at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_wiring.c line 54
0x40204c38: BearSSL::WiFiClientSecure::~WiFiClientSecure() at /home/earle/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 123
0x402028db: loop() at /tmp/arduino_modified_sketch_483132/StreamHttpsClient.ino line 114
0x40208230: esp_yield() at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_main.cpp line 91
0x40202f90: ESP8266WiFiMulti::addAP(char const*, char const*) at /home/earle/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiMulti.cpp line 39
0x402082bc: loop_wrapper() at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_main.cpp line 125

Got to get to work now, but the member _client in HTTPClient class is invalid and causing the boom. Need to trace where it's set/unset.

[earlephilhower]

Problem by inspection is the example code is that it delete client; .... (implicit delete HTTPClient at end of loop)

You've destroyed the WiFiClientSecure, but HTTPClient is using it in its destructor, which is a no-no. Need to rethink the HTTPClient destructor to not touch _client or something equivalent.

[earlephilhower]

The best solution to me looks like just clearing out _client and _tcpdeprecated in the HTTPClient library when HTTP.end() is called. No need for them to live past the .end() call.

[Jeroen88]

@earlephilhower just created PR #5220 to fix the issue. Thanks for your help!

[earlephilhower]

Reopening to track. Will close when the PR is approved and merged.

[earlephilhower]

Fixed w/#5220

[Jeroen88]

@earlephilhower, the fix is not handling a Connection: Keep-Alive correctly. The Object should survive an end() to be reused for a next connection. Because of the fix the connection is always closed now. Keeping it open is really simple, but at destruction of the ESP8266HTTPClient the connection should be closed. This is what the object did before the fixes. But, this leads to the same issue: the destructor ~WiFiClientSecure is called before the ~ESP8266HTTPClient destructor, making it impossible to close the connection.
I think this should be solved but I do not know how. Any suggestions?