Feature/http update wi fi client parameter

[Jeroen88]

Pass Client parameter to httpUpdate to use any BearSSL security method for a firmware update (with example). This PR replaces #4832.

This is a combination with PR #4979, because The ESP8266HTTPUpdate class uses the ESP8266HTTPClient class from this PR to pass the BearSLL WiFiClient.

Added two new update() and one updateSpiffs() function that uses the ESP8266HTTPClient adapted by me in PR #4825. With these adaptations, it is possible to do an encrypted and authenticated server firmware or SPIFFS update, using any of the supported security options of the WiFiClientSecureBearSSL library. httpUpdateSecure.ino is a complete example.

To try this, you need either a sketch with enough free memory to allocate the full TLS/SSL-buffer (16k), or a, currently not very widely supported Maximum Fragment Length Negotiation enabled, server (or do the same trick as I do, using a nodeJS server and set maxSendFragmentLength) that sends the correct response headers and serves the binary. You need a certificate store in SPIFFs that includes your root CA, or a self signed certificate, or a certificate fingerprint. In the latter two cases you need to edit the example to use allowSelfSignedCerts() or setFingerprint(). It is also possible to use encryption without server authentication. In this case use setInsecure(). Of course any of the other supported WiFiClientSecureBearSSL functions may be used as well.

[d-a-v]

Thanks very much for your time with this confirmation !
We can thank lwIP devs for SACK :)
I indeed should have told you to restart the IDE.
(about upstream remote name, I assumed that origin was the default but I may be wrong?)

[Jeroen88]

@earlephilhower

LGTM, but maybe you could update the example code to use this new feature? W/o an example, it's not exceptionally clear from the header what's going on.

Sure! All 3 examples updated in this PR with the usage of setLedPin() and a comment on how to use it

[Jeroen88]

@d-a-v You're welcome, happy to help! I am glad that the lwIP devs SACKed us ;)
Maybe origin is the default, but I am a rather newbie with github and I do not comprehend it fully yet. My 'own' fork is origin and upstream is the original repo I cloned from. Before doing any changes I pull from upstream and push to origin to sync them

[earlephilhower]

You had a trailing space in a comment, causing the CI style_check to fail. Fixed it for you.

[earlephilhower]

I just tried this code with the original example code (simulating someone running their existing app) and it looks like the _tcpDeprecated is not quite there yet:

[HTTP] begin...
[HTTP-Client][end] tcp is closed
[HTTP-Client][begin] url: http://192.168.1.12/test.html
[HTTP-Client][begin] host: 192.168.1.12 port: 80 url: /test.html
[HTTP] GET...
[HTTP-Client] connect: HTTPClient::begin was not called or returned error
[HTTP-Client][returnError] error(-1): connection refused
[HTTP] GET... failed, error: connection refused
[HTTP-Client][end] tcp is closed

[earlephilhower]

Actually, I see the old sample code was silly and required a local 192.168.x.x server.

Changing the server to one that exists doesn't change the results, unfortunately:

[HTTP] begin...
[HTTP-Client][end] tcp is closed
[HTTP-Client][begin] url: http://jigsaw.w3.org/HTTP/connection.html
[HTTP-Client][begin] host: jigsaw.w3.org port: 80 url: /HTTP/connection.html
[HTTP] GET...
[HTTP-Client] connect: HTTPClient::begin was not called or returned error
[HTTP-Client][returnError] error(-1): connection refused
[HTTP] GET... failed, error: connection refused
[HTTP-Client][end] tcp is closed

[earlephilhower]

Original code output (StreamHTTPClient with the URL moved to the jigsaw.w3.org server)

SDK:2.2.1(cfd48f3)/Core:2.4.2-70-g9bc8ea1/lwIP:2.0.3(STABLE-2_0_3_RELEASE/glue:arduino-2.4.1-13-g163bb82)/BearSSL:f55a6ad



[SETUP] WAIT 4...
wifi evt: 2
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 3
cnt 

connected with NOBABIES, channel 10
dhcp client start...
wifi evt: 0
[SETUP] WAIT 3...
ip:192.168.1.138,mask:255.255.255.0,gw:192.168.1.1
wifi evt: 3
[SETUP] WAIT 2...
[SETUP] WAIT 1...
[WIFI][APlistAdd] add SSID: NOBABIES
[HTTP] begin...
[HTTP-Client][begin] url: http://jigsaw.w3.org/HTTP/connection.html
[HTTP-Client][begin] host: jigsaw.w3.org port: 80 url: /HTTP/connection.html
[HTTP] GET...
[hostByName] request IP for: jigsaw.w3.org
[hostByName] Host: jigsaw.w3.org IP: 128.30.52.21
:ref 1
[HTTP-Client] connected to jigsaw.w3.org:80
[HTTP-Client] sending request header
-----
GET /HTTP/connection.html HTTP/1.1
Host: jigsaw.w3.org
User-Agent: ESP8266HTTPClient
Connection: close
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0

-----
:wr 160 0
:wrc 160 160 0
:ack 160
:rn 536
:rch 536, 284
:rcl
:abort
[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
[HTTP-Client][handleHeaderResponse] RX: 'Connection: extensionheader,close'
[HTTP-Client][handleHeaderResponse] RX: 'Date: Sat, 06 Oct 2018 00:06:57 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 550'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Type: text/html'
[HTTP-Client][handleHeaderResponse] RX: 'Etag: "1giilod:q0efehi8"'
[HTTP-Client][handleHeaderResponse] RX: 'Last-Modified: Tue, 20 Jun 2000 13:33:22 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Server: Jigsaw/2.3.0-beta3'
[HTTP-Client][handleHeaderResponse] RX: 'ExtensionHeader: ExtensionValue'
[HTTP-Client][handleHeaderResponse] RX: ''
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 550
[HTTP] GET... code: 200
<HTML>
<HEAD>
  <!-- Created with AOLpress/2.0 -->
  <TITLE>Connection Header</TITLE>
</HEAD>
<BODY>
<P>
<IMG ALT="Jigsaw" BORDER="0" WIDTH="212" HEIGHT="49" SRC="/icons/jigsaw">
<H1>
  The <I>Connection</I> header
</H1>
<P>
This page will be served to you:c 1, 536, 820
 with the following headers:
<P>
<CODE>ExtensionHeader: ExtensionValue<BR>
Connection: ExtensionHeader</CODE>
<P>
If you're getting this page through a proxy, you should <I>not</I> see the
<I>ExtensionHeader</I> !
<P>
  <HR>
<BR>
<A HREF="mailto:jigsaw@w3.:c0 1, 284
org">[email protected]</A>
</BODY></HTML>

[HTTP] connection closed or file end.
[HTTP-Client][end] tcp is closed
:ur 1
:del
pm open,type:2 0

[earlephilhower]

@Jeroen88, hope you don't mind but the fix was simple so I just moved the problem NULL check to after the _client is actually set and committed. Now old samples seem to work fine with the new class.

[Jeroen88]

@earlephilhower the check for _client should indeed be after the assignment, thnx for the fix!

[Jeroen88]

@earlephilhower did you see my comment:

I still have a crash running the example streamHttpsClient.ino of this PR. I did not apply GIT head because I do not know how to do that (I am still a git newbie).
You are right that the problem is not the delete's. I am still not sure of the root cause, but: when the streamHttpsClient.ino-sketch calls delete client at the end of the loop ~WiFiClientSecure() is called before ~HTTPClient(). The ~HTTPClient() destructor calls _client->stop() with _client being a WiFiClientSecure and stop() is defined in WiFiClient.cpp and stop() calls flush() and _client->close(). However: the client context is already gone, because the ~WiFiClientSecure() destructor was already called. I think this is the problem. I do not know how it should be solved, in WiFiClientSecureBearSSL of in ESP8266HTTPClient. In case of the latter I also do not know how...

This is with streamHttpsClient.ino. Any suggestions?

[earlephilhower]

@Jeroen88, I'm going to commit this now, but if after this is committed you pull git head and still see the crash open an issue and we can look at it.

I'm running the StreamHTTPSClient (i.e. BSSL) example now and it's running just fine, so I have no idea what you're seeing, unfortunately.

[scottchiefbaker]

Now that the ESP8266 call requires a WiFiClient object for the begin() method it's no longer the same as the ESP32 class. If I'm trying to make code that compiles on both ESP8266 and ESP32 do I have to wrap every begin() call in:

#ifdef ESP32
http.begin(url);
#else
http.begin(client, url);
#endif

Or is there another easier way to make cross-platform code?

[Jeroen88]

@scottchiefbaker ESP32 HTTPClient supports the same ::begin() function as ESP8266HTTPClient as you can see here

[scottchiefbaker]

@Jeroen88 oh I didn't know the ESP32 libraries we're already update. I fixed my code to reflect the new calling parameters and now it compiles on both platforms correctly. Thank you.

[Jeroen88]

@scottchiefbaker you're welcome!