SSL support

[anteph]

Hi!
I would like to know if you plan to include SSL support in the libraries.
I've tryed to send some https requests with no sucess.

It would be cool if it the ESP could act as a secure server too.

I've checked the some examples from the sdk and they have a code to create a secure server with digital certificate.

Thanks!

[abl]

SSL has been ~broken since 0.9.3-patch1 (and you needed a new libssl, separate from patch1, from Espressif.)

It's supposedly fixed again in 1.0.1b2 (you still need a special libssl for the moment) - http://bbs.espressif.com/viewtopic.php?f=5&t=382 - I haven't had a chance to try it yet, but I've heard good reports.

There are still some features missing for practical SSL - iirc there's no way to pin a certificate or do trust validation, it'll silently accept self-signed certs - but it's still a huge positive step.

[Links2004]

i have add the latest sdk + ssl patches
see here:
Links2004@af6c400
pull request is waiting for merged (#93)

[igrr]

Merged that, thanks.
Just one (obvious) thing to note: adding SSL libraries (axTLS) is one thing, but to make them useful we also need to add SSL support to WiFiClient/WiFiServer.

[vicatcu]

@igrr does this issue cover adding HTTPS support to a WebClient application for accessing secure-connection-only APIs? What's the roadmap/timeline for including this feature? Is it actually possible?

[vicatcu]

@Links2004 @igrr Do I read this thread correctly that there is support for an HTTPS WebClient built into the core now, but WiFiClient class "just" needs that core functionality integrated into it?

[abl]

@vicatcu:

does this issue cover adding HTTPS support to a WebClient application for accessing secure-connection-only APIs?

Yes - HTTPS is just HTTP over SSL sockets, so once SSL sockets are integrated with WiFiClient/WiFiServer, you should be able to talk to HTTPS servers.

What's the roadmap/timeline for including this feature?

I'm not very familiar with this project's codebase.

Is it actually possible?

Yes. Other projects (like https://github.com/tuanpmt/esp_bridge) are able to make HTTPS requests from an ESP8266. The primary limiting factor is certificate size and protocol support (TLS v1.2 is not yet supported and is mandatory for some servers) - and that the Espressif SDK doesn't always have working SSL built in to it (it works now!)

Do I read this thread correctly that there is support for an HTTPS WebClient built into the core now, but WiFiClient class "just" needs that core functionality integrated into it?

It's now possible to write a WiFiClient that opens a secure socket. The existing WebClient shouldn't need major changes.

[vicatcu]

@abl cool, I need to come up to speed on on getting a dev environment set up so I can contribute...

[glennalgie]

looking forward to SSL support as well for ESP8266. I am looking at using Golgi.io arduino stack as an option for the SSL shim as they provide other time saving and mission critical networking support functions.

[vicatcu]

@igrr @Links2004 can you give any pointers on how to use axTLS to implement HTTPS capability into WiFiClient/WiFiServer? Any updates on progress here would be appreciated. I personally would like to see the WiFiClient capability implemented as the priority. Would it be beneficial splitting this issue into two; one for WiFiClient and one for WiFiServer?

[igrr]

@vicatcu
WiFiClient needs only TLS, not HTTPS. You can use WiFiClient for things other than HTTP — SSH and SMTP to name a few. The way to do this is to create a new class, let's say WiFiSecureClient, and override read/write methods of WiFiClient to pump data through TLS engine. Some additional methods for certificate management should be added as well. I'm not sure if there is anything that needs to be done on the server side, since WiFiServer's only job is to create a WiFiClient instance for every new connection. I suppose someone more familiar with TLS can comment on this.

[vicatcu]

@igrr ah yes of course, that (that WiFiClient is not just about HTTP) makes a lot of sense.

[fmgomes]

@vicatcu , have you started to implement the TLS on the WiFiClient? I need to connect to a webserver using https, and I'm interested in this functionality, and could try to help (don't know if I am able to do it, no previous experience with HTTPS or TLS).

Best regards

Fernando

[vicatcu]

@fmgomes honestly I haven't really gotten anywhere yet - it's a pretty steep learning curve for me so I'm kind of in the same boat as you :-/. It's a bit frustrating to know it's possible for the last few months, but that I haven't had time / know-how / etc to help make it happen. @igrr spelled out what needs to happen pretty well in his last response, but I'm not sure what "certificate management" requires...

[abl]

When a client connects to a server and performs a TLS handshake (which is what turns a socket in to a secure socket, and HTTPS is just HTTP on a secure socket) the server sends over a certificate. The certificate is what identifies the server; without the certificate validation step, HTTPS prevents against eavesdropping but you could potentially be communicating with an attacker (aka a MitM attack.)

Validating a certificate:

1. Check to see if the name of the server you're trying to connect to matches the server name on the certificate.
2. Check to see if the certificate has expired.
3. Check to see if the certificate was issued by a trusted CA.
4. Check to see if the certificate has been revoked.

1 should be pretty easy to implement; the names should be exactly equal. Wildcard certificates exist but are considered harmful; not supporting them is probably fine for v1.

2 can probably be ignored for now; the vast majority of ESP8266 work will be bound to specific certificates because...

3 is the most complex part. On a normal computer, the OS (and sometimes the browser) maintains a list of trusted root certificates. For Debian, the size of this certificate package is 502kB (see https://packages.debian.org/sid/ca-certificates) - it's possible to shrink this down a bit but it's obviously a nonstarter for the average ESP8266 project. An alternative to this is simply maintaining a list of certificate fingerprints and checking that instead - creating an explicit whitelist of certificates. That's what I'd recommend - and when you're using certificate fingerprints, you can be lax about expiration and revocation and manage it yourself.

4 is a complicated topic - check out CRL and OCSP if you're curious - and way beyond what we'd want to do in an embedded system.

AxTLS probably has simple-ish calls to handle all of this; they'd need to be exposed via SecureWifiClient.

SecureWifiServer would need a new method that allows a user to set the public and private key to use.

[mtnbrit]

does thread this also cover applying TLS to MQTT eventually?

[anteph]

For what I've read, adding https support is a little bit hard. But what about simple ciphering, for direct socket communication? Something with a symmetric key, that would be a good start.

I've read in the Espressif page that ESP8266 has a built in AES engine, but they don't specify if it is hardware or software based. Has anyone tried it?

[torntrousers]

Just a comment on that list of work from @abl - to get going i'd be quite happy to not have any of that certificate validation stuff. Wont be sending my bank account details or anything top secret just need to post to a remote service that only supports https. So just binning the certificate after its read would be fine for my uses.

[drmpf]

If you just want to prevent un-authorized use rather then hid the messages, check out this page on using SipHash as a secure hash.
http://www.forward.com.au/pfod/secureChallengeResponse/index.html

[cottsak]

So what's the latest?

• Can I make a HTTPS request?
• can I use TLS 1.2?
• can I pin a certificate (avoid the CA lookups)?

[ghost]

Is it now possible to make a HTTPS request?
Or are there any libraries out there that I could use?

[igrr]

Started working on TLS support: https://github.com/igrr/axtls-8266
This is the same library that Espressif uses for their libssl, so TLS 1.0/1.1 only.

[abl]

Going that route, it seems like wolfSSL and mbed TLS (well, especially mbed TLS) might be easier to bring in, although we know axTLS works since Espressif has it working :)

They also support TLS 1.2.

[igrr]

Both wolfSSL and PolarSSL (aka mbed TLS) are GPL (not LGPL) unless you
apply for a commercial license. I would like to keep the library
LGPL-compatible.

On Wed, Sep 2, 2015, 21:24 Alexander [email protected] wrote:

Going that route, it seems like wolfSSL and mbed TLS (well, especially
mbed TLS) might be easier to bring in, although we know axTLS works since
Espressif has it working :)

They also support TLS 1.2.

—
Reply to this email directly or view it on GitHub
#43 (comment).

[madsci1016]

@igrr one of your future milestones was "release memory allocated for certificate storage after check is complete" has that been implemented yet?

I'm now running into a memory issue:

State:    receiving Certificate (11)
crypto/bigint.c:1072 realloc 1032, left 15480
crypto/bigint.c:1072 realloc 1032, left 11944
crypto/bigint.c:1072 realloc 1028, left 7584
crypto/bigint.c:1072 realloc 2056, left 5424
crypto/bigint.c:1072 realloc 1032, left 4344
crypto/bigint.c:1072 realloc 1024, left 3832
crypto/bigint.c:1072 realloc 1032, left 1048
crypto/bigint.c:1072 realloc 512 failed, left 752
Fatal exception 29(StoreProhibitedCause):

I try to malloc them memory I need after the handshake, but at that point ESP.getFreeHeap() only shows 12KB available. Before the client.conneect call it is 27KB+. I

[igrr]

It is implemented in 2.1.0. After WiFiClient::connect returns, certificate will be kept until the first call to read or write functions. This is done to allow one call verify function and check the certificate. Once you call read or write, certificate will be released.

[apigeoje]

@igrr - You're frikin awesome. I've just spent 15 minutes reading this. I haven't even tried to use it yet but I'm already grateful for all your work...

[madsci1016]

Yes, @igrr is super awesome and needs to show us a paypal donate button so we can show our appreciation.

I've added some of my own debug as well as turned on the builtin. Here's what I got:

`Free Mem before connection: 38536

ssl/tls1.c:549 malloc 6864, left 30960

please start sntp first !

State: sending Client Hello (1)

State: receiving Server Hello (2)

State: receiving Certificate (11)

crypto/bigint.c:1072 realloc 1032, left 26584

crypto/bigint.c:1072 realloc 1032, left 23048

crypto/bigint.c:1072 realloc 1028, left 18688

crypto/bigint.c:1072 realloc 2056, left 16528

crypto/bigint.c:1072 realloc 1032, left 15448

crypto/bigint.c:1072 realloc 1024, left 14936

crypto/bigint.c:1072 realloc 1032, left 12024

State: receiving Server Hello Done (14)

crypto/bigint.c:1072 realloc 1024, left 8432

State: sending Client Key Exchange (16)

State: sending Finished (16)

State: receiving Finished (16)

Free Mem after handshake: 12184

ssl/tls1.c:1422 malloc 17408, left 19280

Free Mem after print: 19648

Response: :rd 5, 330, 0

Client closing connection

Alert: close notify`

So with 38KB free originally, 30KB are consumed during the handshake and 4KB are given back right away, leaving the user with 12KB free. After the first write call, an additional 7KB are given back, bringing free mem to 19KB with the connection open.

@igrr Is this the expected behavior? Are there any other improvements you could suggest on the server end to optimize memory usage?

My application is a IoT sensor that will include a 1 or 2 second WAV file recorded by the ESP8266, That's why I'm being a stickler for memory.

[krebbi]

is there a known issue with Let's Encrypt certificates?

because I get

please start sntp first !
State: sending Client Hello (1)
Alert: handshake failure
Error: SSL error 40
Alert: unexpected message
Alert: close notify

[igrr]

@krebbi does the server allow TLS 1.1? axTLS doesn't support TLS 1.2, which may be the reason for this error.

[krebbi]

TLS 1.0, 1.1 and 1.2
https://api.12view.me

[krebbi]

or is RSA 4096 bits too big?

[madsci1016]

I believe this ESP version only supports AES 128 or 256

[krebbi]

these Cipher Suites are supported by my cert:

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) ECDH secp256r1 (eq. 3072 bits RSA) FS 256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) ECDH secp256r1 (eq. 3072 bits RSA) FS 128
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x9f) DH 2048 bits FS 256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e) DH 2048 bits FS 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) ECDH secp256r1 (eq. 3072 bits RSA) FS 256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) ECDH secp256r1 (eq. 3072 bits RSA) FS 256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x6b) DH 2048 bits FS 256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39) DH 2048 bits FS 256

[slaff]

@igrr Do you know if NGINX supports max_fragment_length negotiation ?

[electronicsguy]

@igrr Could someone please tell me if https can work on the ESP-01?

[svdgraaf]

@electronicsguy some https connections will work, some won't. It has to do with the TLS version, the used ciphers, and/or the configuration of the webserver.

[slaff]

..." Nginx hardcodes 16KB size in ngx_event_openssl, which you can change and recompile from source."

Do you know if NGINX supports max_fragment_length negotiation ?

It turns out that you can set SSL fragment size in NGINX (>= 1.5.9 ). The directive is called ssl_buffer_size. And there is no need to recompile NGINX.

[electronicsguy]

@svdgraaf @igrr Thanks for the follow-up. I am still trying to understand the details. All I want to do is send data from esp8266 directly into a google spreadsheet (without using a http to https converter service like pushingbox). So is incorporating support for TLS1.2 unachieveable with the current esp hardware? Or is it a matter of someone writing up that code, which has just not been done as yet?

[AdamMiltonBarker]

Anyone got TLS1.2 working yet and a way to add trusted certs ?

[fsommer1968]

@AdamMiltonBarker!
Have a look at WolfSSL, they support TLS V1.2 but I´m not sure whether the library works with ESP. I want to use TLS 1.2 with Bluemix IoT but I have currently not the time to test. Maybe you can try?

[marvinroger]

wolfSSL has already been discussed, and the problem is it is GPL and not LGPL. I am not sure I understand the difference, to be honest.

[fsommer1968]

For my project it would´nt be important whether GPL or LGPL.

[AdamMiltonBarker]

I agree I think security is more important that the license.

[DaKaZ]

@igrr - I too greatly appreciate all you do for the esp8266 community. Having been developing on this hardware for the last year I could not have done it without you!

I am having trouble with the WiFiClientSecure implementation though. I basically have the exact same code for HTTP as HTTPS but I never get the response back from the server with SSL (unless I remove connection: close from the http headers and wait for a timeout). I have verified that the server is indeed sending a response back (wireshark). The code is here: https://gist.github.com/DaKaZ/5044f3168412ce0b3f5bf1d46130765e

From my debug output, I see the Alert: close notify before I even call sslClient.connected() or sslClient.available() so I fear I am missing something. Any ideas?? Here is a little more from my client debug:

HTTP: connect
please start sntp first !
State:  sending Client Hello (1)
State:  receiving Server Hello (2)
State:  receiving Certificate (11)
State:  receiving Server Hello Done (14)
State:  sending Client Key Exchange (16)
State:  sending Finished (16)
State:  receiving Finished (16)
Verifiying SSL certificate
SSL certificate matches
HTTP: connected
REQUEST: 

SSL Print: POST
SSL Print:  
SSL Print: /api/v1/plugs
SSL Print:  HTTP/1.1

SSL Print: Accept-Encoding: application/json
SSL Print: 

SSL Print: Timestamp: 1462488709
SSL Print: 

SSL Print: Signature: <removed for online post>
SSL Print: 

SSL Print: Host: 
SSL Print: <removed for online post>
SSL Print: 

SSL Print: Connection: close

SSL Print: Content-Length: 31

SSL Print: Content-Type: 
SSL Print: application/json
SSL Print: 

SSL Print: 

SSL Print: { "mac" : "<removed for online post>" }
SSL Print: 

SSL Print: 
Alert: close notify

END REQUEST
HTTP: call readResponse
HTTP: NON-NULL RESPONSE POINTER: 
HTTP: RESPONSE: 
HTTP: Connect: 0 Available: 0
HTTPS client closed 
HTTP: return readResponse3
HTTP: return readResponse
HTTP: stop client
HTTP: client stopped
Got response: 
pm open,type:2 0

[chaeplin]

@DaKaZ I get body of response using @igrr's example
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino#L61-L84

[DaKaZ]

Very interesting... thanks @chaeplin - So I changed the multiple write(...); lines into one string just called write(newString.c_str()); once and now I get the response. Not sure why but its working thats to your reference!!

[DaKaZ]

If anyone comes across this thread - here is a working version of the RestClient with SSL support: https://github.com/DaKaZ/esp8266-restclient

Enjoy

[Potato-Matic]

If there's an issue storing certificates in RAM, is it possible you could stream them into the file system and walk through them there? (Note: I've also seen a hack to upgrade the flash to 4M or bigger, so you might be able to store a fair bit)

[AdamMiltonBarker]

What the plans for secure server? I feel this is a major let down, if we can provide secure endpoints to customers on the ESP I think it would be a massive improvement.

[igrr]

@AdamMiltonBarker see #1740