ESP8266 AWS web socket connection fails because of BearSSL

[WeaVaer]

@odelot
I have been using this library for over a year now to connect ESP8266 to AWS IOT over websockets without any problem what so ever. Works great, thanks for the work.
My ESP8266 core was 2.4.2

After upgrading ESP core to 2.6.3, my good old sketch can not connect to AWS anymore.
I use MQTT_MAX_PACKET_SIZE 1500
I have experimented with different lwIP and SSL settings on Arduino IDE but I admittedly don't know how to use those options very well (not much documentation or info on which lwIP to use for what etc.) but the result did not change.

I am sure this happens because of the depreciation of axTL and moving to BearSSL on 'WifiSecureClient' class on the new ESP8266 core
Here is a piece of debug trace output with some eye-opening info embedded
I dont know how to supply the missing 'authentication method' to BearrSLL and need help at this stage.
Thanks

////////////////////////////////////////////////////////////////////////////////

[WS-Client] connect wss...
[WS-Client] creating WiFiClientSecure
[WS-Client] _client connecting with host =>
**************.iot.us-east-1.amazonaws.com
[hostByName] request IP for: **************.iot.us-east-1.amazonaws.com
[hostByName] Host: ********.iot.us-east-1.amazonaws.com IP: ...
:ref 1
BSSL:_connectSSL: start connection
BSSL:Connection will fail, no authentication method is setup <= <= <= <= <=
:oom(16709)@abi.cpp:48
BSSL:_connectSSL: OOM error
[WS-Client] connection to **************.iot.us-east-1.amazonaws.com:443 failed
DEBUG_WEBSOCKET_MQTT {NOT connected}

[WeaVaer]

** UPDATE **
So, I managed to make it work with ESPCore 2.6.3 but with a big tweak in the core and also some in the Arduino-Web-Sockets library. The latter seems to be upgraded with a bypass tweak but still does not work.
As soon as BearSSL (which seems to be the new SSL processor replacing axTLS methodology in WifiClientSecure class) starts working, your AWS web socket connection fails even with the tweak.
(btw just changing the core from 2.4.2 to 2.6.3 cost me 40k of more sketch size and 4K of less free heap, sigh)
Anyway, I had to block all code in the core (WifiClientSecure.h) that uses this BearSSL thing which does not work for me to connect to AWS somehow and reverted WifiClientSecure class to use axTLS code (BearSSL and axTLS seem to share the same code base, i.e either one or the other) and as expected my setup started to churn along as before.
But we need to establish the proper way to establish connection to AWS with this new SSL piece that seems to be the one to be used for the foreseable future on ESP now, and ArduinoWeb-Sockets guys should do it. (They already have something in place but as I said; its just a bypass hack and at the end of the day it doesn't work with new ESP core to connect to AWS iot.
Esp core 2.6.3 works provided you force it to use axTLS.

[odelot]

Hi... now it works with the most updated code from the esp8266 sdk and the other dependency libraries.

you can also use it without a certificate changing the Arduino-Web-Socket with _client.ssl->setInsecure(); in the else around line 180 from WebSocketsClient.cpp

[WeaVaer]

@odelot
Thanks for this recommendation, it helped.
aws-mqtt-websockets version 1.3.0 is confirmed to work wonderfully on Arduino ESP8266 core 2.7.4
(with dependencies arduinoWebSockets 2.2.0 and pubsubclient 2.8)

I want to mention that when working on ESP8266, I also added
_client.ssl->setBufferSizes(512, 512);
right there, after
_client.ssl->setInsecure();
which enables me to have a connection even with free heap as low as 16k. (wow!)
Otherwise ssl connection fails with OOM error when free heap gets lower than 27-28k.

Implemented from the idea of decreasing BearSSL buffer sizes on
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/BearSSL_MaxFragmentLength/BearSSL_MaxFragmentLength.ino