Added a MQTT pub/sub example of using AWS IoT

[rongsaws]

This example demonstrates ESP8266 is capable of working with the AWS IoT service through TLS1.2 based MQTT connection.

Readme file included.

Some memory usage stats of the example:

[projectgus]

Very exciting! Can't wait to test this.

Do you think we could wrap some of this functionality into a component/library? I know people have tried to port the actual AWS IoT SDK, but this looks like it might be an easier approach as the AWS SDK itself makes a lot of assumptions about the system you're using.

[rongsaws]

We could definitely wrap portion of the code into a library, however the MQTT task and SSL connection are not abstracted enough to be consumed as a library. Given this example, I think it's just matter of time before someone eventually ports the AWS IoT device SDK to this project, which may make more sense to be consumed as a library.

I guess a lot of applications probably just want basic pub/sub function like this sample does so they can save more memory for other uses. I could potentially port the AWS IoT device SDK over if it turns out more people want to use it that way.

[rongsaws]

I added two more commits, specifically around handling over-sized messages received - one is to fix the issue reported in #152 (after the fix, connection will still get reset for sanity purpose but the system won't crash) and the other one is to reset SSL upon read/write errors, such as over-sized record received.

[TheSkorm]

This example is perfect for our use case. Adding basic support for IoT shadows would be nice but as stated, most people just need pub/sub.

Only suggestion is maybe add some comments around the NewMQTTClient on what the values are how you would change them for larger messages.

NewMQTTClient(&client, &network, 5000, mqtt_buf, 100, mqtt_readbuf, 100);

[kanflo]

Does this fix come from upstream or should it be sent upstream?

[rongsaws]

Similar issue was also reported in the upstream Paho client library (eclipse-paho/paho.mqtt.embedded-c#40), but it doesn't seem to be a popular project or getting much attentions. Also, the MQTT client library included here was forked, therefore merging is also going to be a problem.

[kanflo]

Well that's a pity. Thanks for the info.

[toashwin]

This is awesome.. I was able to test it out today, and its all working.
Now, I think I will need the thing-shadows capability.. In my project I will need to check if my device is ON/OFF from a website. Looks like this feature is still missing? Is there something I can do myself? Any recommendations?

(PS: But very glad I got this far. I almost gave up 2months ago. Then I had to go on a vacation. By the time I am back its all implemented. Super excited.)

[kanflo]

Thanks @rongsaws and @toashwin. I will merge this one, sorry it took so long.

[rongsaws]

@toashwin, for simple shadow interactions from the device, you could publish the below shadow update message to topic $aws/things/thingName/shadow/update:

{
    "state": {
        "reported": {
            "switch": "ON",
        }
    }
}

More details can be found here. Please let me know whether this works for you. Thanks.

[toashwin]

@rongsaws,
Cool. Thank you. That seem to be working when tested with MQTT-Client from the AWS-IOT website. I will test with ESP8266 tonight.

I got few questions:
This is what I did:
I created a thing. It had following values:
REST API Endpoint: https://.iot.us-east-1.amazonaws.com/things/mybutton/shadow
MQTT topic: "$aws/things/mybutton/shadow/update"

Question 1: I wonder why $aws in the topic name? Is it like a variable? In the c-code, do I just replace the value in ( #define MQTT_PUB_TOPIC "esp8266/status" ) with "$aws/things/mybutton/shadow/update" along with $sign?

Question 2: For the client endpoint, would it be:
const char *client_endpoint ="https://.iot.us-east-1.amazonaws.com/things/mybutton/shadow ";
OR
const char *client_endpoint ="https://.iot.us-east-1.amazonaws.com/ ";

Question 3: Not related to ESP8266. From the AWS IOT user guide, I see that AWS IOT Button has ARN:
arn:aws:iot:us-east-1:123456789012:topic/iotbutton/G030JF055364XVRB

For me, ARN has, "things" or "policy" in the path. but I dont see "topic" as in the guide.
And how can I create a device ID into the path? I can do something like:
iotbutton_G030JF055364XVRB
but not
iotbutton/G030JF055364XVRB

Thanks a lot.

[rongsaws]

@toashwin, I answered your questions below. If you have any other AWS IoT specific questions, I encourage you to post them on the AWS IoT forum here, so we can stay focused on ESP8266 here. Thanks.

Question 1: I wonder why $aws in the topic name? Is it like a variable? In the c-code, do I just replace the value in ( #define MQTT_PUB_TOPIC "esp8266/status" ) with "$aws/things/mybutton/shadow/update" along with $sign?

Topics beginning with '$' are called reserved topics, and no, it's not a variable. You can subscribe to these reserved shadow topics just as you do with other regular topics. See here for more details.

Question 2: For the client endpoint, would it be:
const char *client_endpoint ="https://.iot.us-east-1.amazonaws.com/things/mybutton/shadow ";
OR
const char *client_endpoint ="https://.iot.us-east-1.amazonaws.com/ ";

The client_endpoint variable defined in client_config.c should just be the host name, without https scheme or path.

Question 3: Not related to ESP8266. From the AWS IOT user guide, I see that AWS IOT Button has ARN:
arn:aws:iot:us-east-1:123456789012:topic/iotbutton/G030JF055364XVRB

Different resource types have different ARNs, such as topic ARN, client ID ARN, and thing ARN, see here for more details. Don't confuse ARN with REST API endpoint path as they are different things. When you access shadow using REST APIs, you use the provided API endpoint, and in this case, no MQTT topics are involved; and when you use MQTT to access the shadows, you publish updates to the reserved shadow topics (as the aws_iot example does) to archive the same purpose as you do with REST APIs.

[toashwin]

@rongsaws - Thanks for your suggestions.. Made very good progress.

How can I retrieve the latest stored data? I want to access that in the device, at start up.

From the code, I see you are only doing subscribe/publish. Do we need something like a "get"?

Thanks
Ashwin

[toashwin]

@rongsaws Thank you so much! This was the last thing missing in my project. I was almost about to move away from ESP8266.

I got the whole project working now..
My device is connecting to AWS IOT over MQTT. I got it hooked up to Alexa as well. And even thats working perfectly.

Next I am going to approach Alexa Fund for some financial help to take this project to the next step. So I can do field testing, get some real customer feedback! Hopefully that goes through. :(

Thank you.

PS: Please answer my previous question. I have a workaround for that. But still good to know.

[rongsaws]

@toashwin,

How can I retrieve the latest stored data? I want to access that in the device, at start up.
From the code, I see you are only doing subscribe/publish. Do we need something like a "get"?

Yes, you can make a Get request by sending an empty message to topic _$aws/things/ThingName/shadow/update, the shadow content will be published to topic **$aws/things/ThingName/shadow/get/accepted**_.

[toashwin]

Hi
Do you have any plans to support AWS IOT for ESP8266 on Arduino SDK?

Thanks
Ashwin

[only1chi]

I'm struggling to get AWS IOT working for my application.
I copied the AWT_IOT files to my project rather than build it directly in examples/aws_iot

I get compile erros such as:
conflicting types for 'sdk_wifi_station_get_connect_status'
/home/chiz/share/esp-open-rtos/include/espressif/esp_sta.h:73:9: note: previous declaration of 'sdk_wifi_station_get_connect_status' was here
uint8_t sdk_wifi_station_get_connect_status(void);

Apparently sdk_wifi_station_get_connect_status(void) is already defined in MQTTClient.h as follows:

void sdk_wifi_station_get_connect_status(mqtt_client_t*, mqtt_network_t*, unsigned int, unsigned char*, size_t, unsigned char*, size_t);

How does this get resolved?

[vlad-ivanov-name]

@only1chi there's no such definition in MQTTClient.h? https://github.com/SuperHouse/esp-open-rtos/blob/master/extras/paho_mqtt_c/MQTTClient.h

[only1chi]

@resetnow I needed to update my local repository for esp-open-rtos.

Thanks.

[tbinswirls]

Hello @toashwin

I got the whole project working now..
My device is connecting to AWS IOT over MQTT. I got it hooked up to Alexa as well. And even thats working perfectly.

Glad to read about your progress. I also trying to subsribe/publish to AWS IOT thing topics, however I am having some issues. I have posted about this in a separate topic as well (#329), below is an excert from that:

I have gotten far enough to be able to publish messages to the shadow. However, there seems to be an issue when ESP is subscribed to the 'thing/shadow/update/delta' topic. The error happens when AWS detects a difference between 'desired' and 'reported' states of the thing shadow and it sends a message to the '/delta' topic. The error message is:
'Connection dropped, request restart'

I am sure that the message being sent to the '/delta' topic reaches ESP because this error happens only when there is a difference between 'desired' and 'reported' states.

I have been successful in subscribing and receiving messages from 'non-thing' topics, but got stuck with receiving the messages from any of the 'thing shadow topics'.

Could it be the size of the payload? The messages received by ESP from 'non-thing' topics were just 'on' and 'off'.