No clear way to handle MQTT broker SSL certificate expiration

[benjamincburns]

It doesn't seem that there's a clear path for the MQTT SSL support to automate broker fingerprint updates prior to server cert expiry?

Ideally this could be automated to happen in some way without any downtime. For instance, it would be nice if there was some way of publishing the fingerprint for the new cert prior to the old cert expiring via some "internal" MQTT topic.

I think apart from the logic to listen on the special fingerprint update topic, this would require the ability to specify two fingerprints to validate against, with one being regarded as the old/current fingerprint, and the other being the next/future fingerprint. Then if there were some mechanism to transparently invalidate the old fingerprint and then make the next/future fingerprint become the old/current when it starts being used by the server, I think this would be done and dusted.

[khcnz]

There is a command https://github.com/arendst/Sonoff-Tasmota/wiki/Commands MQTTFingerprint.

but right now you would either have to:
a) Change your cert immediately after publishing to all devices the new thumbprint (which you can do as a group topic) OR
b) You could push a new thumb and new domain - however i think by default it reboots after changing either of these, you would need to alter the firmware to only reboot after you have completed both

These are small embedded devices with very limited ram/flash - i'd suggest both of the workarounds above would be acceptable. In what case are you expecting your cert to expire anyway? IMO these shouldn't really be connecting directly over the internet - should be connecting to a local broker (and thus you can set a long expiry on a self signed cert).

[benjamincburns]

I'd like to set up tooling to manage these devices remotely, and I think it's unreasonable to run the risk of devices being made inaccessible by risking their becoming orphaned from the MQTT broker.

I definitely like the approach of pushing a new fingerprint and new domain better than my original idea, but what recourse do I have if I've fat-fingered the domain name? More reliable yet would be for there to be the ability to specify a primary and a backup MQTT broker host/port/fingerprint/etc, and some command for switching the priority of the two.

While I appreciate the pragmatism of your suggestions around expiry times and connecting directly over the internet, I think that it would be unwise for that pragmatism to manifest as limitations in scope for a project like this. Long expiry times compromise security. Let's Encrypt explains why this is better than I ever could, and while it's true that these are low RAM/flash devices, I don't see why that should limit them from initiating connections over the open internet. "If something is painful, do more of it." Nothing will make it more robust/reliable/secure than being forced to deal with the problems foisted upon it by the open internet.

[benjamincburns]

@arendst - what's the likelihood of you accepting a PR w/ these changes?

[khcnz]

Well as an example AFAIK it is impossible to run TLS 1.2 on these devices - like literally impossible due to memory constraints. Any other versions of SSL/TLS have either known real world of theoretical attacks - hence my suggestion of a local broker used as a relay.

Having said that a way to have a secondary server makes some sense - currently multiple wifi and multiple NTP servers are all accepted.

[benjamincburns]

@khcnz on the off chance that you have the ability to accept a PR (I see your comments in the issues all over the place here), would you accept a PR for this?

And fair enough re: TLS 1.2. I'm not familiar enough with the limitations there to debate you on that, but if certificate chain validation is the blocker, I thought fingerprint validation was an acceptably secure workaround? Though I could imagine it'd be a lot more likely in the scenario I've mentioned for your devices to be hijacked by one of your own keys being compromised than it would be in the scenario with your certs backed by a proper CA chain...

[davidelang]

In many ways fingerprint validation is more secure than certificate chain validation. The weakness is exactly this topic of changing the cert. Having the ability to have two trusted certs is needed to allow you to change from one cert being in use to the next (as it is impossible to make atomic changes across systems) My personal opinion is that even if we could do certificate chain validation, sticking with fingerprint validation would be better because the space saved can be used for other things.

[khcnz]

@benjamincburns only theo can accept pull requests - but I'd wager on his behalf that he would accept one adding a secondary mqtt broker address / fingerprint as long as existing comamnds are backwards compatible and upgrade happens automatically etc. Agreed with @davidelang fingerprint is probably all you want... but as mentioned you don't ever want to miss updating your certs and let them expire - as you have no 3rd party that can vouch you are you :) (both a good and bad thing). If that ever happened you would have to flash them (or setup a custom NTP server to "roll back time")

[davidelang]

actually, if you are checking the fingerprint, an expired cert really doesn't matter much. You have told the system to accept this specific cert, period. now, you may have trouble getting your server to startup with the expired cert, but that's workable by changing the system date if you have to. kk

[benjamincburns]

@davidelang & @khcnz thanks for your comments, and I agree that fingerprint validation seems sensible for devices like this.

Provided I can my devices connecting to my broker via SSL again, I'll likely send a PR tomorrow.

[arendst]

@benjamincburns as long as your PR is short on code and memory I might be able to implement it. Problem is TLS is using too much code already to use OTA.

I am currently focusing on implementing Arduino-ESP8266 version > 2.3.0 which uses another 25k of code space and 4k of RAM but solves many problems that users currently experience like web page hangs due to better TCP buffer handling and PWM. In the current environment it works great but leaves only 4k code space to play with while keeping OTA available and that's still without TLS...

To be honest, TLS on the sonoff is not my main concern as explained great by @khcnz above.

[benjamincburns]

It looks like the SYSCFG struct is organized such that new entries are added to the end (I assume to avoid breaking backward compatibility with settings which are already written to flash). However I also see that fields with more than one value (e.g. sta_ssid) are defined as a 2D array.

I'm going to assume that breaking backward compatibility w/ the settings in flash is worse than breaking style, and will be adding the fields for the new MQTT values at the end of the struct. If this is an incorrect assumption, please let me know.

[davidelang]

settings.ino handles upgrades from one setting format to the next. make sure that it handles the upgrade properly and then don't worry as much about chnaging the layout in flash. personally, since there are only two items, I don't think it really matters if they are two separate items or an array. I don't think that it would be reasonable to go to more than two items for this (although, if it's used for more than just mqtt, or if we have multiple mqtt servers, that would be a possibility)

[benjamincburns]

@davidelang I am adding the backup host/port/user/pw settings as well. I don't mean to argue with your advice, but since this likely won't be my last PR, can you please explain then why fields like ex_mqtt_button_retain and their neighbours are kept around if alignment of the struct from version to version isn't important? Perhaps this was just an oversight? I'd rather use the arrays than different names, as this makes toggling between the two trivial.

Edit: Now that I've written the CFG_Delta code, I see that alignment with past struct versions is required.

Aside from all that -- regarding code size, I actually got a bit distracted yesterday investigating some code size reductions to try and "pay" for this change. Strangely the PlatformIO build is around 1KiB bigger than the Arduino IDE's build. I suspect some of this is the linker script being used. However I added some optimisation args to my platform.txt which are not covered by -Os and dropped my Arduino build by another ~1KiB. Will open another issue and PR for these changes once I'm done w/ my present work.

[davidelang]

@arendst will have to answer this sort of 'why' questions.

[arendst]

The layout of flash settings is sacred! Any change will likely cause people to loose wifi or mqtt connection leading to serial re-flash.

If you need to add a setting append it please. I'll look into integration in the idle settings to keep the overall settings image as small as possible as someday it has to move to another area of flash due to wear.

I agree that arrays are more usefull and if the values are small that's fine but keep in mind that in that case you also have to write migration code for previous version settings to your newly defined array.

[wiktorschmidt]

Just wanted to share this here: https://github.com/copercini/esp8266-aws_iot

Also a related discussion:
aws/aws-iot-device-sdk-embedded-C#7

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem.