Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python OTA updater script: Wrong datatype for firmware buffer #397

Closed
euphi opened this issue Sep 8, 2017 · 1 comment
Closed

Python OTA updater script: Wrong datatype for firmware buffer #397

euphi opened this issue Sep 8, 2017 · 1 comment

Comments

@euphi
Copy link
Member

euphi commented Sep 8, 2017

The new OTA updater script at https://github.com/marvinroger/homie-esp8266/tree/develop/scripts/ota_updater does not work for me, because the datatype of the firmware buffer is 'str', which results in an error in paho-mqtt library, because it trys to convert it to unicode ?!

I dont really understand why this happens, especially because the firmware file is opened in 'rb' mode, which should results in reading the file as binary_array(?!)

However, the type of firmware is str, see error log (I added a line to print firmware.__class__ to see the datatype):

Connecting to mqtt broker cubietruck on port 1883
Connected with result code 0
Waiting for device info...
Publishing new firmware with checksum b09270aaa47fc5461ae9e16504a3942f
homie/balkctrl2/$implementation/ota/firmware/b09270aaa47fc5461ae9e16504a3942f
<type 'str'>
Traceback (most recent call last):
  File "./ota_updater.py", line 154, in <module>
    args.broker_password, args.base_topic, args.device_id, firmware)
  File "./ota_updater.py", line 110, in main
    client.loop_forever()
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1378, in loop_forever
    rc = self.loop(timeout, max_packets)
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 897, in loop
    rc = self.loop_read(max_packets)
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1177, in loop_read
    rc = self._packet_read()
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1766, in _packet_read
    rc = self._packet_handle()
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2240, in _packet_handle
    return self._handle_publish()
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2415, in _handle_publish
    self._handle_on_message(message)
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2575, in _handle_on_message
    self.on_message(self, self._userdata, message)
  File "./ota_updater.py", line 85, in on_message
    client.publish(topic, userdata['firmware'])
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 980, in publish
    rc = self._send_publish(local_mid, topic, local_payload, qos, retain, False, info)
  File "/home/ian/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1989, in _send_publish
    upayload = payload.encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 0: ordinal not in range(128)

As a workaround, I convert the buffer to bytearray, which works fine for me:

    # read the contents of firmware into buffer
    fw_buffer = args.firmware.read()
    args.firmware.close()
    firmware = bytearray()
    firmware.extend(fw_buffer)
LiyouZhou added a commit to LiyouZhou/homie-esp8266 that referenced this issue Sep 24, 2017
LiyouZhou added a commit to LiyouZhou/homie-esp8266 that referenced this issue Sep 24, 2017
@LiyouZhou
Copy link
Contributor

@euphi #403 should fix your issue.

jesserockz pushed a commit to jesserockz/homie-esp8266 that referenced this issue Mar 12, 2018
euphi added a commit to euphi/homie-esp8266 that referenced this issue Mar 31, 2018
* 🔥 Remove hardcoded keepalive - fix homieiot#301 (homieiot#314)

Remove hardcoded KeepAlive for MQTT connection.
Default value of 15sec is already present in AsyncMQTT library

* 🐎 Improve uptime accuracy (homieiot#315)

improve uptime accuracy by storing milliseconds and only rounding when
publishing value
drawback: less time before rollover, but still long enough

* 🐛 Fix truncated IP (homieiot#318)

published IP is truncated.

* 🎨 Fix warning with parenthesis

* 🐛 Fix pio library.json bad dependency name

* ✨ Add support for static IP and BSSID/MAC, channel of AP (homieiot#327)

* Support for static IP and BSSID/MAC, Channel of AP

- added parameters to config.json which allow to define static ip, mask,
gateway, dns
- added parameters to config.json which allow to define BSSID and
channel of AP

To run device with defined static IP you have to define ip, mask and
gateway together.

To point device to connect to specific BSSID and channel you haveto
define bssid and channel together.

```
{

	"name": "The kitchen light",
	"device_id": "kitchen-light",
	"wifi": {
		"ssid": "Network_1",
		"password": "I'm a Wi-Fi password!",
		"bssid":
		"DE:AD:BE:EF:BA:BE",
		"channel": 1,
		"ip": "192.168.1.5",
		"mask": "255.255.255.0",
		"gw": "192.168.1.1"
	},
	"mqtt": {
		"host": "192.168.1.10",
		"port": 1883,
		"base_topic": "devices/",
		"auth": true,
		"username": "user",
		"password": "pass" i
	},
	"ota": {
		"enabled": true
	},
	"settings": {
		"param1": 55,
		"param2": "abcdefghijklm",
		"param3": true,
		"param4": false,
		"param5": 2147483647,
		"param6": -2147483647,
		"param7": 55,
		"param8": "abcdefghijklm",
		"param9": true,
		"param10": false
	}
}
```

* Addjustments for travis

* Another addjustments for travis

* Fix problem with prepareToSleep

* 🐎 Pass all callbacks by reference
Still store it by value

* 🐎 Make isActive const

* 🎨 Add custom settings value to initial log

* ✨ Abort if default setting value does not pass validator function
Fix homieiot#324

* 📝 Clarify ISSUE_TEMPLATE docs location for homieiot#331

* 📝 Implement new versioned Git docs (homieiot#341)

* ✨ Add versionned in-repo docs

* 🐛 Attempt to fix encrypted key

* 🐛 Fix permission issue on python exec

* 🐛 Add mission import

* 🐛 Fix relative path

* 🐛 Add missing commit

* ✨ Add index

* 🎨 Change index design

* 📝 Update all URLs to new docs

* 🎨 Fix broken doc link in README

* 📝 Add edit link to docs

* 📝 Move firmware_parser.py to scripts folder

* 🐛 Fix out of limits abort message not showing

* 👕 Be less strict on whitespace in comments

* 📝 Add dummy OTA updater script (TODO)

* 🎨 🐎 Use CircleCi instead of Travis CI (homieiot#348)

* ✨ Add Circle CI build

* 🐛 Add working_directory

* 🐛 Attempt to fix perm issue

* ✨ Generate docs from CircleCI

* 🐛 Fix path

* 🐛 Fix chmod permission

* 🐛 Try to run sudo

* 🐛 Attempt to use remote docker

* 🐛 Change bad directories

* 🐛 Remove dependency on Docker

* 🐛 Fix perm problem

* 🐛 Chamge tmp folder

* 🐎 Use CircleCI instead of Travis CI

* 🎨 Add SonoffDualShutters example

* 🐛 Install current lib to platformio

* 🐛 Actually ignore gh-pages in CI

* 🐛 Add missing SonoffDual dep

* 🐛 Fix ArduinoJSON 5.11.0 (homieiot#363)

* ⬆️ Update pio dependencies

* 🎨 Cleanup code a bit

* 🐛 Make sure every announcements packet are sent - closes homieiot#345

* 🎨 Refactor code and implement new OTA system
Closes homieiot#346

* 👕 Fix lint

* ✅ Use new workflow feature from Circle

* ✅ Ignore gh-pages at workflow level

* ✨ Add OTA_PROGRESS event

* 📝 Update docs for new OTA system

* ⬆️ Upgrade AsyncMqttClient dependency to 0.8.0

* 🐛 IHomieSetting::settings first, HomieNode-settings second! (homieiot#335)

*  IHomieSetting::settings first, HomieNode-settings second!

Set initialization priority of IHomieSetting::settings to value of highest allowed priority (101).

This allows other static variables to be of type HomieSetting<T>.

See https://github.com/euphi/HomieNodeCollection/blob/master/src/RGBWNode.cpp for example.

Note: As shown in the example it makes sense to have a static HomieSetting member, if you have a class that may be instantiated multiple times. (e.g. two LED Strips connected to the same ESP8266, as shown in https://github.com/euphi/ESP-LEDCtrl).

* Removed extra whitespaces

* Removed another whitespace

* 🐛 Fix not returning a value in setConfigurationApPassword (homieiot#378)

`HomieClass& HomieClass::setConfigurationApPassword(const char* password)` did not returned a reference to the Homie instance. This PR fixes this.

* 🐛 Fix crash when starting up without any defined node (homieiot#379)

* Fix crash when starting up without any defined node

Skip node publication if HomieNode::nodes.size() == 0

* Update BootNormal.cpp

* 🐛 Fix topic check for OTA upload (homieiot#375)

* Change topic check for OTA upload

* Change firmware topics to remove the 's'

* 🐛 Rename last OTA topic instance

* ✨ Add OTA updater script (homieiot#384)

* 🐍Add python ota updater script

* 💼 Update documentation of ota update script

* 😑 Add comments to ota updater script

* 🔮 Use 127.0.0.1:1883 as default broker setting

For the ota updater script

* 📝 Add details on how to interact with range property (homieiot#393)

Add hint to help people figure out how to interact with
range properties. Especially document the `_` separator.

* 📝 Add warning to input-handlers.md about concurrency (homieiot#400)

* 🐛 Interpret firmware file as an bytearray (homieiot#403)

Fix homieiot#397

* 📝 Update input-handlers.md (homieiot#401)

* Update input-handlers.md

Sorry, it seems that I used the `!!! warning` block in a wrong way.

* Update input-handlers.md

* Update input-handlers.md

* ✨ Use AsyncWebServer + Refactoring (homieiot#425)

* Initial AsyncWebServer

* Fixed Proccessing Body Requests (JSON)

* Doc Fixes + typo

* Added Missing Method in Timer.cpp

* Lots of Refactoring + Moved Reset Button to its own helper class for boots to use

* 🎨 Update BootNormal.cpp (homieiot#426)

Solve "else" errors from https://circleci.com/gh/marvinroger/homie-esp8266/136#tests/containers/0

* ⬆️ Update dependency to "ESP Async WebServer" (homieiot#434)

* 📝 Add instruction for @platformio (homieiot#435)

* Instruction for @platformio

* Explain how to used tagged version with @platformio

* Explain how to use tagged version with @platformio

* 🐛 Use v2.0.0-beta.2 as a working tagged example for @platformio (homieiot#437)

* 🎨 Simplify CI with @platformio (homieiot#438)

* 🐛 Install library via @platformio with all dependencies (homieiot#439)

* 💚 CI: Install staging version of Arduino Core for ESP8266 & @platformio (homieiot#440)

* 🐛 Pin Shutters dep version

* 🔥 Don't fail on CI docs step when testing a fork

* 🎨 More Refactoring + Deep Sleep + Prevent WiFi Reconnect when reboot (homieiot#432)

* Initial AsyncWebServer

* Fixed Proccessing Body Requests (JSON)

* Doc Fixes + typo

* Added Missing Method in Timer.cpp

* Lots of Refactoring + Moved Reset Button to its own helper class for boots to use

* Lots of Refactoring.

* Refactored ResetHandler + More Refactoring + Testing

* Added deep sleep function + Prevent Wifi reconnect before reboot (homieiot#380)[homieiot#380]

* Small typo fixes

* Small Rearange of code in BootConfig + Minor Refactor of function names

* Minior Commit to Triger a Github Action

* circleci build fix

* 🎨 First pass to fix linting

* 🎨 Second pass of lint

* 🎨 Final lint fix

* 📝 Update docs deps

* 📝 🎨 Update docs manifest

* 📝 🎨 Adjust HTTP JSON API doc

* 📝 Add configurators on website

* 📝 Update links to configurator

* Fix links under Features (homieiot#452)

* Set Device Stats Interval (homieiot#451) (homieiot#455)

* Add last step to uibundle README (homieiot#460)

* Add last step to uibundle README

* Added Arduino Support for doc

* Update updater script addressing quirks (homieiot#461)

* Update API for /wifi/connect from GET to PUT (homieiot#468)

Docs show `/wifi/connect` as `GET` when it should actually be `PUT`

* Proposal to optionally run HomieNode::loop() also in disconnected state

* Show Homie version

* Fix homieiot#446 homieiot#477 (homieiot#501)

* Fix homieiot#446 CORS Issue

* Fix for homieiot#477

* Fix Lint

* Fix Safari not displaying the config bundle HTML page (Fix homieiot#476) (homieiot#502)

* Fix Safari not displaying the config bundle HTML page

Safari cannot deal with gzip files that have a "*.gz" file extension.
Simply faking the filename solves the problem though.

* Update Readme Homie Version

* Fix Warnings (homieiot#503)

* Update Readme - Homie Convention

* Update to Homie Convention v2.0.1 (homieiot#507)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants