Help: Upload instructions for custom rnode firmware

[azidoazid]

Main goal

Make the rnode firmware compatible with diy hardware that contains the ebyte e22-900m30s 1w LoRa moden. In my case it's the hydra board: https://github.com/Hydra-Designs/project-hydra-meshtastic-pcb
My main idea was to bootstrap from the existing "generic esp32" device since it's technically the same thing, but different pins and different RF chipset/rxtx switching compared to the original design.

Current state of my efforts

my repo: https://github.com/azidoazid/RNode_Firmware

• Manual RX/TX switching implemented, borrowed from the sx128x driver
• Custom board definition for the hydra board
• Makefile: firmware and upload section implemented, borrowed from the "generic esp32" target, modified compile flags for different board ID and radio
• I can compile custom firmware
• upload does not work

Error messages

• make upload fails with: This device has not been provisioned yet, cannot set firmware hash
• OLED shows: Hardware Failure

Steps to reproduce

• make firmware-mt_hydra
• make upload-mt_hydra

Question

• What steps are required to go from sourcecode to working device?

I have only limited coding experience, but as far as I understand there is some post-upload process involved that generates some checksums that are stored in flash memory. I also got the same error message on the OLED while using rnodeconf to install firmware on supported hardware, but only for a short period so there must be some step that I am missing.

[cobraPA]

Hopefully Mark will drop in and give pointers to 'a better way'...
There are firmware CRC checks and signing checks done for standard installs. For the BLE test version I am working on for iPhone, you can have a look at the (hacky) overrides I am using.
https://github.com/cobraPA/RNode_Firmware_iOS/blob/kb_ble_test1/RNode_Firmware.ino

1. Line 1153 with the "if(1)" replacement of the initial check.
2. Line 1169 and 1172 for startup check overrides
3. Might need 1225 also

Depends on what is installed and working on your board. The printouts in that section might help you when you try to enable your Lora radio. One of those sections will fail if the radio init fails.

The above should let you run your 'make upload-xxxx' command. It will normally throw an error for the second step, but the two flashing phases before and after should run. The overrides in the firmware I showed will then allow the firmware to boot without failing the CRC and signing checks which you may not have in eeprom yet.

To fully provision a radio locally, you will need to sign the firmware with rnodeconf. But I think you also need to modify rnodeconf to add proper product IDs, if you have an entirely new board type.

[azidoazid]

Thanks for the quick reply! I initially forgot to set the proper SPI pins and got the error message "radio not found" on the OLED, so I am quite sure that at least my sx126x driver hacking and board definition seems to be working, at least the communication part.
I thought that I could work off the existing "generic esp32" and just add some switches that trick the whole ecosystem into thinking that it's a "generic esp32" target (which is seemingly implemented in rnodeconf) while under the hood it's running the sx126x driver with my rxtx switching hacks.
I even tried to recycle the generic esp32 board ID but no luck :(
I am going to give your overrides a try!

[cobraPA]

There is also a recent change by Mark to help with some of this, I think:
markqvist/RNode_Firmware@854b345
It is not in my tree yet, because I haven't merged it, but it could also help. I think my overrides will already override that piece though.

[markqvist]

See my full reply below, but just wanted to note here too, that I think you should definitely define your own correct product and model codes from the beginning. Just choose something that doesn't collide with anything else. You will need to add your product definition to the eeprom_product_valid function and likewise add required logic to the eeprom_model_valid function. Other than that, I think you already have the idea of it.

[azidoazid]

Gave it a try, no luck :(

[cobraPA]

What happened? Oh, the provisioning errors?
Is there documentation on the provisioning commands?
I did go through rnodeconf... you need something like these commands the first time, unless you can flash your board with a known config first:

rnodeconf --key
rnodeconf --platform ESP32 --product f0 --model ff --hwrev 1 --rom serial_port
product and model would come from those functions Mark showed
hwrev I assume is based on your product
where serial_port is a platform style port /dev/usbxx or COM3, etc

[markqvist]

The firmware won't boot without a valid EEPROM config, but since rnodeconf doesn't have any idea about your board model and product codes yet, there's no easy way to actually write the config, which is basically the gist of the problem.

The rnodeconf utility has the --rom option, which allows manual EEPROM bootstrap, but I just checked, and to my actual own surprise, I found that you can't just pass any codes to it, but it was limited to a few. Hm, I did write that code a long time ago, so there's to that.

Either way, I just pushed a new commit to rnodeconf, that will allow arbitrary EEPROM bootstrap values. You'll have to set these correctly once the firmware has been built and flashed via the makefile (when the device says "Hardware failure"). You will have to do something like:

rnodeconf --rom --product f0 --model 43 --hwrev 1

The actual product and model codes will have to match what you used in the firmware, though! I just guessed from a quick look at your repo, but you may or may not have changed them or set them to something else. For the final version, choose a product and model code that doesn't collide with any other boards already defined (so we can add a nice description and such to rnodeconf and other tools for displaying your board info).

I hope that gets your firmware running correctly :)

[cobraPA]

Although, I may have bypassed one of the startup checks to get past that also.

[liamcottle]

@cobraPA:

I had a fully running RNode with BLE and Lora, even when that was displayed.

Could you please give me any pointers on how you were using BLE?

I see a new commit about BLE support and would love to know how to try this out.

I compiled the latest RNode_Firmware commit, and flashed it to my node, even tried using the --bluetooth-on and --bluetooth-pair flags via latest commit of rnodeconf.py, but I never see any bluetooth devices show up in device settings on my iOS device or Android device that look to be the node.

I'm doing this with a Heltec v3 board btw. I understand this board is only very recently in the process of being supported, but just wondering if this already works and how to go about using it with BLE.

For extra info, this is how I flashed latest commits;

# clone latest RNode Firmware
git clone https://github.com/markqvist/RNode_Firmware
cd RNode_Firmware

# build for heltec v3
make clean
make prep-esp32 
make firmware-heltec32_v3

# flash it
arduino-cli upload -p /dev/cu.usbserial-0001 --fqbn esp32:esp32:heltec_wifi_lora_32_V3
python3 ~/Desktop/Reticulum/RNS/Utilities/rnodeconf.py /dev/cu.usbserial-0001 --firmware-hash $(./partition_hashes ./build/esp32.esp32.heltec_wifi_lora_32_V3/RNode_Firmware.ino.bin)
python ./Release/esptool/esptool.py --chip esp32-s3 --port /dev/cu.usbserial-0001 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size 4MB 0x210000 ./Release/console_image.bin

# enable bluetooth????
python3 ~/Desktop/Reticulum/RNS/Utilities/rnodeconf.py /dev/cu.usbserial-0001 --bluetooth-on
python3 ~/Desktop/Reticulum/RNS/Utilities/rnodeconf.py /dev/cu.usbserial-0001 --bluetooth-pair

[cobraPA]

Hi @liamcottle,

Well, not short answer... but here goes.

In device settings on your iOS device, you will not see BLE devices before the are connected or paired (at least custom ones)! I learned this early on and it was a lot of confusion initially. This is a difference from classic Bluetooth where you commonly pair from the device list. I would suspect Android is the same but I may be wrong there. BLE is kind of a different beast when it comes to bluetooth. It is often used for devices that are custom to the application they attach to, so the applications are expected to know how to identify them, either with their own custom type or something standard. If you want to be able to scan for them, on iOS there is a good BLE diagnostic app called LightBlue.

For Android with Heltec V3 you will need to use a serial connection for now. For iOS, if you install the test SidebandOS app and then flash one of your RNodes with https://github.com/cobraPA/RNode_Firmware_iOS/tree/kb_ble_test1, then you can connect it.

[liamcottle]

@cobraPA Perfect! Thanks, that was the info I needed. I was mistaken thinking those recent BLE commits to master firmware included this. After flashing the firmware from your kb_ble_test1 branch, against all odds, this worked for me.

I was able to disable WiFi and Cellular on my iOS device, launch Sideband, pair with the Heltec v3 RNode via BLE and then send a message from Sideband to my Windows PC that has another RNode connected to it via USB (running Sideband as well).

Great work on this so far! There are a few bugs I noticed, but it works! I noticed if you minimise the iOS app at all, Sideband disconnects from the RNode. You can see on the RNode screen that the BLE and LoRa icons change to disconnected state, and the RF traffic graph stops moving. I am unable to get the app to auto reconnect at all, by toggling any bluetooth settings etc... However, if I force close Sideband and relaunch it, it does in fact reconnect to BLE and everything starts working again!

One other thing I wanted to mention, is it possible to update the Sideband version on Test Flight? Currently it's on v0.7.7 and is missing the functionality to send attachments/images. These features exist in the v0.8.3 version I'm using on Windows/Android. It might be painful to do so since you have these in-development changes for BLE, so no problem if you can't do so easily.

Happy to test changes and report back if needed.

Thanks again!

[cobraPA]

Thanks for checking it out. Yes, updating to newer version of Sideband is on my todo. It's usually a large merge, although not always a hard one. The image send may take more work to hook in to iOS though. I also want to get the telemetry/gps integration working. You should find that the map displays, but the gps is not wired in on iOS yet.
There is some in-progress work on BLE background operation for the RNode. It should function for a short time at least, but more testing to be done there. I believe with the proper setup that it should work in the background, as supporting custom BLE devices is one of the allowed background modes for iOS (but that is not all done yet.)