Best BLE proxy for Xiaomi Mijia?

[pzich]

I just picked up some Xiaomi Mijia units, and due to the location of my Home Assistant setup, I'm going to need some sort of Bluetooth proxy for my devices to talk to. It seems like most of the articles about this suggest using some form of ESP32, but I see your README says not to use ESP32xxx devices:

Do not use ESP32xxx - it cannot work with BLE at normal speed and drains the batteries of all BLE devices when connected!

ESPHome does not work with Bluetooth 5.0 and misses a lot of advertising packets from BLE devices.

I was wondering if anyone has recommendations for a device that can proxy BLE 4.2 and 5.0+ signals over WiFi? Thank you!

[AleXSR700]

Interesting. Never saw that comment on ESP32 devices. Maybe someone knows details on this?
I use a ESP32 running tasmota. No idea if this causes additional battery consumption or not.

[AleXSR700]

I wonder if this only applies to devices with default broadcast. I don't quite see how a broadcast every 10 seconds is an issue for the ESP32.

[pvvx]

• One antenna for BT and WiFi. Most of the time the antenna is used for Wi-Fi.
• FIFO is not used when receiving BLE advertising.
• Message processing speed is limited by MQTT transaction over WiFi. BLE is not serviced during this period.
• Overloaded software. Slow processing speed of BLE stack on connection. Plus there are a lot of gaps due to Wi-Fi switching. Requires repeated duplication of message transmission for BLE devices. Battery consumption for BLE devices increases tenfold.

With the current bloated codes in the SDK and the rest of the software for simultaneous operation of MQTT-WiFi-BLE, which has long been out of the way of the Flash cache of the ESP32, with 2 cores the ESP32 has performance no higher than Cortex M0 at a frequency of 10..16 MHz. This is due to the speed of reading code from SPI-Flash, which does not fit into the cache. At the same time, the ESP32 heats up thoroughly and consumes more.

[pvvx]

I use a ESP32 running tasmota.

https://tasmota.github.io/docs/Bluetooth/#bluetooth-low-energy-sensors
The other advantage is, that scanning for BLE advertisements can happen nearly parallel (very quickly one after the other), while a direct connection must be established for at least a few seconds and will then block both involved devices for that time.

With normal BT adapters, no blocking occurs. The BT adapter (client) can simultaneously receive advertising packets and support connection with multiple devices. Also, depending on the software implementation, a BLE device has the ability to simultaneously transmit advertisements and maintain connections with multiple clients.

_ (very quickly one after the other)_ - the required speed of full processing of BLE advertisements from devices is up to 3 ms, which ESP cannot provide. The ESP receives the message and transmits it to MQTT via WiFi. There are delays in WiFi that last more than 1 second. During this time, hundreds of advertising messages from BLE devices will be missed.

WiFi works with a strict step - the default WiFi AP sends a beacon every 102.4 ms. WiFi AP distributes communication time for clients between these beacons.
To reduce collisions in the BLE radio air, the device transmits advertising messages at intervals using a low random delay. But at some time interval, the step to go on air for WiFi and BLE may coincide.
Regular (modern) WIFIs have Bluetooth device detection to reduce collisions. But not ESP.
This is observed as long gaps in data in graphs from users using ESP.

For normal operation, BLE and WiFi antennas must be separated by a distance of half a meter. To work with WiFi you need to use one chip, and to work with BLE - another. Otherwise, there will be a lot of gaps, because the receiver needs to have time to adjust the automatic reception level depending on the noise on the air - to have time to adapt to each BLE advertisement.

https://tasmota.github.io/docs/Bluetooth/#bluetooth-low-energy-sensors
pvvx Telink Flasher also allows to flash custom firmware on supported sensors. This will work out of the box with Tasmota. This firmware does send data more frequently and is a bit more power hungry than the stock firmware.

As a result, it turns out that with tasmota, the consumption of the BLE device increases.
With standard BT adapters operating in passive scanning mode, the situation is the opposite - the firmware from MiHome consumes more than alternative firmware.
https://pvvx.github.io/ATC_MiThermometer/OriginalPower.html
https://pvvx.github.io/ATC_MiThermometer/CustPower.html

[bam80]

Requires repeated duplication of message transmission for BLE devices. Battery consumption for BLE devices increases tenfold.

Just to clarify - is this only applies to the bi-directional connection mode?

Tasmota recommends "passive" mode instead which apparently should not suffer:

Basically all of them use the so-called "MiBeacons" which are BLE advertisement packets with a certain data structure, which are broadcasted by the devices automatically while the device is not in an active Bluetooth connection. The frequency of these messages is set by the vendor and ranges from one per 3 seconds to one per hour (e.g., for the battery status of the LYWSD03MMC). Motion sensors and BLE remote controls start to send when an event is triggered. These packets already contain the sensor data and can be passively received by other devices and will be published regardless if a user decides to read out the sensors via connections or not. Thus the battery life of a BLE sensor is not influenced by reading these advertisements and the big advantage is the power efficiency as no active bi-directional connection has to be established. The other advantage is, that scanning for BLE advertisements can happen nearly parallel (very quickly one after the other), while a direct connection must be established for at least a few seconds and will then block both involved devices for that time.

This is therefore the preferred option, if supported by the sensor.

https://tasmota.github.io/docs/Bluetooth_ESP32/#mi32-sensors

[pvvx]

Just to clarify - is this only applies to the bi-directional connection mode?

Yes.

And multiple losses when receive advertising events BLE on ESP - for other cases.

[pzich]

So it sounds like ESP32 is not a good choice, but what is? Could I use a raspberry pi, perhaps connected via ethernet and with wifi turned off, so the radio can be used exclusively for bluetooth?

Or what about using its wifi, but with an external bluetooth usb dongle, that could also be placed more than half a meter from the pi itself?

[pvvx]

ESP32 cannot successfully handle BLE in Arduino (and over SDK) at 160MHz CLK. Even at 240 MHz, packet acknowledgments are delayed, which greatly affects the batteries of the sensors.

An example of reading 50 measurements from history in a thermometer.

50 measurements are transferred to the ESP32 in 12 seconds

(power СR2032 mA/ms)

ESP32 does not have time to process incoming packets and the thermometer has to constantly duplicate transmissions before receiving confirmations from ESP32. Wi-Fi on the ESP32 is disabled so as not to further slow down BLE.

TelinkMiFlasher.html read 50 measurements (0.2 sec):

(power СR2032 mA/ms)

Reading the full history from the thermometer’s memory (20 thousand measurements) on any BT adapter (except ESPxx).

ATC_v39, pin-code enable.
Chrome log:
Read time: 18.542 sec

Real read cycle: 17320 ms (by measurement on the graph)

Average per package: 17320/19632 = 0.8822 ms -> 1133.5 package per second

[pvvx]

Due to errors and limitations in the software of some BT adapters. Where Bluetooth SIG standards do not apply.

The standard specifies a BLE advertising interval of up to 10 seconds.
Windows and Linux require two advertising events to be received to make connection requests. On Android, receiving one advertising event is enough.
The reason is the laziness of software developers for Windows and Linux operating systems.

[AleXSR700]

I am using Android.
Hmmm. So in theory setting advertising interval to 10000 ms should work with flasher in Chrome for Android?

[pvvx]

Yes. In the nRFConnect program for Android, you can create BLE advertisements in 10-second increments and check.
Also there you can set BLE advertising in the "Coded PHY S8" (LE Long Range) format and check other systems for work (for connection and reception of advertising).

In addition, advertising interval to 10000 ms is not a limitation in the new Bluetooth standards. But we are limited by Bluez and Bleak.

[AleXSR700]

Hmm. Could not get it to work on Windows or Android machines with 10 s. Also not with nRFConnect.

[pvvx]

#172 (comment)

[pvvx]

Low cost BLE Advertisements Repeater into Ethernet TCP/IP

[pvvx]

Low cost BLE Advertisements Repeater into Zigbee

BLE advertisements are accepted 99.5% of the time: Zigbee polling occurs 16 ms every 4 seconds. This results in minimal reception loss.

[AleXSR700]

Low cost BLE Advertisements Repeater into Ethernet TCP/IP

Aren't they ~ 20 €? So roughly 5 times the price of an esp32 with wifi?

[pvvx]

$0.5 per single chip.
In Russia, in an expensive store, CH32V208W costs $1 if you buy one chip.
Debugger WCH-LinkW with CH32W208W on aliexpress costs $0.5

There is no ESP device kit with Ethernet from the manufacturer at all.

ESP32 does not work well with BLE. ESP32-C3 is better. And the consumption of ESP+Ethernet is 20 times greater than that of CH32V208W.

[AleXSR700]

Admittedly I do not speak russian but I searched for the CH32V208WBU6-EVT-R0 you mentioned and the cheapest one on aliexpress (searching in the european version) is 13 €.
I couls not find any versions witg ethernet for a euro.

I am aware that the esp32 do not offer ethernet, that is why I said wifi :)
Wifi is of course a million times more convenient, especially when the device is supposed to act as a relay/hub.
But I understand your point. RPi is far worse even when it comes to simultaneous use of bluetooth and wifi. Can't even use bluetooth mouse + keyboard while surfing wirelessly. But the esp32 works great as a ble to wifi relay.

But probably, as you mentioned elsewhere in the past, at the cost of battery life of the sensors.

[pvvx]

I am aware that the esp32 do not offer ethernet, that is why I said wifi :)

1. WiFi is not suitable for a Smart Home. I have a hundred devices and I don't want to live in a microwave. 100 BLE devices on average have the same emission as one BLE. Same with Zigbee, but usually Zigbee has a little more power, but not like WiFi. Also, I don’t play with one sensor, but use home automation.
2. Wi-Fi has the lowest reliability. There are many reasons for this. One of them is high consumption and cheap power supplies fail within two years. Cheap Wi-Fi routers do not support hundreds of devices and often freeze. My statistics for 5 years show that on average there is at least 1 connection failure with a WiFi device per day. This can be monitored in Home Assistant logs.
3. Wi-Fi requires powerful routers and each sensor consumes 3 W or more. As a result, my entire smart home network will consume more than 400 W constantly. And this will add air conditioning to cool them.
4. WiFi interferes with Zigbee and BLE. EPS devices with WiFi do not work on 5GHz.
5. Available BLE-WiFi gateway (ESP) implementations have long data transmission intervals. This is not suitable for home automation.
6. BLE-Wifi repeaters on ESP are unstable and have multiple reception dropouts from BLE sensors. That is, they have the lowest level of reliability of all commercially available options. Even if you take BLE routers with WiFi from well-known manufacturers, none of them use ESP chips. The cheap ones use Realtek chips...

No one is stopping you from playing with ESP and a couple of sensors. There are a lot of resources for this, where there are ready-made programs for beginners. ESP software is designed for beginners with the goal of learning and immersing themselves in all the nuances :P

If you need BLE/Zigbee in WiFi, then on aliexpress there is Tuya Smart Gateway BLE Zigbee WiFI RSH-GW018_DM with modules:

• WBRG1 - RTL872xDM ARM®Cortex™-M4F+M0, 2.5MB RAM, 8MB Flash
• ZS3L - EFR32MG21A020F768IM32-B
  2.5MB RAM - This amount of RAM fits a minimal TCP compatible socket with a TCP stack. ESP uses stripped-down ones - used only for toys.
  And there are all SDKs for them. You can find out how to flash your programs in such hubs at https://github.com/parasite85/tuya_tygwzw1_hack. But this doesn’t interest me, because I’ve been using such modules for a long time...

EFR32MG21 has good reception sensitivity.

LILYGO T-Zigbee on ESP32-C3 with TLSR8258 is more expensive, does not have a housing and the characteristics are much worse.

[AleXSR700]

I think a lot of people consider ble to be the worst possible choice for Home Automation.

I do not have hundreds of wifi devices but the ~50 I have running in parallel show no signs of failing. Nor does the router seem to have any issues.
Power consumption of a wifi device like a power socket seems to be around 1 W, maybe less, maybe slightly more. Would need a very precise measurement device to check. Not sure if two identical devices but one with ble and one with wifi would have really different consumption (talking mains power, not battery).

The 2.4 Ghz is chosen for the same reason why ble is probably at a disadvantage when it comes to in-wall placement: range.
5 Ghz has poor range through thick walls. And sensors need basically no bandwidth, so 2.4 Ghz is sufficient.
I had to place my esp32 quite carefully to be able to teceive the packages from all bluetooth sensors. And the farthest one is less than 10 m away. Wifi does not present this issue for me.

[pvvx]

Write the name of a Wi-Fi router that supports “about 50 parallel connected” Wi-Fi devices.

Power consumption of a wifi device like a power socket seems to be around 1 W, maybe less, maybe slightly more.

Place one socket in another and see what it shows :)

I had to place my esp32 quite carefully to be able to teceive the packages from all bluetooth sensors.

LE Long Range has a greater transmission range than household WiFi. And no repeaters are required.
The ESP32 has poor sensitivity for BLE and only supports the legacy Bluetooth standard from 2014.

[ioukos]

Hi,

I'm no expert ; I don't understand why/how, could an ESP32 "listenning" to adverstisment from Xiaomi Sensor drain the battery of the Xiaomi sensor ?

Is it because even if it's an advertisment there is a connexion established between the ESP32 and the sensor ?

I'm more used to 433MHz devices that just "send" their data into the air no matter if another device is listenning.

Anyway, thank you Victor for the good work.

I'm discovering your firmware, and I can read data from ESP32 (through serial) using :
https://github.com/matthias-bs/ATC_MiThermometer/blob/main/src/ATC_MiThermometer.h

what could be the limitations ? If I run Wifi to send data to MQTT let's say, I won't get all the updates ?

Thank you

[pvvx]

I'm no expert ; I don't understand why/how, could an ESP32 "listenning" to adverstisment from Xiaomi Sensor drain the battery of the Xiaomi sensor ?

We are talking about the connection mode. A full-fledged BLE gateway must be able to connect to devices. For example, to read history or set the time. Possibly OTA too...

And when listening to advertising, if Wi-Fi is on at the same time, there are many reception gaps. And the ESP32 is bad with BLE.
For ESP32 with Bluetooth version 4.2, the average reception range is up to 10 meters.
Even ESP32 lovers write about this:

I had to place my esp32 quite carefully to be able to teceive the packages from all bluetooth sensors. And the farthest one is less than 10 m away.

For BLE receiver, it is better to use ESP32-C3 or later. My tests showed 500 meters in BLE LR.

For the WiFi-BLE-Zigbee gateway to work properly, the WiFi and BLE antennas must be at a distance of at least half a meter.

[BelgarionNL]

so TLDR whats the best device to get for these bluetooth devices? I still dont see a definitive device I should get from ali express.

[pvvx]

This depends on the configuration of your Smart Home.
I use two USB-BT adapters with Home Assistant.
One on the CSR8510A10 works with near 40 sensors, and the second on the RTL8761BU with long-range sensors (covers an area of 500 meters).
For reliability, all this is duplicated by BLE devices of our own assembly. Autonomous control is activated for everyone if the system on Home Assistant stops transmitting alerts or commands at a specified interval.
In addition, most executive devices themselves receive and process data from BLE sensors, and Home Assistant is a recording device for them.
All WiFi routers turn off when operating autonomously due to high consumption. All that remains is a router with Wi-Fi at 5 GHz for communication with the external Internet and local wired Internet to cameras, etc.

The simplest option, published a long time ago https://github.com/pvvx/AdScanerTrg

More complex devices, such as a controller for a water storage tank with support for temperature, fill levels, turning pumps on and off with monitoring of all parameters such as overcurrent, jamming, etc. I don't see the point in publishing it. Because it works differently for everyone.
This is also affected by the weather conditions you live in and the options you enabled when creating your personal living area. For example, in my area of residence in winter, no solar panels work and autonomy is built differently...

[BelgarionNL]

ok. I already use a asus bt500. which I believe is the RTL8761BU. its just in use for my pc instead of my unraid server. But I can use it for some testing in my server.

Thanks! I didnt know it could reach 500m.

[pvvx]

But such distances are achieved when operating in LE Long Range mode. And while working in LE LR, regular "Legacy" BLE is not accepted. When working on two types of PHY, the adapter periodically switches, which will also contribute to reception failures.
This is why you have to use two adapters. One for BT4.2, and the second in LE LR mode with BT5.0+

In LE mode, the LR RTL8761BU accepts signals down to -110 dBm.
This is not achievable for chips running Wi-Fi or Zigbee, as well as BT4.2 due to the RF protocols' modulation and restoration techniques. The rest relates to the antennas, not the sensitivity of the chips.

[1igeza]

I have EDUP LOVE EP-B3536 [RTL8761BU] and it's working without any problems with Home Assistant. Can I use second one and set it to LE LR mode? How to select between LE LR and BT4.2 mode?

[pvvx]

#297

[billyl]

I've been following this thread but I am still confused with the answer that is selected. If my goal is just have a couple of these xiaomi sensors connected to a ESP32-C3 on tasmota within a couple meter range . Is that fine? I am still confused as to what is the recommended device without getting to usb adapters. I would like something that is standalone that can perdiocally log temperatures to a database.

There is a new ESP32C6 that supports zigbee: https://wiki.seeedstudio.com/xiao_esp32c6_getting_started/
Would that be the best alternative?

[pvvx]

There is a new ESP32C6 that supports zigbee: https://wiki.seeedstudio.com/xiao_esp32c6_getting_started/
Would that be the best alternative?

Try it and write.