Question to all users: Change sensor mqtt format?

[MichaelDvP]

The dallas sensors can be published in two different formats (nested/single), and i see disadvantages in both formats.
Nested (and HA):

sensor_data = {"sensor1":{"id":"28-EA41-9497-0E03-5F","temp":"23.2"},"sensor2":{"id":"28-233D-9497-0C03-8B","temp":"24.0"}}

The sensors are numbered to sensor1, sensor2, ..., but if a sensor fails or is temporary disturbed the numbering changes. Reading only sensor3.temp will give the value of sensor4 every time sensor1, 2, or 3 fails a readout. To make sure the right sensor is read, we have to search the right id in all sensors.

Single format publishes each sensor as a single message, topic sensor<id>

sensor_28-EA41-9497-0E03-5F = {"temp":23.2}
sensor_28-233D-9497-0C03-8B = {"temp":24.0}

Here the sensors are clearly identified, but the messages are published one after another. If you want to calculate the difference between two sensors, say solar flow and backflow, you have to check the timing. If calculated on publish the difference is always calculated alternating with a new and a old value, and than with both new. This looks like fluctuation.

Both massage formats are not consistent with other published values, the boiler always publishes {name:value}, i.e {wwtemp:45} and not {wwtemp:{temp:45}} and also not as a own topic boiler_wtemp = {temp:45}.

I want to ask all users: Can we change the single format to:

sensor_data = {"28-EA41-9497-0E03-5F": "23.2" ,"28-233D-9497-0C03-8B": "24.0"}

Than all simultaniously measured values are published the same time and the sensors are clearly identified. Also i find it more consistent to the other mqtt messages.

[bbqkees]

The alias 'sensor' works much nicer than the long ID. Nobody can remember the difference between 28-EA41-9497-0E03-5F and 28-233D-9497-0C03-8B.
Also it's easier in MQTT if you know you can expect (in a single topic) sensor1, sensor 2 than f.i. '28-EA41-9497-0E03-5F'.
If its in a single topic you also don't need to take time differences into account.

If a specific sensor is giving an alias sensor1, sensor2 that should not change anymore if one of the sensors is removed.

[MichaelDvP]

But sensor1,.. is not an alias, it's a count how the sensors answer to 1wire.
To show the problem: here is my configuration (left), i disconnect sensor2 (middle), and attach a different sensor instead (right):

And i suggest only to change the single format, not the nested format with sensor-number, id and temp.

[bbqkees]

I get the problem but I thought it was an alias all along.

[MichaelDvP]

I thought about that. For a real alias we have to store the sensors to filesystem, with possibility to clear the growing database. This is a lot of work and than we can also add offset, gain and 2.order for #520.

But the main problem is that sensors are renumbered in one is missing (dead or temporary fail) in normal running.
I tested to keep numbers fixed until reboot, so this is what it look like if i disconnect sensor 5 while running:

A new sensor will be added at the end of the list:

But after a reboot, this sensor8 will becomes 1, 1-4 are shifted and 5 isn't detected anymore.

It seems that all other users do not have a preference of mqtt-single-format. Maybe i should change to my preferred format and wait for the complains.

[proddy]

makes sense to remove the id (which is the order the sensors are detected) and replace it with the sensor's real ID.

[MichaelDvP]

As bbqkees wrote "The alias 'sensor' works much nicer than the long ID" and "if a specific sensor is giving an alias sensor1, sensor2 that should not change anymore if one of the sensors is removed."

That's what i have working here, the numbers do not change until next reboot. On startup i make 3 scans to be sure the numbering is stable. But i preferre a always stable detection with only id.

But maybe a compromise:

• skip the CRC from id, it's only one byte but makes it a little shorter.
• change HA format to sensor_data = {"<id>":<val>, ..} , with autodiscover nobody has to type the long <id>
• do not renumber sensors once detected (until reboot)
• keep nested format as it is sensor_data = {"sensor1": {"id":"<id>, "temp":<val>}, "sensor2": {,...}}
• change single format back to sensor1 = <temp> , that's easy to type but have both disadvantages (not synchronous, not stable numbering). Or leave it as is, or [insert your favorite format here].

I can push these changes to my github if you want to have a look, or try out.

[proddy]

sounds good, I'm fine with what you propose. I just did a big change so sync up first. Hope I didn't break anything. I think the new report command will be useful.

[proddy]

would it not make more sense to use the float values in the JSON payload instead of the rendered strings? In case someone wants to use gafana or something else to actually do something with the values.

{"28-EA41-9497-0E03":"19.0","28-233D-9497-0C03":"19.2"}

[proddy]

I made some edits. Can you take a look?

• reverted back to using floats for json. Reason is because of i18n where I want the Web to localize to the language/format. e.g in some countries its 19,6 and others 19.6
• fixed the HA (needed to use Nested format)
• removed ltoa, can use itoa also for 4-byte ints
• didn't fix the NAN/-1.-1 issue. I'm not sure how to recreate it. If there is no value the sensor would not be published/rendered. I may have missed something mind you.

[MichaelDvP]

And once again my favorite format is killed :-(
I stay with custom format in my own fork.

reverted back to using floats for json. Reason is because of i18n where I want the Web to localize to the language/format. e.g in some countries its 19,6 and others 19.6

All device values in web uses render_value via print_value_json, why not sensors? If you want localization you want it only for the sensors?

As shown in the picture above i wanted to show sensor and id but empty value it a sensor fails (crc-error or loose contact). With float the web crashes if only the value is empty.
Also you have to add back .toFixed(1), otherwise it looks odd if one sensor shows 22.1 and the next 22 instead of 22.0.

fixed the HA (needed to use Nested format)

boiler HA works without nested format, so it's not really needed, only the config-settings were wrong.

With the float output you have to check hasValue before mqtt output, otherwise every sensor read error gives 3200 degree (warmwhite), charts with autoscaling will only show spikes than. But for mqtt and api only skipping output["temp"] = .. works, whereas in web it crashes.

[proddy]

the reason I made the edits was because the MQTT payload was to sending the sensor values as strings and not floats anymore which broke my grafana logs and for other users too. Also the (unit) tests were failing and the GitHub build broke and rejected the PR.

But what I didn't realize is that you wanted also non-functioning sensors to be recognized and an empty temperature to be sent. I missed that in the screen-shot and you didn't explicitly mention that. Is this only in the Web or also in the MQTT payloads? What should the value be, just blank? Let me know and I'll make the changes.

You're right about all values being sent to the web are converted to strings, so i18n wouldn't work there. I was mistaken.

[proddy]

ok, think its ok now. I made one change to show "(not available)" in the WebUI when you have a sensor that is present but can't take a reading. It's never happened to me so I tested it manually.

[proddy]

@MichaelDvP you ok with the latest changes? happy now? :)

[MichaelDvP]

The missing sensor in web is clearly identified, looks good.

But why the api is called now dallassensor, whereas headline is Sensors and mqtt is sensor_data.

It's never happened to me

before added the retries i get two or three times a day a sensor shift. and last sensor NULL.

happy now? :)

Not really until i get my mqtt sensor_data which isn't renumbered on reboots.
{"28-7284-131F-1901":47.1,"28-3AA4-F513-1901":31.4,"28-7AF8-131F-1901":34.6,"28-967A-1F1C-1901":27.8,"28-1E84-2D1F-1901":29.9,"28-D1F1-3914-1901":26.9,"28-996C-181F-1901":30}
I stay with Custom added, so we can close this (it seems the other users don't have a meaning to this topic).

[bbqkees]

So are the sensors numbers now persistent between reboots or not?

[MichaelDvP]

Sensornumbers scanned on boot and kept until powerloss, The numbering is not stored to nonvolatile mem, on next reboot a missing sensor is skiped and numbers change.
Also a new sensor attached after startup is iset to the end of the list, but after reboot is sorted due to it's lasered id-code.

But temporary resorting while running if there is a crc-error is fixed. Also every sensor is retried 5 times (~ 30 sec) before marked as missing.

[proddy]

@MichaelDvP what's the difference between your Custom format and what we have now?

[MichaelDvP]

We have now:
Single:
sensor_data = {"sensor1":46.9,"sensor2":48.6,"sensor3":41.8,"sensor4":38.6,"sensor5":26.6,"sensor6":37.8,"sensor7":27.2}
=> Value: sensor_data.sensor1

Nested+HA:
sensor_data = {"sensor1":{"id":"28-7284-131F-1901","temp":46.9},"sensor2":{"id":"28-3AA4-F513-1901","temp":48.9},"sensor3":{"id":"28-7AF8-131F-1901","temp":41.9},"sensor4":{"id":"28-967A-1F1C-1901","temp":38.7},"sensor5":{"id":"28-1E84-2D1F-1901","temp":26.6},"sensor6":{"id":"28-D1F1-3914-1901","temp":37.9},"sensor7":{"id":"28-996C-181F-1901","temp":27.2}}
=> Value: sensor_data.sensor1.temp

I want to have (custom):
sensor_data = {"28-7284-131F-1901":46.9,"28-3AA4-F513-1901":47.7,"28-7AF8-131F-1901":41.8,"28-967A-1F1C-1901":38.1,"28-1E84-2D1F-1901":26.6,"28-D1F1-3914-1901":37.9,"28-996C-181F-1901":27.3}
=> Value: sensor_data.28-7284-131F-1901
The sequence can change , but the value will never change! It's a bit more to type after first connect, but than it's stable for all times.

[proddy]

I honestly thought we decided to replace the single format with your custom one? I don't think anyone would notice or mind. Let's just get it right. As long as HA works and the values are sent to MQTT as floats I'm happy

[MichaelDvP]

Ok, but imho there was no decision, there was only one vote (my) for change and one against. A voting participation of 0,2% users can never be a decision. But i change it an see what happen.

[proddy]

99.8% of users use the default MQTT format which is Nested, so it'll be fine.

By the way, the initial reason I made the 'single' format was for Home Assistant since the components couldn't parse JSON objects and I needed to have seperate topics and payloads for each heating circuit. Now with MQTT Discovery that problem has been resolved.

In my opinion I don't really see a need for having the single option anymore, but best to leave it as not to upset anyone.

Also in Sensors its not really 'single' as all the sensors are posted in one json payload. single should really be 'sensor_data_1', 'sensor_data_2', etc

[MichaelDvP]

Maybe there are other systems that does not like nested. Also splitting the thermostat can be good for 4 heatingcircuits getting to long in one nested message.
And all other messages are also not really single, but unnested (only the hc-nests from thermostat set to an extra topic), so the sensor-format does not break the logic.

[Oderik]

I want to have (custom):
sensor_data = {"28-7284-131F-1901":46.9,"28-3AA4-F513-1901":47.7,"28-7AF8-131F-1901":41.8,"28-967A-1F1C-1901":38.1,"28-1E84-2D1F-1901":26.6,"28-D1F1-3914-1901":37.9,"28-996C-181F-1901":27.3}
=> Value: sensor_data.28-7284-131F-1901
The sequence can change , but the value will never change! It's a bit more to type after first connect, but than it's stable for all times.

Me too! :D