MQTT reports zero values when no valid data is available

[Argafal]

Running 0.5.32, 64fb587, with three inverters.

In a setup with more than one inverter, MQTT sends data for inverters that are not yet available. These data are zero. See inverter/Second in the example below.

This behaviour causes downstream issues: MQTT listeners such as Domoticz expect only valid data. Counters such as YieldTotal must be strictly increasing. Since zero values are reported in-between valid data, statistics break down on the receiving end.

Considerations:

• It is not possible to simply ignore all zero values on the receiving end: For example, YieldDay will be zero at the beginning of the day, and rightfully so.
• It is neither (easily) possible to check validity of the data using /inverter/name/available or /inverter/name/last_success: each MQTT message is a stand-alone message received and processed separately and asynchronously.
• One possible solution could be to send one large json MQTT message, which encodes both the available flag and all the data points of the various channels (ch0/#, ch1/#) together. This can then be processed at once at the receiving end.
• However, I believe a better solution is to keep the existing MQTT messages. MQTT messages for the channels should then only be sent if valid data is available. Otherwise the channel MQTT messages should be skipped. /inverter/name/available and last_success can always be sent, providing information downstream that an inverter is not (yet) available.

$ mosquitto_sub -h localhost -v -t 'inverter/Pilot/ch0/YieldTotal' -t 'inverter/total/YieldTotal' -t 'inverter/Second/ch0/YieldTotal' -t 'inverter/First/ch0/YieldTotal';
inverter/Pilot/ch0/YieldTotal 579.034
inverter/First/ch0/YieldTotal 1056.533
inverter/Second/ch0/YieldTotal 0.000
inverter/total/YieldTotal 1635.567

[Argafal]

Proposed patch below, seems to resolve the issue for me.

~/bin/ahoy/tools/esp8266$ git diff
diff --git a/tools/esp8266/app.cpp b/tools/esp8266/app.cpp
index 52f22f4..c2857c5 100644
--- a/tools/esp8266/app.cpp
+++ b/tools/esp8266/app.cpp
@@ -632,7 +632,9 @@ void app::sendMqtt(void) {
             for (uint8_t i = 0; i < rec->length; i++) {
                 snprintf(topic, 32 + MAX_NAME_LENGTH, "%s/ch%d/%s", iv->name, rec->assign[i].ch, fields[rec->assign[i].fieldId]);
                 snprintf(val, 10, "%.3f", iv->getValue(i, rec));
-                mMqtt.sendMsg(topic, val);
+                               if(iv->isAvailable(mUtcTimestamp, rec)) {
+                                       mMqtt.sendMsg(topic, val);
+                               }
 
                 // calculate total values for RealTimeRunData_Debug
                 if (mMqttSendList.front() == RealTimeRunData_Debug) {

[stefan123t]

@Argafal this is only preventing individual values being sent for inverters which are not (yet) available, which is a good thing [TM] and should address your point 4. I understand that both 1. & 2. can neither be easily fixed on Sending (our) nor Receiving (in your case Domoticz) end of the MQTT communication. But what is the result / warning / error in case of 1. Domoticz realising that the value is going down and not continuously rising ? Could you use NodeRed or some other intermediary to validate the data coming in single pieces and not as a concise JSON value for 2. ?

@lumapu I believe we also had point 3. on our Todo list once in a while:

One possible solution could be to send one large json MQTT message, which encodes both the available flag and all the data points of the various channels (ch0/#, ch1/#) together. This can then be processed at once at the receiving end.

Should we create a separate issue for the "multiple MQTT values as one concise JSON topic" feature request ?

[Argafal]

@stefan123t

1. You asked about Domoticz's behaviour. Counters are expected to be strictly increasing. A decreasing value is not handled gracefully: Say the (correct) value of 2600 kWh is followed by a 0. The true difference is -2600, but since the values are expected to be strictly increasing the difference is held in an unsigned integer. It now becomes a gigantic number, more energy production than the Sun. This gigantic number is reported by domoticz as the truth, and as such messes up all statistics. For sure Domoticz should handle this better. However, I find that from ahoy's side it seems easier to not report zeros when no data is available.
2. I don't see an easy way to perform a consistency check in node red. Ad-hoc there is no knowledge of the previous max value available. Neither are the separate messages coupled in any way, so a more complicated logic would need to be resolve this at node red/domoticz's level. I would argue that the smarter solution is for ahoy to only send valid data.

[stefan123t]

@lumapu in order to prevent empty or uninitialized data to be sent via MQTT for each inverter,
we should have a flag somewhere that is only set after we got a complete RealTimeRunData_Debug Payload from that inverter.
This flag could also be read / used by our various output Plugins from MQTT over JSON to Serial / Display.