[TB] Unable to de-serialize received json data with error (DeserializationError::NoMemory)

[TPCQitek]

I have made the changes as in the readme, but still get the de-serialize error, the RPC is currently not firing the function.

the debug shows:

14:50:44.234 -> Subscribed to RPC methods
14:51:00.405 -> [TB] Received data from server over topic (v1/devices/me/rpc/request/387)
14:51:00.405 -> [TB] Unable to de-serialize received json data with error (DeserializationError::NoMemory)

the TB Log shows
{
"entityId": "6c5e7550-4327-11ef-86ad-337f063d8218",
"oneWay": true,
"method": "power",
"params": "true"
}

I currently have:
// ********************* start up events **********************
#define THINGSBOARD_ENABLE_DYNAMIC 1
#define THINGSBOARD_ENABLE_PROGMEM 0
#define THINGSBOARD_JSON_BUFFER_SIZE 256
#define THINGSBOARD_ENABLE_DEBUG 1
#define THINGSBOARD_ENABLE_STREAM_UTILS 1

#include <ThingsBoard.h> // 13.0
#include <ArduinoJson.h> // 7.1.0
#include <Arduino_MQTT_Client.h> // 1.7.3
#include <StreamUtils.h> // 1.9
#include <WiFi.h>
#include <HTTPClient.h>

WiFiClient espClient;
Arduino_MQTT_Client mqttClient(espClient);
ThingsBoard tb(mqttClient);

// Enable verbose debugging output
#define DEBUG_LEVEL_TLS_ERROR

is it related to the Lib versions?

not sure if its connected, i updated StreamUtils to 1.9 and now I get a

16:55:04.868 -> ELF file SHA256: a9b6c20fd1abb1d5
16:55:04.868 ->
16:55:04.868 -> E (337) esp_core_dump_flash: Core dump flash config is corrupted! CRC=0x7bd5c66f instead of 0x0
16:55:04.868 -> Rebooting...

clearly an error. I can recover the core and reprogram.

any suggestions to change / remove?

thanks in advance..

[MathewHDYT]

The aforementioned error, DeserializationError::NoMemory can occur because of 2 different faults:

Either there is not enough heap memory to use the THINGSBOARD_ENABLE_DYNAMIC mode, to check that please add this code to the onMQTTMessage method.

const size_t document_size = JSON_OBJECT_SIZE(Helper::getOccurences(reinterpret_cast<char *>(payload), ':'));
TBJsonDocument jsonBuffer(document_size);
if (jsonBuffer.capacity() != document_size) {
    Logger::printfln("Could not allocate required size (%u) for JsonDocument, allocated only (%u). Ensure there is enough heap memory left", document_size, jsonBuffer.capacity());
    return;
}
....
#if THINGSBOARD_ENABLE_DEBUG
Logger::printfln("Allocated internal JsonDocument for MQTT server response with size (%u)", document_size);
#endif // THINGSBOARD_ENABLE_DEBUG

The additional log message, prints the actual size of the JsonDocument, because the only remaining option is that the internal allocated JsonDocument size is to small to hold the actual data, could be the case if the Helper::getOccurences() method does not work correctly, because of the reinterpret_cast<char *>(payload).

For now if you could add the above code and retry the same behaviour and send the new log messages that would help a lot with debugging this issue thanks.

[TPCQitek]

Mathew hi:

I am using the 8mB ESP32, and the compile stats are:
uses 769497 bytes (23%) of program storage space. Maximum is 3342336 bytes.
Global variables use 44456 bytes (13%) of dynamic memory, leaving 283224 bytes for local variables. Maximum is 327680 bytes.

can run over the code and see what might be now corrupting the core when this loads, I have not seen that before..

this just has the 2 RPC,

can you confirm the lib versions, as I updated the StreamUtil manually from 1.7 to 1.9
or what over Lib do I need to remove and reinstall now?

// getting serialize error 9 PM Monday
// save Tuesday with error to preserve semi working

// build RPC_Subs_test_aug27a tuesday AM

// ********************* start up events **********************
#define THINGSBOARD_ENABLE_DYNAMIC 1
#define THINGSBOARD_ENABLE_PROGMEM 0
#define THINGSBOARD_JSON_BUFFER_SIZE 256
#define THINGSBOARD_ENABLE_DEBUG 1
#define THINGSBOARD_ENABLE_STREAM_UTILS 1

#include <ThingsBoard.h> // 0.13.0
#include <ArduinoJson.h> // 7.1.0
#include <Arduino_MQTT_Client.h>
#include <StreamUtils.h> // 1.9.0
#include <WiFi.h>
#include <HTTPClient.h>

WiFiClient espClient;
Arduino_MQTT_Client mqttClient(espClient);
ThingsBoard tb(mqttClient);

// Enable verbose debugging output
#define DEBUG_LEVEL_TLS_ERROR

// WiFi credentials
const char* ssid = "telstra";
const char* password = "1234";

// ThingsBoard credentials
const char* TOKEN = "abcd";
const char* SERVER = "thingsboard.cloud";

// Forward declaration of RPC functions
void processPower(const JsonVariantConst& data, JsonDocument& response);
void processRebootRPC(const JsonVariantConst& data, JsonDocument& response);

// Define RPC callbacks as a global variable
const std::array<RPC_Callback, 2U> callbacks = {
RPC_Callback{ "power", processPower },
RPC_Callback{ "reboot_device", processRebootRPC },

};

bool TB_connected = false;
bool RPC_subscribed = false;
unsigned long lastHeartbeatTime = 0;
const unsigned long heartbeatInterval = 30000; // 30 seconds for testing

void setup() {
tb.setBufferSize(128);
Serial.begin(115200);
WiFi.begin(ssid, password);
Serial.println(" RPC_Subs_test_aug27a");
// Connect to WiFi
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("WiFi connected");

// Connect to ThingsBoard
if (tb.connect(SERVER, TOKEN)) {
TB_connected = true;
Serial.println("Connected to ThingsBoard");
} else {
TB_connected = false;
Serial.println("Failed to connect to ThingsBoard");
}

// Initial RPC subscription
subscribeToRPCs();
}

void reconnect() {
if (!WiFi.isConnected()) {
Serial.println("Reconnecting to WiFi...");
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("WiFi reconnected");
}

// Reconnect to ThingsBoard
if (!tb.connected()) {
Serial.println("Reconnecting to ThingsBoard...");
if (tb.connect(SERVER, TOKEN)) {
TB_connected = true;
Serial.println("Reconnected to ThingsBoard");
} else {
TB_connected = false;
Serial.println("Failed to reconnect to ThingsBoard");
}
}
}

void loop() {
tb.loop();

// Heartbeat check
if (millis() - lastHeartbeatTime > heartbeatInterval) {
lastHeartbeatTime = millis();
heartbeatCheck();
}
}

void heartbeatCheck() {
// Check if connected
TB_connected = tb.connected();

if (!TB_connected) {
Serial.println("Reconnecting to ThingsBoard...");
reconnect();
}

// Subscribe to RPC if not subscribed
if (TB_connected && !RPC_subscribed) {
subscribeToRPCs();
}
}

void subscribeToRPCs() {
Serial.println("Subscribing to RPC methods...");

const RPC_Callback* begin = callbacks.data();
const RPC_Callback* end = callbacks.data() + callbacks.size();

if (!tb.RPC_Subscribe(begin, end)) {
RPC_subscribed = false;
Serial.println("Failed to subscribe for RPC");
} else {
RPC_subscribed = true;
Serial.println("Subscribed to RPC methods");
}
}

// Simplified RPC handlers

void processPower(const JsonVariantConst& data, JsonDocument& response) {
Serial.println("processPower RPC triggered");
bool power_state = data.as(); // Assuming a simple true/false JSON payload

// Implement your power control logic here
Serial.print("Power State: ");
Serial.println(power_state ? "ON" : "OFF");

response["success"] = true;
}
void processRebootRPC(const JsonVariantConst& data, JsonDocument& response) {
Serial.println("processRebootRPC RPC triggered");

// Check if the data is a JsonObject
if (!data.is()) {
Serial.println("Received data is not a JSON object");
response["success"] = false;
return;
}

// Get the size needed for the JSON object in the document
const size_t document_size = JSON_OBJECT_SIZE(data.size());

// Allocate the JsonDocument with the calculated size
StaticJsonDocument<256> jsonBuffer; // Adjust size as needed

// Log the allocated size for debugging
#if THINGSBOARD_ENABLE_DEBUG
Serial.printf("Allocated internal JsonDocument for MQTT server response with size (%u)\n", document_size);
#endif

// Deserialize the incoming data into the jsonBuffer
DeserializationError error = deserializeJson(jsonBuffer, data);
if (error) {
Serial.print("Failed to deserialize JSON: ");
Serial.println(error.c_str());
response["success"] = false;
return;
}

// Assuming that the reboot command does not need any specific data
response["success"] = true;
serializeJson(response, Serial);
Serial.println();

delay(1000); // Delay for debugging, remove in production
ESP.restart(); // Reboot the device
}

[MathewHDYT]

You need to adjust the onMQTTMessage like shown above, this is required because to debug the issue I require more information on what part exactly fails in the deserialization.

---

Furthermore when you use RPC with Dynamic mode, the RPC_Response has a third argument that defines the size of the response JsonVariant, per default it is only big enough to hold response.set(...). If you want to set response["success"] = true; you need to increase the size over the third argument to JSON_OBJECT_SIZE(1).

Additionally I'm not sure why you are copying the JsonDocument in the processPower method, because you are not even using it. Could you not simply call ESP.restart(); when the method is called?

Because setting the response is useless anyway, because it will never be sent if you restart the device before the method finished executing. Because the response data is only sent after the method has finished executing.

[TPCQitek]

Mathew, hi,
firstly, thanks for the prompt replies. As a bit of background, I had the RPC working ok, but it had not been used much recently other than I had people using the TB Scheduler and that was fine, but when I was reimplementing the extra RPC (currently 8 functions) I started to get these errors, and it snowballed from there, as these things do. I had migrated from the ESP32 4mB to the 8mB and that was at the same time the json issues were there back earlier.

I have gone back to basics and refined the set up. and reverted back to non dynamic operations:

thats solved the memory and Unable to de-serialize json etc..
and also solved the error too many subscriptions.

Arduino_MQTT_Client mqttClient(espClient);
ThingsBoardSized<32, 8> tb(mqttClient, 128);

the code example was simply a strip down from the mainstream code, to test and reasonably evaluate. So, the function to reboot was simply a place holder and like the others not final operations. For example, the reboot function includes resolving the cause of the reboot, Put to the eeprom and the doing the reboot itself. So when it restarts the eeprom has the reason stored. and that is sent at start up... etc.

The mainstream code is around 15K lines and have a lot of things going on. Now including ChatGPT and building Dynamic Prompts, where the sensor constructs Air Quality messages for things like Pollen detection (Indoor readings from the Sensor itself, and from Polling EPA and Government formal Pollen sensing and then used "descriptions" fear, anxiety, gen types, and others to send out social media messages to the user and carers etc.) It becomes very interactive.

So, a strip down is necessary to isolate issues.

I will close the issue now, as the RPC is largely OK, and later experiment with options. As I am getting a more refined understanding and impacts.

Testing looks ok now; I will post on my AWS - OTA in a few days.

Thanks again.. cheers.