Serialization and deserialization too slow

[trbKnl]

Serialization and deserialization of objects to pass between pydodide and the main threat causes very large donation to fail with memory overflows.

[trbKnl]

Serialization is not necessary. Make it optional.

This means to pd.DataFrame.to_json() step is skipped. But also json.dumps can be avoided. Both options consume too much memory.

This means data needs to be extracted into the dict equivalent of pd.DataFrame.to_json().

solution

send back dict object like so:

{ 
    "variable_x": {"0":1},
    "variable_x": {"0":1},
}

This results in tables being able to load massive amounts of data: