Browser crashing using a large dataset

[yannisoo]

Hi, I've been looking for a way to store large amount of data with flutter on a Web/iOS/Android app. For my project I need to use a large set of data (48mb) which I can get through either json or sqlite. I then want to store it within my flutter app. However when I try to store my data decoded from json my browser freezes without any error indicating something is going wrong. I tried with a "smaller" json (4.7mb) but nothing I'm facing the same issue. Is there any work around to this ? Thanks in advance!

[alextekartik]

I'm not sure what solution you are using currently to store your data. I'm assuming that you using sembast io (but I recommend sembast_sqflite) and sembast web.
Since sembast run on the UI thread a big map could indeed freeze during (sembast also encodes the data as json)

Storing a 48MB json as a single record won't work on sembast (with sqlite backend, records are limited to 1Mb on some platforms).

If you want to store raw json as a string, on mobile you could use compute to encode/decode map/list to/from string in a different isolate.
On the web, if you have json that freezes the UI when decoding/encoding there is nothing much we can do but to split the json in smaller parts stored separately.

You can then store the big string in indexed_db (web) or file system (io). fs_shim allow writing big data as a file on any platform or you can use any other key value database (hive).

If you can split your data in small records, sembast should work, see some recommendations

Maybe I misunderstood your question...