Big data set issue

[kailniris]

Hello,
I have an issue with handling big data set in Vue.js

The app:
The app has like 50k objects in its memory, lets call them items. Every item has an _id and a name property. I want to list the items to the user one at a time. The user picks an item then i put the item in the Vue instance to render it on screen.

The problem:
The data set is like 4 mb big. If i put all the data set (the items object) into a vue instance the 4mb data with the observer properties will be 85 mb. I tried to put the data set into a vuex store, the result was 89mb of memory usage. (This is only a test case, the real production data is like 100mb so in a vuex store it will be instant out of memory i guess).

So I did not put all the data in the instance cuz i do not want reactivity for all the objects, only the one the user picked at that time. So i store the data set in a plain object and if a user pick a an item i put the item into the vue instance then i will have reactivity for that one item. Everything good expects it causes memory leak. Every time i put an item into a vue instance a new observer property will be added to the object. Even if the object was in the instance before a new observer is added next to the unused ones. This works as intended i guess but my issue/feature request is:
Can i somehow remove unused observers and getter/setters or are there any other practices to handle big data set in vue?

Things that would solve the issue but I can't use them:
I could deep clone the item before put it into the vue instance, but i need to keep the original object refence cuz other parts of the app updates the object and i want to keep reactivity.
Could use a database engine but I need to have offline capability and fast IOless memory search locally.

Here are the test cases I made:

// data in Vuex store 89mb
var store = new Vuex.Store({
    state: {
        items: {}
    },
    mutations: {
        addItem(state, item) {
            Vue.set(state.items, item._id, item);
        }
    }
});

for (var index = 0; index < 50000; index++) {
    store.commit('addItem', { _id: index, name: 'item' + index })
}

var app = new Vue({
    el: '#app',
    data: {
        pickedItemId: 0
    },
    computed: {
        item() {
            return store.state.items[this.pickedItemId]
        }
    }
});

//all in the instance 85mb
var app = new Vue({
    el: '#app',
    data: {
        items: (function () {
            var elems = {};

            for (var index = 0; index < 50000; index++) {
                elems[index] = { _id: index, name: 'item' + index }
            }

            return elems;
        } ()),
        item: {}
    }
});

//plain object 4mb but memory leak every time when i update app.item with the picked item
var items = (function () {
    var elems = {};

    for (var index = 0; index < 50000; index++) {
        elems[index] = { _id: index, name: 'item' + index }
    }

    return elems;
} ())

var app = new Vue({
    el: '#app',
    data: {
        item: {}
    }
});

[LinusBorg]

So i store the data set in a plain object and if a user pick a an item i put the item into the vue instance then i will have reactivity for that one item. Everything good expects it causes memory leak. Every time i put an item into a vue instance a new observer property will be added to the object. Even if the object was in the instance before a new observer is added next to the unused ones.

Could you please provide a demonstration of this, on jsfiddle for example? How ecactly do you "put an item into a vue instance"?

[kailniris]

Sorry for not being clear enough, here is the fiddle you asked:
http://jsfiddle.net/bLaowkjf/6/

[kailniris]

I made some more test and it looks like I made some false statements about memory leak. It looks like if you make an object reactive in a vue instance and put that object in another instance or again in the first instance then the observer property is not added again, nor additional getters and setters are added. Am i right now?
However the high memory usage with a large data set is still an issue.

[HerringtonDarkholme]

the observer property is not added again

Yes, https://github.com/vuejs/vue/blob/dev/src/core/observer/index.js#L112

Can i somehow remove unused observers and getter/setters

And

other parts of the app updates the object and i want to keep reactivity.

are conflicting requirements, IMHO. To observe one object you have to maintain an observer.

Maybe Vue can have smaller observer instances, I'm not sure. But currently the workaround might be partition your data. That is, only maintain a set of objects you want to keep reactive and clean the reactive set.

Another suggestion is mark immutable properties with _ prefix, https://vuejs.org/v2/api/#data. It can waive several watcher instances.

[kailniris]

Can i somehow remove unused observers and getter/setters
And
other parts of the app updates the object and i want to keep reactivity.

Are not conflicting anyhow.
Lets say you have an object, you list the object properties to the user with vue. Then the user picks another object, so the first object is no longer visible for the user in the view layer. Why would you want to maintain reactivity anymore on that object? But if a user observes an object and an object property is changed by a function somewhere in the code, the update is displayed to the user.
So the basic idea is you don't want to maintain observers in an object which is not displayed in the view layer currently, its cpu and memory efficient this way on the long term. This is what i wanted to represent in the fiddle i posted earlier.
The problem is if you make an object reactive once, the observer stays even if the object is not really observed at that point but might in the future and of course using memory while doing nothing.
So the feature/method i wanted to ask here is a way to remove observers from objects what are not really observed at the point or determine objects which have an observer but not part of any instances without deleting the object.

[kailniris]

A suitable workaround would be if i could determine unused observers in object (objects which does not have reference in a vue instance or vuex) somehow, so i could delete unused observer properties, and unused getters and setters, freeing up memory from currently unused objects. Then readd observers again as default when the object is used again.
This would allow vue to work with extremely high amount of data with low memory consumption.
I know this is cpu intensive dirty checking makeshift garbage collection but its still better then nothing at all.

[LinusBorg]

Might something like this work out for you?

http://jsfiddle.net/Linusborg/bLaowkjf/7/

Essentially,:

• I clone the picked Item using Object.assign() before adding it to the vm,
• and that cloned object is made reactive by Vue. The original data stays non-reactive
• I also set up a watcher on that item, so any changes are mirrored to the original, non-reactive object
• as soon as you change the picked Item, that one reactive item will become unreferenced and be garbage-collected.

This is a quick&dirty demo, of course - it can be further improved, e.g. using deep-cloning when you work with nested objects/arrays.

[kailniris]

Thanks for working with the issue,
I know this is strictly my applications specification but i have an issue with this workaround.
The object reactive or not is always strictly modified from outer code, like a message from the server and never from inside a vue instance, even if the vue instance has an update method first its must be validated from the server and a message from the server will update the global items object.

What i was thinking but not sure if it can work, is:

1. Setup a vue instance
2. Create a mixin, for every vue instance which listens for global update requests.
3. Make an object reactive in a vue instance, but instead making the global reference reactive, deep clone the object
4. Global updates will update the global object and the reactive objects in vue instance separately.

The thing why i wanted to stay away from this is, the fact its go away from the vue's "the one source of truth" data model.

[sirlancelot]

With your latest comment in mind, my proposal would be to then clone the old item when a new one gets swapped in. The cloned object will not have reactive properties. Then, take that cloned object and assign it back to the array: items[oldIndex] = Object.assign({}, oldItem)

In this method, you maintain the single source of truth because the picked item is being made reactive in-place when you pick it, and then cloned back to a non-reactive item when you're done with it.

[kailniris]

@sirlancelot
This is a very useful idea, thanks for the contribution!
I did not know cloned objects do not get the observer property and the getter setters.
I made some tests and it works fine.
The only thing whats still missing is a way to determine if an object is not part of another instance before you can safely reassign it. I guess this can be achieved in the application logic.
Other thing I was think of some kind of dirty checking by resigning observed objects when the user is idle and the view layer is empty.
So before any official way is developed (if developed) I think this is a suitable workaround.
Thanks for the help!

[yyx990803]

Will any of these items be mutated? If they are not, you can simply call Object.freeze on them before setting it into the Vue instances. This will let Vue know that it can skip observing the internals of these items and basically solve your memory issue.

Even if you do need to mutate them, you can instead do something like this.item = Object.freeze(Object.assign({}, this.item, changedFields)).

[kailniris]

The items global object is mutated all them time from a network source, and that is the point why I chose Vue cuz the reactive objects mutations updates the view anytime without knowing or listening for object mutations.
In my project I wanted to achieve the followings.

• Get a global data set in a variable
• Show data in the view layer
• If an object is mutated in the global data set the view layer must be automatically updated representing these mutations without any other code.

With Vue I manager to achieve these goals without any problem expects from the increase of the memory use during using the app.

So i can't freeze the object or clone it before i make it reactive cuz i need the original object reference to get reactive view update when the global object is mutated .

http://jsfiddle.net/bLaowkjf/9/
I updated the fiddle to demonstrate the idea.

[LinusBorg]

I will close this for now, as this turned into more of a support thread than a bug report or feature request.

Feel free to continue conversation. and open another issue if it leads to an actual issue.

[kailniris]

As I stated before this was an actual feature request.
The request was an api to determine if a reactive object is part of any Vue instance or not.
The examples I provided was use cases to provide better understanding what would the requested feature helps to achieve.
Should I create a new issue with a more specific feature request about this?
Thanks for the help so far!

[LinusBorg]

Should I create a new issue with a more specific feature request about this?

Yes please (no guarantees weither this is feasable, though).

[fnlctrl]

@kailniris
I was thinking about the whether its worth adding the API in #4437, and dug up this old issue, let's continue discussion here.
After going through your jsfiddle, based on @sirlancelot and Evan's suggestion I concluded that maybe you should make the store object (items) reactive and let it notify view updates (instead of making items reactive), then you can use Object.freeze to skip adding watchers on each item for a performance gain.

jsfiddle: http://jsfiddle.net/otcL62wx/

Therefore, #4437 shouldn't be necessary.

[kailniris]

Thanks for working with the issue!
This idea is fascinating, the reactivity works and with 50000 objects its only 24mb of extra memory use compared to plain objects.
I guess this can work with a Vuex store too which can be more suitable with multiple global reactive data sets.
I will try out this idea with a real data set then write a report.
Thanks again!

[fnlctrl]

@kailniris Any updates? 😄

[kailniris]

@fnlctrl Sorry for keep You waiting. I am still making some tests and working to implement this pattern on my code. I will respond back asap as I will have some results.

[kailniris]

Sorry, for the late replay. I managed to implement the pattern and made lots of tests.
Everything works fine. Looks like this will be a good solution for this issue.
I think the feature request is unnecessary now. I will close it.
Maybe this pattern should be added to the docs.
Anyway big thanks for the help!

[fnlctrl]

Glad to help!

[kunKun-tx]

Object.freeze saves almost 800MB of memory on my big 5K nested array!

[AchevezAim]

Just had this issue.
The use case was as follows:

• Had an object containing product info (price included)
• Needed to send the item to cart, but before that I needed to calculate the post tax price.
  Doing it this way:
  let frozenPrice = this.selectedPrice;
frozenPrice.value = this.calculateTax(frozenPrice.value, taxFromSomewhereElse)

Doing it that way caused the UI to update with the post tax value, because somewhere in the HTML we were listening to this.selectedPrice.

So, after a couple of hours of research, trial and error, tried this:

let frozenPrice = {...this.selectedPrice};

This allowed me to make all the operations without activating or modifying the selectedPrice object on data() thus, no unwanted ui updates.

😄