Pushing many records with belongsTo relationships into the store is extremely slow.

[jslippy]

I recently upgraded from beta 12 to beta 16.1. After the upgrade, pushing multiple records with belongsTo relationships into the store is very slow. What used to take about 350ms for ~450 records now takes about 4 seconds in my application. With thousands of records it takes several minutes just to push the records into the store.

Most of the delay seems to be generated by the ManyArray.flushCanonical function:
https://github.com/emberjs/data/blob/v1.0.0-beta.16.1/packages/ember-data/lib/system/many-array.js#L67

There are a couple of TODO comments indicating this code is a work-in-progress and will be improved once 'proper diffing' has been implemented. Does anyone know what the status is on this? I didn't see an issue on it anywhere (although @lsowen asked the same question in issue #2981). @igorT also mentioned 'smarter diffing' in issue #2666. @igorT, do you know if anyone is working on a solution to this performance problem?

Here is a jsbin demonstrating the problem using ember 1.11.3, ember-data 1.0.0-beta.16.1, and jQuery 1.11.2 (which are the versions I use in my application):
http://emberjs.jsbin.com/foxalozeba/2/edit?html,js,output

If you vary the len variable in the for loop which creates the books records you can see that the amount of time to push the records into the store increases exponentially with the number of records being pushed.

With the canary builds of ember, ember-data, and jQuery 1.10.2, performance seems slightly better (~2 minutes to push 8000 records vs ~3 minutes) but still grows exponentially with the number of records.
http://emberjs.jsbin.com/qubisekige/2/edit?html,js,output

[igorT]

Are you pushing records manually or doing that within an adapter?

[jslippy]

I'm pushing them manually.

[igorT]

Are you pushing just a single type or multiple types?

On Thu, May 7, 2015 at 5:17 PM, jslippy [email protected] wrote:

I'm pushing them manually.

—
Reply to this email directly or view it on GitHub
#3052 (comment).

[igorT]

Can you try this branch https://github.com/emberjs/data/tree/pushMany_optimization
And use pushMany instead of iterating over push

[jslippy]

I'm pushing multiple types. I have two models each with a belongsTo relationship to a third model which has two attributes with hasMany relationships to the other models.

With the current version, pushMany had the same performance as iterating over push but I'll try the branch with the optimization you provided. Thanks!

[igorT]

Just updated the branch. If you have multiple types, could you try pushPayload? Also there is upcoming perf work on the references branch that is going to make this much faster as well. The current issue is that we need to deal with all the foreign keys, so we need to coalesce all the pushes together in a runloop. You could also try wrapping all the pushes in a store._adapterRun call

[jslippy]

The pushMany_optimization branch seemed to do the trick. The delays are much more reasonable now. Pushing ~5000 records only took a little over 1 second in my application. Thank you.

[igorT]

How about the store._adapterRun?

[igorT]

Also, fyi I think 1 second is too slow, the references branch might be able to make this much faster. I'll ping you for a test run pretty soon

[jslippy]

That would be great. I agree 1 second is less than ideal but it is much better than the multiple minutes I was seeing before.

On May 7, 2015, at 3:40 PM, Igor Terzic [email protected] wrote:

Also, fyi I think 1 second is too slow, the references branch might be able to make this much faster. I'll ping you for a test run pretty soon

—
Reply to this email directly or view it on GitHub #3052 (comment).

[jslippy]

store._adapterRun worked too--maybe a little slower. For the same use case it was about 1.3 seconds but I didn't test it extensively.

[jslippy]

Do you know when we can expect to see the next release incorporating these fixes?

[rtm]

This was closed because....
Any update?

[greyhwndz]

@igorT: Any update regarding this?

[rtm]

In our case we have about 5000 records taking 10 seconds to load on a fast desktop. On an slower Android device, it takes 3 minutes. This is Ember Data beta 18.

I have done an unreasonable amount of sleuthing to track down this problem but it's like pinning jello to the wall. As far as I can tell, the problem is related to one hasMany/belongsTo relationship in one model. When removed, the time per record drops by a factor of five or so. Of course, then the app doesn't work. I can only imagine that there is some kind of resolution issue which is exponential or polynomial vis-a-vis the number of records.

Since I ran out of simple ideas to solve this problem, and could not ship the app as it was, I resorted to a chunking strategy where I load records into the store in chunks of half a dozen, and waiting for another tick to implement the next chunk. I implemented this as an adapter mixin, together with a custom adapter using it for the primary record type. This was complex because the payload contains multiple record types, and after I identify a chunk of primary records to load into the store, I have to then identify all the records of other types which must be present for the entire chunk to be well-formed and resolvable.

This worked well as far as it went, but I then ran into the problem that since I was displaying filtered and sorted sets of records on various pages, the array of records to be displayed was recalculated each time another chunk was loaded into the store, resulting in the entire page re-rendering again and again. I suppose Glimmer might help here, but I live in the real world and this is a large app and I need to solve this problem before I can take the time to upgrade to 1.13. My solution was another form of chunking, where the display data was divided into chunks structured in such a way as to not trigger re-rendering, and displayed with a special {{chunked-each}} component.

The ajax chunking revealed the interesting fact that load time is related to the number of records already in the store. For instance, an initial chunk of five primary records and another twenty sideloaded records took a few dozen milliseconds. However, by the time we got to chunk 100, each chunk of five was taking up to one second. There is clearly some extremely unoptimized code lurking somewhere; I just wish I could figure out where it was.

There is some possibility that the problem is not related to Ember Data per se but rather to the way Ember handles relationships. I tried an experiment where I managed the two-way relationships myself (adding an observer to add the relationship on the hasMany side) and also experienced roughly equivalent performance. By the way, some of these hasMany relationships number in the hundreds. So it's conceivable the problem is related to Ember Data being slow adding records where the record being added is one side of a large hasMany relationship.