Massive performance decrease on Migrate

[JakobFerdinand]

Performance decrease on migrate

We upgraded our project from EF 2.2.6 to EF 7.0.
Now we notice a massive performance decrease on dbContext.Database.Migrate().
We built a feature around that method where we migrate old databases during runtime.
We have a very big model with about 400 entity types.

Several teams are creating migrations independently -> we create about 250 migrations a year.
For maintenance we aggregate the migrations on regular bases.
Since we did that just after the migration the following measurement ist with only 10 migrations.

In the profiling screenshot you can see, that the creation of the relational model takes most of the time.

Include provider and version information

EF Core version: 7.0.0
Database provider: Microsoft.EntityFrameworkCore.SqlServer
Target framework: .NET 7.0
Operating system: Windows 11
IDE: Visual Studio 2022 17.4

[roji]

Possible dup of #29642, which has been fixed for 7.0.3. Can you drill down and verify which method is taking up the time under AddDefaultMappings?

[JakobFerdinand]

[roji]

Can you please try with the latest daily build of 8.0, where #29642 has been fixed?

[JakobFerdinand]

I will try it in the next days. Had no time to do it today.

[JakobFerdinand]

With the latest version of the release/7.0 branch it only takes 1.8 seconds instead of 6 seconds.

[roji]

@JakobFerdinand how much was it taking with 2.2? Can you also try with the latest 6.0 version to exclude some regression in 7.0?

[JakobFerdinand]

In EF Core 2.2.6 the FinalizeModel method does not exist.

EF Core 2.2.6

I´m sorry but I can´t do the regression test with 6.0.
We use our own fork of EF Core where we added some features. We made the jump from 2.2.6 to 7.0 directly, so I don´t have a version of 6.0 that is compatible with our solution.
Creating that version and integrating it into our code would be a massive effort.

[roji]

In EF Core 2.2.6 the FinalizeModel method does not exist.

My question was how long the whole process takes on 2.2.6; since you reported that the latest 7.0 takes 1.8 seconds instead of 6 seconds, it's important to compare than against 2.2.6 (maybe there's no more regression).

[JakobFerdinand]

As you can see in the screenshots the times mentioned above is only the time consumed by FinalizeModel.
That method does not exist in 2.2.6.

Comparing the over all time is also not representative because of our migration merge after the change to 7.0.

One thing that can be compared is the method BuildTargetModel that is called on every migration.
In Ef 2.2.6 that took about ~230ms. (See latest screenshot)
In EF 7.0.0 (and also in 7.0.3), with almost the same model, it´s a bit slower. (See very first screenshot)

[roji]

In Ef 2.2.6 that took about ~230ms. (See latest screenshot)
In EF 7.0.0 (and also in 7.0.3), with almost the same model, it´s a bit slower. (See very first screenshot)

I'm confused - you're writing here that 7.0.0/7.0.3 are only a bit slower than 2.2.6 (~230ms), but also that the latest release/7.0 (i.e. 7.0.3) branch is significantly faster than 7.0.0 (1.8 seconds instead of 6 seconds). Not saying there isn't a problem, but can you provide a bit of clarity here?

7.0.3 has bug fix #29642 which could be affecting perf in a massive way. What we ideally need is a one-to-one comparison between 7.0.3 and e.g. 2.2.6, to see where we actually stand.

[JakobFerdinand]

Yes I will try to provide some more clarity. :D

If we take a look at the Migrate() method we can see, that it calls the GenerateUpSql() method which calls two other methods which each has a significant time factor:

• FinalizeModel()
• get_TargetModel() -> calls BuildTargetModel() on each migration

Screenshot from 7.0.0

Let´s focus on FinalizeModel() first.

EF Version	Time consumed	Comment
2.2.6	0ms	It does not exist yet and there is no other method taking a long time.
7.0.0	~6000ms
7.0.2	~1800ms	with fixed #29642

The second part is get_TargetModel().
It´s not useful to compare the duration of that call by itself because it depends on the amount of migrations executed (BuildTargetModel()) inside.
But we can compare how long the execution of one migration takes.
I made new tests with the focus on only that method and it turns out there is no regression on BuildTargetModel().

EF Version	Time consumed	Comment
2.2.6	~230ms
7.0.0	~225ms

EF Core 2.2.6

EF Core 7.0.0

Important: The model in both screenshots is about the same. But in 7.0.0 all migrations have been merged together to save time.

Does that answer your questions? :)

[roji]

Does that answer your questions? :)

To a certain extent :)

Putting aside the perf issue that #29642 fixed, we don't learn much from focusing on FinalizeModel(), precisely because it didn't exist in 2.2; whatever FinalizeModel does in modern EF versions is probably spread out across other methods in 2.2 - it's most probably not the case that it's an added 1800ms in 7, compared to 2.2.

This is why I'm not looking for how much specific methods took in the profiler - it's generally not very useful to simply compare specific method profiler times across versions, since things changed overall. What's interesting is an apples-to-apples comparison of the entire operation, i.e. Migrate as a whole. If I understand correctly, you can't produce this since you've already merged the migrations in 7.0.0, so an apples-to-apples comparison isn't possible.

To summarize: are you saying that you think there's still a perf problem in 7.0, after #29642?

[JakobFerdinand]

We created a special version of our program to create a direct comparison.

In both cases the same 366 migrations were executed by creating a new empty database.

EF Version	Elapsed time
2.2.6	2min 22s (142881ms)
7.0.3 (after #29642)	2min 27s (147060ms)

As a summary I would say there is no regression from 2.2.6 to 7.0.3.

Can be closed.

[roji]

@JakobFerdinand thank you very much for doing the apples-to-apples comparison - it's good to see confirmation that there's no regression here. Please don't hesitate to write back if you encounter further issues.

[roji]

Duplicate of #29642