[Azure Search] Catalog2AzureSearch performance seems worse than Catalog2Lucene against packages with many versions

[scottbommarito]

Earlier today I ran a test on DEV by bulk updating 10,000 versions of a package with 10,000 versions.

The results looked roughly like this:

(Db2Catalog's cursor is off-screen, but it started running at 22:10.)

As you can see, Global finished in roughly 30 minutes and China finished in roughly 75.

Global Catalog2Registration took roughly 14 minutes to finish.
Global Catalog2AzureSearch took roughly 8-12 minutes to finish.

China Catalog2Registration took roughly 11 minutes to finish.
China Catalog2AzureSearch took roughly 36-38 minutes to finish!!!

In the past, Catalog2Lucene performed much better against these bulk update events, taking only 5-6 minutes on both regions. Additionally, the slowness of China Catalog2AzureSearch seems extremely suspicious.

Ideally, Catalog2AzureSearch should perform just as fast as Catalog2Lucene did.

[joelverhagen]

We haven't done any load testing to determine the scale we need for our Azure Search resource. I was hoping to let query volume dictate this.

That being said, we could easily increase the parallel catalog leaf downloads to match that behavior in Catalog2Lucene. Currently Catalog2AzureSearch is using 4 parallel workers for both catalog leaf downloads and document pushes. It would be easy to download the catalog leafs with a higher degree of parallelism.

Nice find!