[Package Renames 5] Transfer package popularity

[loic-sharma]

This updates auxiliary2azuresearch to push download changes whenever a popularity transfer changes, or, whenever a download change affects a popularity transfer.

Previous changes: #765, #766, #767, and #768

Config changes: https://nuget.visualstudio.com/NuGetMicrosoft/_git/NuGetDeployment/pullrequest/1402?_a=overview

Part of NuGet/NuGetGallery#7898

[zhhyu]

I am thinking that there is an edge case, but as I am not sure about the whole picture here, like how Azure Search actually works, so I just list it here:
If the original package is hard deleted, it seem that we will still keep it with the download counts 0, which will be fed to Azure Search, in my understanding. ("affectedPackages" will contain this deleted package Id.) Is this expected?

[joelverhagen]

If the original package is hard deleted, it seem that we will still keep it with the download counts 0, which will be fed to Azure Search, in my understanding. ("affectedPackages" will contain this deleted package Id.) Is this expected?

Great thing to be thinking about. Given renames are on the package registration level, deletes would only cascade to the rename record if we release the package registration, I think. Given this option is unchecked by default and only available if all versions are deleted, I think it would very rarely occur. This begs the question, if all versions are deleted, should we clear the rename?

[loic-sharma]

If the original package is hard deleted, it seem that we will still keep it with the download counts 0....

Yup, you are correct that this will produce a transfer of 0. This behavior is covered by the DownloadTransferrerFacts.UnknownPackagesTransferZeroDownloads test.

we will still keep it with the download counts 0, which will be fed to Azure Search

The search pipeline has a "hidden" V3 resource called the version list, which is used by catalog2azuresearch and auxiliary2azuresearch to know which search documents should be updated. Using this resource, we detect and then skip updates to packages that do not exist. This behavior is covered by SearchIndexActionBuilderFacts.UpdatesSearchDocumentsWithNoVersions.

This begs the question, if all versions are deleted, should we clear the rename?

The popularity transfer feature does little for packages with low download counts. I doubt we will need to clean "dangling" renames on "small" packages. Now say that we delete all versions of a popular package but keep its registration... This seems like a truly exceptional scenario, which is hard to anticipate. We may want to evaluate keeping/removing the popularity transfers on a case-by-case basis.

[joelverhagen]

Thanks for the added tests. Looks good.