Skip to content
This repository has been archived by the owner on Mar 16, 2021. It is now read-only.

[Package Renames 4] Refactor download overrides #768

Merged
merged 1 commit into from
Apr 20, 2020

Conversation

loic-sharma
Copy link
Contributor

@loic-sharma loic-sharma commented Apr 14, 2020

This change introduces a new type, IDownloadTransferrer, which determines what downloads should be changed to reflect the latest download overrides. In the future, this type will also apply popularity transfers.

Previous changes: #765, #766, and #767.
Part of NuGet/NuGetGallery#7898

Background

Originally, auxiliary2azuresearch updated downloads with these steps:

  1. Load the previously indexed downloads
  2. Load the latest downloads
  3. Compare the previously indexed downloads and latest downloads
  4. Push the download changes to search

Later we added download overrides. The steps became:

  1. Load the previously indexed downloads
  2. Load the latest downloads
  3. Apply the download overrides to the latest downloads
  4. Compare the previously indexed downloads and latest downloads
  5. Push the download changes to search

Now we want to add popularity transfers. These popularity transfers are affected by download changes and can be overriden by download overrides. The new steps will be:

  1. Load the previously indexed downloads
  2. Load the latest downloads
  3. Compare the previously indexed downloads and latest downloads
  4. Apply popularity transfers to download changes. This uses download changes to determine which popularity transfers need to be recalculated
  5. Apply download overrides to download changes
  6. Push the download changes to search

@loic-sharma loic-sharma force-pushed the loshar-popxfer-database branch from 18ac316 to 028849b Compare April 14, 2020 18:29
@loic-sharma loic-sharma force-pushed the loshar-popxfer-xferrer branch from 6bbc513 to 010687c Compare April 14, 2020 19:03
@loic-sharma loic-sharma changed the base branch from loshar-popxfer-database to dev April 14, 2020 19:04
@loic-sharma loic-sharma force-pushed the loshar-popxfer-xferrer branch from 010687c to 432dd7c Compare April 14, 2020 19:05
return await GetTransferChangesAsync(
downloads,
outgoingTransfers);
}
Copy link
Contributor Author

@loic-sharma loic-sharma Apr 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InitializeDownloadTransfers and UpdateDownloadTransfers are very similar right now, but they will diverge in a subsequent change.

Task<DownloadTransferResult> GetUpdatedTransferChangesAsync(
DownloadData downloads,
SortedDictionary<string, long> downloadChanges,
SortedDictionary<string, SortedSet<string>> oldTransfers);
Copy link
Contributor Author

@loic-sharma loic-sharma Apr 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentionally not using IReadOnlyX interfaces as this method needs to check the .Comparer property.

_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}

public async Task<DownloadTransferResult> GetTransferChangesAsync(DownloadData downloads)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about the name InitializeTransferChangesAsync? Seems like this should only be called in a case like Db2AzureSearch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed this to InitializeDownloadTransfers and renamed the other method to UpdateDownloadTransfers

Copy link
Member

@joelverhagen joelverhagen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@@ -195,6 +201,12 @@ private async Task WriteVerifiedPackagesDataAsync(HashSet<string> verifiedPackag
_logger.LogInformation("Done uploading the initial verified packages data file.");
}

private async Task WritePopularityTransfersDataAsync(SortedDictionary<string, SortedSet<string>> popularityTransfers)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest moving this to a common place and call this from auxiliary2azuresearch TODO above... the implementation can be TODO, one less thing to worry about.

Copy link
Contributor Author

@loic-sharma loic-sharma Apr 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand this comment as this code path is separate from auxiliary2azuresearch.

In any case, I removed this method since having an empty method is weird though

Update tests

Update comments

Update comments

Update Db2AzureSearchCommandFacts

Update new package registration producer facts

Add test

Add tests for download transferrer

Rename method

Fix test

Don't save popularity transfers for now

Work

Fix

Fix

Clean
@loic-sharma loic-sharma force-pushed the loshar-popxfer-xferrer branch from 653afdb to 49cac2c Compare April 20, 2020 18:01

public SortedDictionary<string, long> InitializeDownloadTransfers(
DownloadData downloads,
SortedDictionary<string, SortedSet<string>> outgoingTransfers,
Copy link
Contributor Author

@loic-sharma loic-sharma Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing a guard. I added this in the subsequent PR: https://github.com/NuGet/NuGet.Services.Metadata/pull/769/files

(Porting the changes from the next PR back into this one is a little work intensive, hence why I punted it to the next PR)

DownloadData downloads,
SortedDictionary<string, long> downloadChanges,
SortedDictionary<string, SortedSet<string>> oldTransfers,
SortedDictionary<string, SortedSet<string>> newTransfers,
Copy link
Contributor Author

@loic-sharma loic-sharma Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing a guard. I added this in the subsequent PR: https://github.com/NuGet/NuGet.Services.Metadata/pull/769/files

Copy link
Member

@joelverhagen joelverhagen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks better. Ship it.

@loic-sharma loic-sharma merged commit 5fe1ba7 into dev Apr 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants