Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RepositorySimulator: add support for delegated hash bins #1639

Closed
sechkova opened this issue Oct 27, 2021 · 5 comments · Fixed by #1808
Closed

RepositorySimulator: add support for delegated hash bins #1639

sechkova opened this issue Oct 27, 2021 · 5 comments · Fixed by #1808
Assignees
Labels
backlog Issues to address with priority for current development goals testing
Milestone

Comments

@sechkova
Copy link
Contributor

Description of issue or feature request:

Currently repository_tool.py provides an easy way to distribute a large number of target files over multiple delegated roles (hashed bins). A feature described in PEP458 that would be used in warehouse.

def delegate_hashed_bins(self, list_of_targets, keys_of_hashed_bins,

Having such functionality in RepositorySimulator that uses metadata api will be useful for testing the newer code.

@lukpueh
Copy link
Member

lukpueh commented Oct 27, 2021

IMO this feature should not be implemented in test code but rather be part of repository tooling (#1136).

@jku
Copy link
Member

jku commented Oct 27, 2021

IMO this feature should not be implemented in test code but rather be part of repository tooling (#1136).

absolutely true as the mid/long term goal but it might still make sense to implement in RepositorySimulator first:

  • no need to wait for the library module to be able to test ngclient
  • gives us data about the complexity and what a good API (implemented on top of Metadata API) might look like

@sechkova sechkova added the backlog Issues to address with priority for current development goals label Oct 27, 2021
@joshuagl
Copy link
Member

joshuagl commented Oct 27, 2021

The hashed bin delegation code in repository_tool was cleaned up and simplified when the Warehouse work kicked off. See, for example, #1007 and #1012. Specifically:

As a follow-on we refactored some of the helper functions to be public API (see #1058) in repository_lib for use with an early iteration of the metadata API:

@sechkova sechkova added this to the Sprint 14 milestone Dec 8, 2021
@ivanayov
Copy link
Collaborator

ivanayov commented Dec 8, 2021

Can I take this?

@sechkova
Copy link
Contributor Author

sechkova commented Dec 8, 2021

Seems like the latest tendency is towards implementing the delegated hash bins functionality in a repository-side and test code at a later stage. For this reason the issue can be re-scoped to include only the needed tests from a client point of view.

Let's use the newly added delegated hash bins example as a reference #1700.

An Updater test needs to include:

  • on the repository side:
    • creating one (or a couple of) delegated role(s). Naming does not matter for the client but in order to provide a good example, the rolename should represent the range of hash prefixes it delegates.
    • setting DelegatedRole.path_hash_prefixes to the list of path hash prefixes the role delegates.
    • adding a target file by choosing a target path whose hash digest prefix fits in the delegated role range.

Following the chosen prefix length and number of bins in the example:

rolename = "f8-ff"
path_hash_prefixes = ["f8", "f9", "fa", "fb", "fc", "fd", "fe", "ff"]
`test_hashed_bins.ext` -> fff0838676c1dacb8827b02511e89f85b43814a74dd2c3ffb31d952a1db94311
  • on the client side:
    • perform the usual search for target file metadata Updater.get_targetinfo("test_hashed_bins.ext")
    • verify that the target info was successfully found and the correct delegated metadata file was downloaded
    • add any other error cases that can come to mind

Currently two PRs add tests for delegated roles and it would be nice if the hashed bins test is included after them in test_delegation_graphs.py #1689, #1711

@sechkova sechkova mentioned this issue Dec 15, 2021
3 tasks
@jku jku modified the milestones: Sprint 14, Sprint 15 Jan 12, 2022
@jku jku modified the milestones: Sprint 15, sprint 16 Jan 26, 2022
@jku jku modified the milestones: sprint 16, sprint 17 Feb 9, 2022
@ivanayov ivanayov modified the milestones: sprint 17, sprint 18 Feb 23, 2022
@jku jku modified the milestones: sprint 18, sprint 19 Mar 9, 2022
@jku jku modified the milestones: sprint 19, Sprint 20 Mar 23, 2022
@jku jku closed this as completed in #1808 Apr 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backlog Issues to address with priority for current development goals testing
Projects
None yet
5 participants