Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

179 focused maven visitor #192

Merged
merged 19 commits into from
Oct 18, 2023
Merged

179 focused maven visitor #192

merged 19 commits into from
Oct 18, 2023

Conversation

JonoYang
Copy link
Member

@JonoYang JonoYang commented Oct 4, 2023

This PR includes:

  • Updates to the packagedb migrations
  • New functions in minecode/visitor/maven.py to scrape information from a maven repo html page
  • New ImportQueue for crawling maven to create minimal packages (purl fields + download url + sha1)
  • Fix to the filter_by_checksums api endpoint where all of the packages were being returned

    * This is to avoid using get_maven_root repeatedly
    * Save versionless purl to importable_uris

Signed-off-by: Jono Yang <[email protected]>
Signed-off-by: Jono Yang <[email protected]>
    * Get links and timestamps at the same time
    * Create command that gets release_date for maven packages

Signed-off-by: Jono Yang <[email protected]>
    * Add logging messages

Signed-off-by: Jono Yang <[email protected]>
    * Only update release_date for packages from maven.org

Signed-off-by: Jono Yang <[email protected]>
@JonoYang JonoYang merged commit ad37ccc into main Oct 18, 2023
9 checks passed
@JonoYang JonoYang deleted the 179-focused-maven-visitor branch October 25, 2023 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant