This repository has been archived by the owner on Jul 30, 2024. It is now read-only.
[GH Idx] Save blob to azurestorage #785
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-Authored-By: Loïc Sharma <[email protected]>
Co-Authored-By: Loïc Sharma <[email protected]>
mogah
requested review from
joelverhagen,
agr,
loic-sharma,
xavierdecoster,
shishirx34,
ryuyu,
cristinamanum,
skofman1,
dannyjdev,
scottbommarito and
zhhyu
loic-sharma
reviewed
Jul 29, 2019
loic-sharma
reviewed
Jul 29, 2019
scottbommarito
approved these changes
Jul 29, 2019
loic-sharma
approved these changes
Jul 29, 2019
joelverhagen
added a commit
that referenced
this pull request
Jul 31, 2020
Address NuGet/Engineering#3212 Only added retry for WebException codes that were clearly transient in logs
joelverhagen
pushed a commit
that referenced
this pull request
Oct 26, 2020
* [GH Index] Initial commit * [GH Index] Fixed build * Added License headers * Changed Nuspec Id * Changed Nuspec script include * Added empty job * [GH Idx] Added Octokit and LibGit2Sharp dependencies * [GH Idx] Add initial GHSearcher * [GH Idx] Add GitRepoSearcher * [GH Idx] Add dependency injection * [GH Idx] Add null check * [GH Idx] Add tests * [GH Idx] Extracted constants * [GH Idx] Fixed tests * [GH Idx] Add Filters class * Update src/NuGet.Jobs.GitHubIndexer/GitRepoSearchers/GitHubSearcher.cs Co-Authored-By: Loïc Sharma <[email protected]> * Update src/NuGet.Jobs.GitHubIndexer/GitRepoSearchers/GitHubSearcher.cs Co-Authored-By: Loïc Sharma <[email protected]> * [GH Idx] Removed duplicate class RepositoryInformation * [GH Idx] Refactored the code a bit * [GH Idx] Fix possible deadlock * [GH Idx] Add config section in the appsettings.json * [GH Idx] GitHubSearcher is not recursive anymore! * [GH Idx] Removed redundant comparer * [GH Idx] Fix upperStarBound wrongly set on request * [GH Idx] Fixed sleep time * [GH Idx] Fix typo * [GH Idx] Made fields private * [GH Idx] Changed UA * [GH Idx] Made the configuration not static * [GH Idx] Add ApiInfo doc in the tests * [GH Idx] Refactor GH Search API requester * [GH Idx] Removed redundant import in csproj * [GH Idx] Add documentation to the configuration * [GH Idx] Move the IGitHubClient to the GitHubSearchWrapper * [GH Idx] Remove redundant variable * [GH Idx] Trim tests Assembly info * [GH Idx] Add checks to ensure the required info is in the GitHub response * [GH Idx] Moved public method before private methods * [GH Idx] Extract retry time in a static variable * [GH Idx] Add typecheck and fix tests * [GH Idx] Remove redundant using * [GH Idx] Nit space formatting * [GH Idx] Change UserAgent to use assembly name and version * [GH Idx] Remove extra line * [GH Idx] Fix nit picks * [GH Idx] Fix merge * [GH Idx] First iteration of the filtering * [GH Idx] Simplified Job class * [GH Idx] WIP * [GH Idx] Process repo is now run in parallel * [GH Idx] Removed debug code * [GH Idx] WIP 2 * [GH Idx] Modify Filters doc * [GH Idx] Refactor WritableRepositoryInformation * [GH Idx] WIP 3 * [GH Idx] Add WritableRepoInfo doc * [GH Idx] Made the MaxDegreeOfParallelism configurable * [GH Idx] WIP before tests The pipeline is working, now I need to refactor it to make it testable by adding a bunch of interfaces to decouple with LibGit2Sharp * [GH Idx] Refactor to decouple from LibGit2Sharp * [GH Idx] Using immutable collections * [GH Idx] Add tests * [GH Idx] Clean old code * [GH Idx] Remove unused imports * [GH Idx] Add logging * [GH Idx] Cleanup * [GH Idx] Bumping up the NuGetGalery.Core dependency version * [GH Idx] Reverting changes to web.config * [GH Idx] Add docs * Cleaned up dependencies and bumped up NuGetGalery.Core version * [GH Idx] Fix PascalCase method name * [GH Idx] Fix space * [GH Idx] Remove redundant comment * [GH Idx] Now using proper logger creation * [GH Idx] Add new line for constructor * [GH Idx] Remove redundant filter config file type * [GH Idx] Add RegEx timeout * [GH Idx] Remove empty line * [GH Idx] Add named params and remove redundant code for FetchedRepo * [GH Idx] Add basePathLength to optimize Select * [GH Idx] Move the static constructor * [GH Idx] Cache hit now logged as an information * [GH Idx] Use Path.Combine instead of string concatenation * [GH Idx] Remove redundant comment * [GH Idx] Extract GitFileInfo class * [GH Idx] Remove redundant imports * [GH Idx] Replace "as" cast * [GH Idx] Simplify LINQ statement * [GH Idx] Simplify config file parsing * [GH Idx] Simplify Thread construction * [GH Idx] Move cache files to their own directory * [GH Idx] Remove transitive exception throws in documentation * [GH Idx] Wrap long line in Filters * [GH Idx] Make dependencies case-insensitive * [GH Idx] Use Path.Combine and remove extra line * [GH Idx] Add named param * Update src/NuGet.Jobs.GitHubIndexer/CheckedOutFile.cs Co-Authored-By: Loïc Sharma <[email protected]> * [GH Idx] Move config in same section * [GH Idx] Remove redundant documentation * [GH Idx] Check for unhandled config file type * [GH Idx] Rename function * [GH Idx] Using Path.Combine in RepoUtils * [GH Idx] Move isValidPackageId to RepoUtils * [GH Idx] Optimal LINQ usage in ReposIndexer * [GH Idx] Move TODO * [GH Idx] Log Trace and Debug --> Information * [GH IDx] Expanded msBuild and PkgConfig enums * [GH Idx] Remove special regex case * [GH Idx] Using stringComparer instead of ToLower() then comparing * [GH Idx] Use repo.FullName instead of manually creating it * [GH Idx] Filters early return * [GH Idx] Log warning for long paths * [GH Idx] Run --> RunAsync * [GH Idx] Remove ServicePointManager init setup * [GH Idx] workdir --> work * [GH Idx] Remove as cast * [GH Idx] LogTrace --> LogInformation for disk cache * [GH Idx] Save final blob to Azure Storage * [GH Idx] Forgot a LogTrace there... * Update src/NuGet.Jobs.GitHubIndexer/ConfigFileParser.cs Co-Authored-By: Loïc Sharma <[email protected]> * [GH Idx] Update FetchedRepo comment * [GH Idx] Update EndsWith --> Equals * [GH Idx] Fix wrong documentation * [GH Idx] Revert Config properties for Azure BlobStorage * [GH Idx] Simplify LINQ statement * [GH Idx] Format RepoUtils line to make it more readable * [GH Idx] Got rid of few Singletons * [GH Idx] Scope logging * [GH Idx] "No Description." --> "" * [GH Idx] Fix config * Update src/NuGet.Jobs.GitHubIndexer/FetchedRepo.cs Co-Authored-By: Loïc Sharma <[email protected]> * Update src/NuGet.Jobs.GitHubIndexer/ReposIndexer.cs Co-Authored-By: Loïc Sharma <[email protected]> * [GH Idx] Inverted if stattement in TryGetCachedVersion * [GH Idx] Fix timing to use UTC * [GH Idx] Fix timing to use UTC * [GH Idx] Move assignment * [GH Idx][ Extract container name to constant * [GH Idx] Move serializer * [GH Idx] Function rename * [GH Idx] Add tests to make sure blob is serialized correctly
joelverhagen
added a commit
that referenced
this pull request
Oct 26, 2020
Address NuGet/Engineering#3212 Only added retry for WebException codes that were clearly transient in logs
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the ability for the GitHubIndexer job to upload its final blob to AzureBlobStorage