Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration PR not align with original dependabot config variables #3762

Closed
jasonycw opened this issue May 21, 2021 · 11 comments
Closed

Migration PR not align with original dependabot config variables #3762

jasonycw opened this issue May 21, 2021 · 11 comments
Assignees
Labels
E: dependabot-preview Issues specific to Dependabot Preview F: preview-migration Issue relates to migrating from Dependabot Preview

Comments

@jasonycw
Copy link

Basic info:

Package ecosystem
nuget
Language version
C#
Manifest location and content prior to update
n/a
Updated dependency
n/a
Native package manager behavior
n/a

In our dependabot setup, we have multiple nuget feeds setup so that we can update both the public nuget packages from https://api.nuget.org/v3/index.json and the private nuget packages hosted on github package.

However, I tried update a few of our repo to native dependabot with the migration PR and seems like the migration PR doesn't account for multiple nuget feeds. The result .github/dependabot.yml only contain the the github package link which breaks the package update process.

This migration PR is not ready for one click migration like #3591 (comment) said

Previous behavior in Dependabot Preview:

image

Update logs on app.dependabot.com

  proxy | 2021/05/21 01:14:07 * authenticating nuget feed request (host: api.nuget.org, bearer auth)
  proxy | 2021/05/21 01:14:07 * authenticating nuget feed request (host: nuget.organization.net, bearer auth)
  proxy | 2021/05/21 01:14:08 * authenticating nuget feed request (host: nuget.pkg.github.com, bearer auth)
  proxy | 2021/05/21 01:14:08 Listening (:1080)

Current behavior in GitHub-native Dependabot:

image

Update logs in repo's Dependency graph > Dependabot

  proxy | time="2021-05-21T01:08:21Z" level=info msg="proxy starting" commit=asdfdsf20f0c7297a2340dd3181976c0c2b20872cd
  proxy | 2021/05/21 01:08:21 * authenticating nuget feed request (host: nuget.pkg.github.com, bearer auth)
  proxy | 2021/05/21 01:08:21 Listening (:1080)
version: 2
registries:
  nuget-feed-nuget-pkg-github-com-organizationname-index-json:
    type: nuget-feed
    url: https://nuget.pkg.github.com/organizationname/index.json
    token: "${{secrets.NUGET_FEED_NUGET_PKG_GITHUB_COM_ORGANIZATIONNAME_INDEX_JSON_TOKEN}}"

updates:
- package-ecosystem: nuget
  directory: "/"
  schedule:
    interval: daily
    time: "09:00"
    timezone: Hongkong
  open-pull-requests-limit: 99
  reviewers:
  - someone
  - jasonycw
  assignees:
  - someone
  registries:
  - nuget-feed-nuget-pkg-github-com-organizationname-index-json
@jasonycw jasonycw changed the title Migration PR not align with original dependabot config Migration PR not align with original dependabot config variables May 21, 2021
@jurre
Copy link
Member

jurre commented May 21, 2021

In our dependabot setup, we have multiple nuget feeds setup so that we can update both the public nuget packages from https://api.nuget.org/v3/index.json and the private nuget packages hosted on github package.

You do not need to configure the public registry, it was a noop in preview and it is stripped out for the native version.

breaks the package update process

Have you configured the secrets as mentioned here?

@jasonycw
Copy link
Author

Have you configured the secrets as mentioned here?

Yes, I have followed the PR and added the secret to the repo(some private repo not in organization) and organization level.

But in the update log, there is nothing related to api.nuget.org is shown, only the github package feed

@jasonycw
Copy link
Author

Here are more update logs from the GitHub native dependabot showing package not being checked from public nuget feed

updater | INFO <job_137293849> Checking if AWSSDK.S3 3.7.0.25 needs updating
updater | INFO <job_137293849> Ignored versions:
updater | INFO <job_137293849>   3.5.7.11 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.5.7.12 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.5.8 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.5.8.3 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.5.8.6 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.5.8.8 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.5.9 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.5.9.2 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.5.9.5 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.7.0.10 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.7.0.12 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.7.0.16 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.7.0.2 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.7.0.3 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.7.0.4 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.7.0.7 - from .github/dependabot.yml
updater | INFO <job_137293849>   3.7.0.9 - from .github/dependabot.yml
  proxy | 2021/05/24 01:08:23 [078] GET https://nuget.pkg.github.com:443/organizationname/index.json
  proxy | 2021/05/24 01:08:23 [078] * authenticating nuget feed request (host: nuget.pkg.github.com, bearer auth)
  proxy | 2021/05/24 01:08:23 [078] 200 https://nuget.pkg.github.com:443/organizationname/index.json
  proxy | 2021/05/24 01:08:23 [080] GET https://nuget.pkg.github.com:443/organizationname/query?q=awssdk.s3&prerelease=true
  proxy | 2021/05/24 01:08:23 [080] * authenticating nuget feed request (host: nuget.pkg.github.com, bearer auth)
  proxy | 2021/05/24 01:08:24 [080] 200 https://nuget.pkg.github.com:443/organizationname/query?q=awssdk.s3&prerelease=true
updater | INFO <job_137293849> Latest version is 
updater | INFO <job_137293849> Requirements to unlock update_not_possible
updater | INFO <job_137293849> Requirements update strategy 
updater | INFO <job_137293849> No update possible for AWSSDK.S3 3.7.0.25

Where AWSSDK.S3 should be updated to 3.7.0.27 3 days ago according to
https://www.nuget.org/packages/AWSSDK.S3/ and https://api-v2v3search-0.nuget.org/query?q=AWSSDK.S3&prerelease=true

And other repo which is not migrated to native dependabot yet, @dependabot-preview can still create PR correctly
Here is part of the logs from https://app.dependabot.com/

updater | INFO <job_80911760> Checking if AWSSDK.S3 3.7.0.25 needs updating
  proxy | 2021/05/24 01:15:22 [011] GET http://nuget.organization.net/index.json
  proxy | 2021/05/24 01:15:22 [011] 200 http://nuget.organization.net/index.json
  proxy | 2021/05/24 01:15:22 [013] GET https://nuget.pkg.github.com:443/organizationname/index.json
  proxy | 2021/05/24 01:15:22 [013] * authenticating nuget feed request (host: nuget.pkg.github.com, bearer auth)
  proxy | 2021/05/24 01:15:22 [013] 200 https://nuget.pkg.github.com:443/organizationname/index.json
  proxy | 2021/05/24 01:15:22 [014] GET http://nuget.organization.net/flatcontainer/awssdk.s3/index.json
  proxy | 2021/05/24 01:15:22 [014] 404 http://nuget.organization.net/flatcontainer/awssdk.s3/index.json
  proxy | 2021/05/24 01:15:22 [016] GET https://api-v2v3search-0.nuget.org:443/query?q=awssdk.s3&prerelease=true
  proxy | 2021/05/24 01:15:23 [016] 200 https://api-v2v3search-0.nuget.org:443/query?q=awssdk.s3&prerelease=true
  proxy | 2021/05/24 01:15:23 [018] GET https://nuget.pkg.github.com:443/organizationname/query?q=awssdk.s3&prerelease=true
  proxy | 2021/05/24 01:15:23 [018] * authenticating nuget feed request (host: nuget.pkg.github.com, bearer auth)
  proxy | 2021/05/24 01:15:23 [018] 200 https://nuget.pkg.github.com:443/organizationname/query?q=awssdk.s3&prerelease=true
updater | INFO <job_80911760> Latest version is 3.7.0.27
updater | INFO <job_80911760> Requirements to unlock own
updater | INFO <job_80911760> Requirements update strategy 
updater | INFO <job_80911760> Updating AWSSDK.S3 from 3.7.0.25 to 3.7.0.27
updater | INFO <job_80911760> Submitting AWSSDK.S3 pull request for creation

@brrygrdn brrygrdn added the E: dependabot-preview Issues specific to Dependabot Preview label May 24, 2021
@jasonycw
Copy link
Author

This maybe the same issue as #3724

If default public nuget feed should always be check at the first place, it's the same bug.
Or if there is a need to support multiple nuget feeds inside dependabot.yml?

@brrygrdn brrygrdn added the F: preview-migration Issue relates to migrating from Dependabot Preview label Jun 10, 2021
@jasonycw
Copy link
Author

jasonycw commented Jul 26, 2021

Hi @jurre and @brrygrdn, any update on this bug?

Dependabot Preview will be shut down on August 3rd, 2021.

It's only a week left before dependabot preview shut down but the native dependabot still not pulling public nuget feed properly when there are 3rd party nuget feed.

Are we going to be stuck with the bugged native dependabot or no dependabot after August 3rd?

@brrygrdn
Copy link
Contributor

Sorry for the delay in replying @jasonycw - to elaborate on my comment in #3724, I believe the correct way to configure this is to add a nuget.config file to your project root directory as @feelepxyz mentioned:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
        <add key="NuGet official package source" value="https://api.nuget.org/v3/index.json" />
    </packageSources>
</configuration>

Dependabot builds a list of repositories to look up packages in the RepositoryFinder class which inserts https://api.nuget.org/v3/index.json if no other source is configured here.

In Dependabot Preview this is not a problem as looking at your existing configuration, you have a config variable that provides this URI:

image

I believe this workflow existed historically so that using a mix of public and private sources had to be explicitly configured, which core 'opting out' of the public repo once any other source was provided. By explicitly indicating the official public source in your nuget.config you will be providing Native Dependabot with instruction to use it.

Let me know if this helps or if you have any further issues.

@jasonycw
Copy link
Author

jasonycw commented Jul 28, 2021

What about if dependabot.yml can support multiple feeds?

version: 2
registries:
  nuget-feed-nuget-pkg-github-com-organizationname-index-json:
    type: nuget-feed
    url: https://nuget.pkg.github.com/organizationname/index.json
    token: "${{secrets.NUGET_FEED_NUGET_PKG_GITHUB_COM_ORGANIZATIONNAME_INDEX_JSON_TOKEN}}"
  public-nuget:
    type: nuget-feed
    url: https://api.nuget.org/v3/index.json

updates:
- package-ecosystem: nuget
  directory: "/"
  schedule:
    interval: daily
    time: "09:00"
    timezone: Hongkong
  open-pull-requests-limit: 99
  reviewers:
  - someone
  - jasonycw
  assignees:
  - someone
  registries:
  - nuget-feed-nuget-pkg-github-com-organizationname-index-json
  - public-nuget

Currently, native dependabot doesn't support this, it will say
image

Since the purpose of the migration PR is to smooth the transition process, we were not expecting the migration will need to add a nuget.config just for the public feed, which may break some of the CI/CD flow

@brrygrdn
Copy link
Contributor

Thanks for the feedback @jasonycw, we're taking a look at this change internally to see what is practical.

@michakinchen1988
Copy link

michakinchen1988 commented Jul 28, 2021 via email

@brrygrdn
Copy link
Contributor

👋🏻 @jasonycw I've deployed a change that permits you to add the public feed explicitly to the config file, e.g.

version: 2
registries:
  nuget-feed-nuget-pkg-github-com-organizationname-index-json:
    type: nuget-feed
    url: https://nuget.pkg.github.com/organizationname/index.json
    token: "${{secrets.NUGET_FEED_NUGET_PKG_GITHUB_COM_ORGANIZATIONNAME_INDEX_JSON_TOKEN}}"
  public:
    type: nuget-feed
    url: https://api.nuget.org/v3/index.json

updates:
- package-ecosystem: nuget
  directory: "/"
  schedule:
    interval: daily
    time: "09:00"
    timezone: Hongkong
  open-pull-requests-limit: 99
  reviewers:
  - someone
  - jasonycw
  assignees:
  - someone
  registries:
  - nuget-feed-nuget-pkg-github-com-organizationname-index-json
  - public

Hopefully that helps resolve your issue.

@jasonycw
Copy link
Author

Thanks, just tested with a few of our repositories and dependabot can finally query from both feeds

updater | INFO <job_176046954> Checking if AWSSDK.S3 3.7.0.25 needs updating
  proxy | 2021/07/30 01:52:04 [182] GET https://nuget.organization.net/index.json
  proxy | 2021/07/30 01:52:04 [182] * authenticating nuget feed request (host: nuget.pkg.github.com, bearer auth)
  proxy | 2021/07/30 01:52:04 [182] 200 https://nuget.organization.net/index.json
  proxy | 2021/07/30 01:52:04 [184] GET https://nuget.organization.net/query?q=awssdk.s3&prerelease=true
  proxy | 2021/07/30 01:52:04 [184] * authenticating nuget feed request (host: nuget.pkg.github.com, bearer auth)
  proxy | 2021/07/30 01:52:05 [184] 200 https://nuget.organization.net/query?q=awssdk.s3&prerelease=true
  proxy | 2021/07/30 01:52:05 [186] GET https://api-v2v3search-0.nuget.org:443/query?q=awssdk.s3&prerelease=true
  proxy | 2021/07/30 01:52:05 [186] 200 https://api-v2v3search-0.nuget.org:443/query?q=awssdk.s3&prerelease=true
updater | INFO <job_176046954> Latest version is 3.7.1.18
updater | INFO <job_176046954> Requirements to unlock own
updater | INFO <job_176046954> Requirements update strategy 
updater | INFO <job_176046954> Updating AWSSDK.S3 from 3.7.0.25 to 3.7.1.18
  proxy | 2021/07/30 01:52:05 [188] GET https://api.github.com:443/repos/organization/repo/commits?per_page=100
  proxy | 2021/07/30 01:52:05 [188] * authenticating github api request
  proxy | 2021/07/30 01:52:05 [188] 200 https://api.github.com:443/repos/organization/repo/commits?per_page=100
  proxy | 2021/07/30 01:52:05 [190] GET https://api.nuget.org:443/v3-flatcontainer/awssdk.s3/3.7.1.18/awssdk.s3.nuspec
  proxy | 2021/07/30 01:52:05 [190] 200 https://api.nuget.org:443/v3-flatcontainer/awssdk.s3/3.7.1.18/awssdk.s3.nuspec
  proxy | 2021/07/30 01:52:05 [192] GET https://api.github.com:443/repos/aws/aws-sdk-net/releases?per_page=100
  proxy | 2021/07/30 01:52:05 [192] * authenticating github api request
  proxy | 2021/07/30 01:52:05 [192] 200 https://api.github.com:443/repos/aws/aws-sdk-net/releases?per_page=100
  proxy | 2021/07/30 01:52:05 [194] GET https://api.github.com:443/repos/aws/aws-sdk-net/contents/
  proxy | 2021/07/30 01:52:05 [194] * authenticating github api request
  proxy | 2021/07/30 01:52:05 [194] 200 https://api.github.com:443/repos/aws/aws-sdk-net/contents/
  proxy | 2021/07/30 01:52:05 [196] GET https://api.github.com:443/repos/aws/aws-sdk-net/contents/SDK.CHANGELOG.md?ref=master
  proxy | 2021/07/30 01:52:05 [196] * authenticating github api request
  proxy | 2021/07/30 01:52:06 [196] 200 https://api.github.com:443/repos/aws/aws-sdk-net/contents/SDK.CHANGELOG.md?ref=master
  proxy | 2021/07/30 01:52:06 [198] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:06 [198] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:06 [198] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:06 [200] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:06 [200] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:06 [200] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:06 [202] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:06 [202] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:06 [202] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:06 [204] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:06 [204] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:07 [204] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:07 [206] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:07 [206] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:07 [206] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:07 [208] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:07 [208] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:07 [208] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:08 [210] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:08 [210] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:08 [210] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:08 [212] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:08 [212] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:08 [212] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:08 [214] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:08 [214] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:08 [214] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:08 [216] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:08 [216] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:08 [216] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:09 [218] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:09 [218] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:09 [218] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:09 [220] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:09 [220] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:09 [220] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:09 [222] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:09 [222] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:10 [222] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:10 [224] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:10 [224] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:10 [224] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:10 [226] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:10 [226] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:10 [226] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:10 [228] GET https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
  proxy | 2021/07/30 01:52:10 [228] * authenticating git server request (host: github.com)
  proxy | 2021/07/30 01:52:11 [228] 200 https://github.com:443/aws/aws-sdk-net.git/info/refs?service=git-upload-pack
updater | INFO <job_176046954> Submitting AWSSDK.S3 pull request for creation

It will be great if the migration PR be updated as well, but even if it doesn't, I'll merge them and update all the dependabot.yml manually one by one.

Thanks again!

@jasonycw jasonycw closed this as completed Aug 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
E: dependabot-preview Issues specific to Dependabot Preview F: preview-migration Issue relates to migrating from Dependabot Preview
Projects
None yet
Development

No branches or pull requests

4 participants