Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the Github-URL for fetching dependeny changelogs configurable #12042

Open
Tracked by #14138
dprilmueller opened this issue Oct 6, 2021 · 14 comments
Open
Tracked by #14138
Labels
core:changelogs Related to changelogs/release notes fetching priority-4-low Low priority, unlikely to be done unless it becomes important to more people type:feature Feature (new functionality)

Comments

@dprilmueller
Copy link

dprilmueller commented Oct 6, 2021

What would you like Renovate to be able to do?

I would like to make use of the Github.com changelog fetching feature in a gitlab-renovate-runner project.
The problem is, that direct access to the internet is not possible. Instead all requests have to be proxied via nexus.
To make this possible, a custom base-url for the changelog requests would be an option.

Current state when fetching changelog for dependency mapstruct for example:
Fetching changelog: https://github.com/mapstruct/mapstruct (1.4.0.Final -> 1.4.2.Final) (repository=***/rvs-node-dependency-upgrades-poc)
"url": "https://api.github.com/repos/mapstruct/mapstruct/tags?per_page=100",

The github.com part (and/or the api.github.com?) would be replaced with the proxy-url.

If you have any ideas on how this should be implemented, please tell us here.

Is this a feature you are interested in implementing yourself?

No

@dprilmueller dprilmueller added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:feature Feature (new functionality) labels Oct 6, 2021
@rarkins
Copy link
Collaborator

rarkins commented Oct 7, 2021

Can you show a before/after example of the URLs?

@dprilmueller
Copy link
Author

Before: https://github.com/mapstruct/mapstruct
After: http(s)://proxyurl.com/mapstruct/mapstruct

Of course the "mapstruct/mapstruct" part is just an example for a dependency on github.

@rarkins
Copy link
Collaborator

rarkins commented Oct 7, 2021

So there's a dedicated proxyurl host which only proxies GitHub.com? Or is there a path prefix not shown in your example?

@dprilmueller
Copy link
Author

"https://github.com" is the part of the URL that should be configurable.

Otherwise fetching changelogs is not possible due to network restrictions. In our case, our gitlab instance is not allowed to access external services like github. We have to use an internal proxy therefore. (We do the same for the npm-registry, in this case renovate supports custom registries which solves the problem).

I guess others have the same problem and need to set fetchReleaseNotes to false in this case, but thats no soluation.

@rarkins
Copy link
Collaborator

rarkins commented Oct 7, 2021

And you can't use a HTTPS_PROXY?

@HonkingGoose HonkingGoose added status:waiting-on-response and removed status:requirements Full requirements are not yet known, so implementation should not be started labels Oct 18, 2021
@thor
Copy link

thor commented Nov 10, 2021

To chime in, @rarkins: we have a similar use-case, except it's not Nexus we're using. In our case, perhaps similarly to @dprilmueller's, we cannot use a HTTPS_PROXY.

In our case the URLs look something like https://service.company.com/some/api/path/github/$GITHUB_ENDPOINT.

Alternatively, we'd need to look closer into "solving" the problem of missing HTTPS_PROXY for us to use.

@HonkingGoose HonkingGoose added status:requirements Full requirements are not yet known, so implementation should not be started and removed status:waiting-on-response labels Nov 11, 2021
@rarkins
Copy link
Collaborator

rarkins commented Nov 11, 2021

Does your alternative URL support the full API of GitHub.com? As in it's essentially a full transparent proxy?

@thor
Copy link

thor commented Nov 15, 2021

Yes and no: in terms of making simple reading-based requests, I'd say yes. In terms of altering any content or pushing any updates, I'm not sure: I've never tried to POST through it. When it comes to fetching, it does feel as though it is a fully transparent proxy. We've never had issues with any typically read-based method like LIST, GET or OPTIONS.

EDIT: We do have some API gateways too that can work as fully transparent proxies if need be. The service we use is Artifactory, which is similar to Nexus. We point all of our internal registryUrls to it for the different datasources we have available.

@rarkins
Copy link
Collaborator

rarkins commented Nov 15, 2021

You'd need token authentication for github.com too, otherwise you'd hit rate limiting very quickly. How is that handled?

@thor
Copy link

thor commented Nov 15, 2021

Artifactory handles that on its side; we configure Artifactory to use the specified token for communication with github.com. If multiple teams need access to the extent that we need multiple tokens, we could/would create some/api/path/new-team/github and allow users to point to that. Currently we only use the one some/api/path/github where we have a read token.

If it's a straightforward Bearer token (IIRC) it could be used to transparently authenticate to the API proxy endpoint that Artifactory provides, except it'd be a token for the specific Artifactory endpoint, rather than necessarily being for the GitHub API (as that would be handled inside Artifactory).

@rarkins
Copy link
Collaborator

rarkins commented Nov 16, 2021

We could potentially solve this with a new config option as part of hostRules, e.g. called replaceHost or similar.

Can someone give a more concrete example, for example what would the URL https://api.github.com/repos/mapstruct/mapstruct/tags?per_page=100 be replaced with?

And do you have a separate proxy endpoint for https://github.com/ too?

@dprilmueller
Copy link
Author

The URL would be replaced with something like:
https://proxy.company.com/some/repos/mapstruct/tags?per_page=100

Yes, if renovate also accesses https://github.com (not only api.github.com) we need a seperate proxy endpoint for that as well.

@rarkins
Copy link
Collaborator

rarkins commented Nov 16, 2021

So the new option could be called hostProxy and accepts a URL like https://githubapiproxy.company.com or https://proxy.company.com/githubapi/.

It would be somewhat similar to insecureRegistry in that it changes the URL (

if (opts?.insecureRegistry) {
registryHost = registryHost.replace('https', 'http');
}
) although should be implemented in lib/util/http.

Maybe also needs some code changes made so that we don't automatically disable github.com changelog lookups when a github.com token is not found (although you could potentially just configure a dummy token which will never be used).

I'm marking this as "status:ready" but it will need a community contribution from someone who needs this.. including because it should be tested in a real environment before merge.

@rarkins rarkins added priority-4-low Low priority, unlikely to be done unless it becomes important to more people status:ready core:changelogs Related to changelogs/release notes fetching and removed status:requirements Full requirements are not yet known, so implementation should not be started priority-5-triage labels Nov 16, 2021
@gaving

This comment was marked as spam.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core:changelogs Related to changelogs/release notes fetching priority-4-low Low priority, unlikely to be done unless it becomes important to more people type:feature Feature (new functionality)
Projects
None yet
Development

No branches or pull requests

5 participants