feat(datasource/galaxy-collection): migrate from v2 to v3

[secustor]

Changes

Use the v3 API instead of v2

Context

Closes #24981

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

https://github.com/secustor/renovate_ansible_galaxy_test

[viceice]

can we remove unused stuff from those json fixtures files?

[secustor]

can we remove unused stuff from those json fixtures files?

Removed the biggest junks from the detailed versions already. That is the reason I have added the 1 week cache.
Currently it is IMO a good mix between usefulness and size

[jon4hz]

If you're already migrating to galaxy v3, do you think it would be possible to enable support for custom registries aka self hosted automation hubs?

[secustor]

If you're already migrating to galaxy v3, do you think it would be possible to enable support for custom registries aka self hosted automation hubs?

Are you talking about allowing overwriting the registryUrl via packageRule or automatically detecting configured registries?
The later would be considerable work and outside of the scope of this PR.

[jon4hz]

If I understand #14799 correctly, the manager would already detect custom registries per dependency. But I'm not very familiar with the internals of renovate, so I've no idea how difficult it would be to actually use these sources.

As for the overwrite by packageRule, that would probably be nice to have, however I'm not sure how useful that feature actually is in production unless there is a way to configure multiple urls. For example our automation hub has at least 3 different repositories (validated, rh-certified and community content), all of them have different urls and in a perfect world, renovate should be able to check them all for dependency upgrades.

Or the source repository in the requirements.yaml is set explicitly, in that case renovate should only check that specific repository for upgrades.

[secustor]

If I understand #14799 correctly, the manager would already detect custom registries per dependency. But I'm not very familiar with the internals of renovate, so I've no idea how difficult it would be to actually use these sources.

Or the source repository in the requirements.yaml is set explicitly, in that case renovate should only check that specific repository for upgrades.

Looking at the code of ansible-galaxy you are right, with customRegistrySupport = true it should work out of the box with that refactor.

As for the overwrite by packageRule, that would probably be nice to have, however I'm not sure how useful that feature actually is in production unless there is a way to configure multiple urls. For example our automation hub has at least 3 different repositories (validated, rh-certified and community content), all of them have different urls and in a perfect world, renovate should be able to check them all for dependency upgrades.

If you set 3 registryUrls Renovate will use the hunt strategy for galaxy-collection, therefore it tries each registry until it finds a package. https://docs.renovatebot.com/configuration-options/#registryurls

[jon4hz]

Hi again,

I took another look at the API of the automation hub and unfortunately simply enabling custom registry support wont do the trick because there is a small difference in the API path when using an ah compared to galaxy.com.

For example if I have my ah repository at https://ah.mydomain.com/api/galaxy/content/community/ the corresponding API call would be https://ah.mydomain.com/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/. With this PR renovate will try to call https://ah.mydomain.com/api/galaxy/content/community/api/v3/plugin/ansible/content/published/collections/index/ instead from what I understand.

So there are 2 problems here:

1. One "api" to much in the URL -> This could be fixed by changing the default URL to defaultRegistryUrls = ['https://galaxy.ansible.com/api']; and remove the "api" part in the hardcoded url paths.
2. Renovate should replace "published" in the url path with the repository where the collection is actually stored. In my example above that repository would be "community".

The second problem would probably require renovate to somehow "parse" the registry url, extract the last part from the url and then use that information in the other API calls.
I can't really estimate how much work this change will take. Do you think that would be doable within this PR?

[secustor]

Does this three redirect you to the proper APIs?

• https://ah.mydomain.com/api/v3/collections/{{namespace}}/{{collection}}/
• https://ah.mydomain.com/api/v3/collections/{{namespace}}/{{collection}}/versions/
• https://ah.mydomain.com/api/v3/collections/{{namespace}}/{{collection}}/versions/{{version}}/

Tough the documentation suggest this shouldn't be used.

https://galaxy.ansible.com/api/v3/swagger-ui

If so I can use these instead and follow the redirects.

Do you have API documentation for the AutomationHub?

[jon4hz]

Does this three redirect you to the proper APIs?

Nope, calling these endpoints returns 404. But I think that makes sense because this isn't how the automation hub works. The ah can contain multiple repositories and if my hub has e.g. 3 repositories, I have to add all 3 of them to my ansible.cfg for example. There is no central API to get information about all repositories at once. You always have to specific about what repo you want to use.

Tough the documentation suggest this shouldn't be used.

I'm not sure why the galaxy API docs suggest not to use them. Until now I thought galaxy (v3) would simply be an automation hub with a single repository "published" but as it seems Redhat added some tweaks to the API.

Do you have API documentation for the AutomationHub?

According to https://access.redhat.com/solutions/6963967 Redhat doesn't provide public API docs. However after a quick search, I could find https://developers.redhat.com/api-catalog/api/automation-hub which seems to be the automation hubs API with the small difference that the automation hub's API actually starts with /api/galaxy/ and not /api/automation-hub...

[secustor]

With the last finding I'm sure this is out of scope for this change right now. Please create a discussion and summarise the findings there.

[renovate-release]

🎉 This PR is included in version 37.56.0 🎉

The release is available on:

• GitHub release
• 37.56.0

Your semantic-release bot 📦🚀