Support hex v2 registry

[rarkins]

Describe the proposed change(s).

Discussed in #24764

In order to support private registries, we need the hex datasource to support the hex v2 registry format described here: https://github.com/hexpm/specifications/blob/main/registry-v2.md

It seems that we should switch to using this registry protocol by default, including for the public registry.

[bryannaegele]

How are the elixir/erlang versions set? And is it possible to include an elixir script to fetch the package info? That would make this very straightforward.

[rarkins]

No, we don't permit the calling of CLI tools as part of lookup

[bryannaegele]

Ok. The suggestion here is to allow a script which is executed with mix run that is just fetching and decoding the list of releases from a registry. Everything needed is already available when installing elixir which is allowed in the mix manager but not currently done in the hex datasource. Hex is shipped as part of Elixir, so there are no other cli requirements.

The current methodology fetches json from an API and performs no validations. Using the registry, responses are protobuf encoded and include other features such as package signing verification. So our options for hex are to either leverage the functionality that ships with elixir for auth, fetching, decoding, and package signing verification or reimplement all of that functionality in TS.

I wasn't able to find any other datasources that are processing protobufs or protobuf related packages in core, so that would be a new addition.

So the proposal here is to let hex interact with hex registries in the hex datasource. Are there concerns or certain pieces of other functionality that are part of the current policy? We're trying to understand the reason for the policy if it pertains only to package lookup.

[bryannaegele]

This is a very basic demo of the proposal.

https://gist.github.com/bryannaegele/4b9d11bc927d3887267028d1ea9f64f9

[rarkins]

We want datasources to be in TS so they can be used in the browser. Not other datasource requires binaries for lookups. If it's extremely hard it perhaps indicates that the registry should be made more accessible in future? Most are REST-based.

[viceice]

running external code for datasources will slow down the lookup phase massively, because we need to ensure to install a proper version of erlang and elixir. that is usually only done when we need to update lockfiles.

so I'm strictly against calling any binaries inside datasources.

There are official JavaScript libraries to decode protobuff messages easily.

[bryannaegele]

Yep. Understandable. I'll get working on it

[bryannaegele]

I'm pretty much done with this but running into one issue. In order to try and keep the previous amount of package detail with this change we have to make two calls. The first is to the registry, then a second to the API when it's hosted on hex.pm to get metadata. The hexpm registry and api are hosted on different domains (registry is a subdomain).

The issue I'm seeing when trying it locally with a private organization package hosted on hex, only the first call matches a hostRule and sets the auth token.

config

module.exports = {
  token: 'github_pat_**',
  repositories: ['simplebet/renovate-registry-test'],
  hostRules: [
    {
      matchHost: "https://hex.pm/api/repos/simplebet/",
      token: "org-token",
      authType: "Token-Only"
    },
    {
      matchHost: "https://repo.hex.pm/repos/simplebet/",
      token: "org-token",
      authType: "Token-Only"
    },
    {
      matchHost: "https://getoban.pro/repo",
      token: "oban-token",
      authType: "Token-Only",
    }
  ],
  registryAliases: {
    "oban": "https://getoban.pro/repo"
  }
};

log

DEBUG: Package registry url: https://repo.hex.pm/repos/simplebet/packages/private_package (repository=simplebet/renovate-registry-test)
DEBUG: hostRules: applying Bearer authentication for repo.hex.pm (repository=simplebet/renovate-registry-test)
DEBUG: Package metadata url: https://hex.pm/api/repos/simplebet/packages/private_package (repository=simplebet/renovate-registry-test)
DEBUG: GET https://hex.pm/api/repos/simplebet/packages/private_package = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=401 retryCount=0, duration=90) (repository=simplebet/renovate-registry-test)
DEBUG: Datasource unauthorized (repository=simplebet/renovate-registry-test)
       "datasource": "hex",
       "packageName": "org:simplebet:private_package",
       "url": "https://hex.pm/api/repos/simplebet/packages/private_package"
DEBUG: Failed to look up hex package org:simplebet:private_package (repository=simplebet/renovate-registry-test, packageFile=mix.exs, dependency=org:simplebet:private_package)

As you can see, a host rule is only applied to the first http call in getReleases. Is this expected behavior or a bug?

This does not crop up in unit tests.

it('processes a private organization repo with auth', async () => {
      httpMock
        .scope(baseRegistryUrl, {
          reqheaders: {
            authorization: 'abc',
          },
        })
        .get('/repos/private_org/packages/tls_certificate_check')
        .reply(200, tlsCertificateCheckRegistryResponse);

      httpMock
        .scope(baseHexpmUrl, {
          reqheaders: {
            authorization: 'abc',
          },
        })
        .get('/repos/private_org/packages/tls_certificate_check')
        .reply(200, tlsCertificateCheckAPIResponse);

      hostRules.find.mockReturnValue({
        authType: 'Token-Only',
        token: 'abc',
      });
      const result = await getPkgReleases({
        ...config,
        packageName: 'org:private_org:tls_certificate_check',
      });

      expect(result).not.toBeNull();
    });

[rarkins]

log

DEBUG: Package registry url: https://repo.hex.pm/repos/simplebet/packages/private_package (repository=simplebet/renovate-registry-test)
DEBUG: hostRules: applying Bearer authentication for repo.hex.pm (repository=simplebet/renovate-registry-test)
DEBUG: Package metadata url: https://hex.pm/api/repos/simplebet/packages/private_package (repository=simplebet/renovate-registry-test)
DEBUG: GET https://hex.pm/api/repos/simplebet/packages/private_package = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=401 retryCount=0, duration=90) (repository=simplebet/renovate-registry-test)
DEBUG: Datasource unauthorized (repository=simplebet/renovate-registry-test)
       "datasource": "hex",
       "packageName": "org:simplebet:private_package",
       "url": "https://hex.pm/api/repos/simplebet/packages/private_package"
DEBUG: Failed to look up hex package org:simplebet:private_package (repository=simplebet/renovate-registry-test, packageFile=mix.exs, dependency=org:simplebet:private_package)

As you can see, a host rule is only applied to the first http call in getReleases. Is this expected behavior or a bug?

FYI you should see that log messageabout applying authentication once per host, not once per request.

Based on your hostRules, I would expect the auth to be applied to https://hex.pm too

[bryannaegele]

Ah. I didn't catch that. I think that could present an edge case if a user needed to auth to multiple orgs but that isn't the case in the example repo. I'll dig into this.

[bryannaegele]

Works today 🤷🏻 . I'll mark the PRs ready for review.

[1oglop1]

@paulo-ferraz-oliveira You might find this interesting/important, there was a lot of work put into this but it wasn't finished.

TL DR; the solution is split into 2 PRs and possibly requires more setup for testing then the issue I pointed out.
#29775
#29756