Add Model Catalog API to Model Registry

[dhirajsb]

Is your feature request related to a problem? Please describe.
Several public ML and LLM model catalogs such as Hugging Face are now available with easily accessible opensource models.
At present Kubeflow Model Registry has an API for registering and managing locally trained and published Registered Models. Also, users have to use a variety of different websites, UIs, or APIs to browse and discover foundation models in various catalogs and manually register them for deployment.

Describe the solution you'd like
There is a need for a uniform and simple way to access various Model Catalogs hosting foundation models to allow users to easily discover and register models in a Model Registry for local training, enhancement, and serving.
The implementation could start simple by allowing users to create a simple curated model catalog source that is backed by a yaml file. The yaml file could contain a list of high level foundation models metadata, some README text, and information about catalog model versions.
In the future other catalog source implementations can be created to allow browsing Hugging Face, OpenAPI, etc.

Describe alternatives you've considered
As an alternative, a common UI could be built that has adhoc client code to access different catalogs to browse and register models to a Kubeflow Model Registry. However, having a consolidated/common simple backend will make developing such a catalog browsing UI simpler in the future.

Additional context
As an example, a simple API that exposes information about models such as the information provided in HuggingFace modelcard, and also supports simple catalog model search by names, tags, etc. would be incredibly useful for Kubeflow users.

[dhirajsb]

@andreyvelich @thesuperzapper we're working on a design for a simple model catalog API wrapper for local and remote catalogs like Hugging Face. I'll add more details with diagram, etc. to this feature request tomorrow.

We're hoping it's a valuable addition to Model Registry and helps make the entry point and user experience easier into Kubeflow Model Registry, especially for foundation models and LLMs.

[dhirajsb]

A proposed high level architecture would like below.

A configmap can be used to configure a small amount of high level metadata about a collection of catalog sources, such as name, type, description, etc. about the catalog. Every catalog source may also include a secretName if credentials are required to connect to the catalog. E.g. HuggingFace could store API secret/token in a Kubernetes secret.

Model registry would implement the logic to handle different catalog types, and process the catalog source information to fetch the catalog or create a remote connection to it. When Catalog API operations to list CatalogModels and CatalogModelRevisions are received, model registry will query the catalog source implementation logic to fetch the requested information.

[pboyd]

Based on some feedback, we need some additional information in the catalog:

• catalog source
• long description
• maturity
• artifacts

Updated schema and example are in this gist: https://gist.github.com/pboyd/278c7b1e9ce0292b82cb871fa7d2103b

[rareddy]

Is there an OpenAPI definition defined for this Catalog model or is it going to use existing Model Registry API?

[dhirajsb]

The OpenAPI version is here in the initial version PR.