Model Registry Exploration for Charmifying It

[MattiaSarti]

Context

A charm for Kubeflow's Model Registry is required. As a consequence, the components of the Model Registry have to be explored to discuss how to best charm them and how the resulting charm(s) can be configured and integrated with other applications. For instance, do we need specific components (e.g.: databases) to be deployed together with the Model Registry? Are there specific tools that need to be bundled for the Model Registry to work?

What needs to get done

1. go through the official documentation of the Model Registry
2. understand how the Model Registry works, as part of the broader Kubeflow ecosystem with the other components
3. go through some usage examples
4. deploy the upstream components to some Kubernetes cluster (using kustomize)

Definition of Done

Having a clear understanding of:

1. how the Model Registry works from a user's perspective
2. what other components need to be deployed together with the Model Registry for it to work (e.g.: a database of a given kind) - both dependencies and core Kubeflow components
3. understand which configuration options can be applied to the different components (e.g.: connecting to databases of different kinds)

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-7506.

This message was autogenerated

[MattiaSarti]

Consider version v0.2.17.

[MattiaSarti]

Update: consider version v0.2.19, the one that at the moment comes with the latest 1.10 release (the 1.10.2 one) of the Kubeflow manifests.

[MattiaSarti]

Model Registry Exploration Notes

Overview

architecture-beta
    group kubeflow[Kubeflow Ecosystem]

    service istio(internet)[Istio] in kubeflow
    service database(database)[Relational Database] in kubeflow
    service registry(server)[Model Registry] in kubeflow
    service api(server)[Model Registry API] in kubeflow
    service ui(server)[Model Registry UI] in kubeflow
    service dashboard(cloud)[Central Dashboard] in kubeflow
    service csi(server)[Custom Storage Initializer] in kubeflow
    service kserve(server)[Kserve] in kubeflow

    api:T <-- B:istio

    registry:T <-- B:api

    registry:R <-- L:ui
    ui:R <-- L:dashboard

    registry:L <-- R:csi
    csi:L <-- R:kserve

    registry:B --> T:database

Loading

Key Features (for Python and REST Clients)

Register Metadata of, Update Metadata of, Index by Name, Retrieve, List and Search by Name:

• Models
• Model Versions (All or Model-Specific)
• Model Artifacts/Metadata (All or Model-Specific or Model-Version-Specific)

Central Dashboard Integration: Model Registry UI

Simply accessing, searching and managing registered models via a user interface integrated in the Central Dashboard.

KServe Integration: Custom Storage Initializer

"The Model Registry Custom Storage Initializer (CSI) is a custom implementation of the KServe ClusterStorageContainer. This feature allows users to utilize Model Registry metadata to download and deploy models efficiently." [blog reference]

sequenceDiagram
    actor U as User
    participant MR as Model Registry
    participant KC as KServe Controller
    participant MD as Model Deployment (Pod)
    participant MRSI as Model Registry Storage Initializer
    U->>+MR: Register ML Model
    MR-->>-U: Indexed Model
    U->>U: Create InferenceService CR
    Note right of U: The InferenceService should<br/>point to the model registry<br/>indexed model, e.g.,:<br/> model-registry://<model-registry-url>/<model>/<version>
    KC->>KC: React to InferenceService creation
    KC->>+MD: Create Model Deployment
    MD->>+MRSI: Initialization (Download Model)
    MRSI->>MRSI: Parse URI
    MRSI->>+MR: Fetch Model Metadata
    MR-->>-MRSI: Model Metadata
    Note over MR,MRSI: The main information that is fetched is the artifact URI which specifies the real model location, e.g.,: https://.. or s3://...
    MRSI->>MRSI: Download Model
    Note right of MRSI: The storage initializer will use<br/> the KServe default providers<br/> to download the model<br/> based on the artifact URI
    MRSI-->>-MD: Downloaded Model
    MD->>-MD: Deploy Model

Loading

Newer Features (on Roadmap to Kubeflow 1.11 ⚠️)

• (Unified) Experiment Tracking

  "Model Registry currently only handles models and versions. ML/AI development starts with experiments and runs. Model Registry API MUST add support for creating and tracking the following: Experiments, Runs within Experiments, DataSets that are part of input data set, Metrics and metric history associated with runs, Parameters associated with runs... ...should also be accessible in Model Versions." [reference]

  "to resolve the current fragmented and limited experiment tracking experience by expanding the
  Kubeflow Model Registry into a unified, centralized metadata store. Currently, experiment tracking is scattered
  across components like Kubeflow Pipelines (which requires pipeline execution for tracking) and Katib (limited to
  hyperparameter tuning)." [reference]

• InferenceService <-> Serving Environment Linking

  "...monitor kserve InferenceService resources with specific labels and handle the lifecycle of related Model Registry ServingEnvironment and InferenceService entities so that they are linked together and let users know when a model is deployed or not." [reference]

• UI-based Registered Model Deployment with KServe

  "A button in the UI which allows the creation of the KServe Isvc without requiring to operate programmatically." [reference]

• Trainer Integration

  "Trainer has implemented initializers for model and dataset, and will support model exporter in the future... ...It might be a good idea to support the model-registry as one of the destinations of the exporter." [reference]

• Model Catalog

  "When a MR instance is deployed, it's "empty" and some MR-admin or Orgs may want to offer a quick way for MR-user to know which are the Company/Org "vetted" Models which can be used." [reference]

  "Several public ML and LLM model catalogs such as Hugging Face are now available with easily accessible opensource models... ...There is a need for a uniform and simple way to access various Model Catalogs hosting foundation models to allow users to easily discover and register models in a Model Registry" [reference]

  "The Model Catalog Service provides a read-only discovery service for ML models across multiple catalog sources. It acts as a federated metadata aggregation layer, allowing users to search and discover models from various external catalogs through a unified REST API." [reference]

• Async Upload (Model Storages)

  "...a feature to be able to copy one model from one storage location to another storage location, registering the model in the process, and generating a ModelCar as a part of that process... ...A common case is when a user wants all of their models to live in a centralized OCI-compliant storage layer (quay, self-hosted, etc), but they may not necessarily have all models they are interested currently in that storage layer." [reference]

  "Lightweight, non‑root Python Job containerised for transferring model artefacts between storage back‑ends (S3, OCI, PVC, …) and registering them in Kubeflow Model Registry." [reference]

• MLflow Integration

  "Model Registry has it's own REST API, but a domain model very similar to MLflow. Model Registry is also going to add support for Experiment Tracking in another enhancement request... ...MLflow SDK supports loading plugins for integrating with other tracking and registry stores... ...Model Registry MUST implement an MLflow plugin to allow millions of existing MLflow users to seamlessly use Model Registry for experiment tracking to begin with. Support for Registry API/store can be added in the future... ...Implement an MLFlow tracking store for tracking experiments and runs." [reference]

• Feast Integration (?)