chore(backend/frontend): Remove MLMD as a depedency of KFP

[HumairAK]

Description

MLMD is effectively no longer supported. As it is such a critical part of KFP, this is not great for KFP outlook and we need to prioritize dropping this dependency. The two options I see are:

1. Find an alternative (which?)
2. Directly bake into KFP and replace the functionality required to support the MLMD dependent code
3. ??

Not sure how feasible (2) is yet, this will require some investigation.

[HumairAK]

fyi @chensun

[franciscojavierarceo]

My recommendation would be to find an alternative (or build one) and allow MLMD to be a configurable option. Given TensorFlow's popularity, we wouldn't want to remove that from the community.

[andreyvelich]

Thank you for starting this @HumairAK!

I recall that during our discussions about the Model Registry proposal, one of the key objectives was to replace KFP MLMD with it: https://docs.google.com/document/d/1T3KfOqIfJohp0s1koQ2XrJJQQhj7TECO-m2xPsW59_c/edit?tab=t.0.
Do we know where we are with that, or I misunderstood something ?
cc @kubeflow/wg-data-leads

On the side note, I think we should discuss a reusable solution that can serve multiple Kubeflow projects. Throughout the entire AI/ML lifecycle, a vast amount of metadata is generated, and we might want to have a standardized, portable, and extensible way to manage it.
For example:

• Pre-trained models, datasets, metrics, and logs from the Kubeflow TrainJobs
• Models from Trials, hyperparameters, and neural architectures from the Kubeflow OptimizeJob (e.g. Katib/Optimizer/AutoML)

It would be great to explore how we can consolidate efforts around MLMD across different Kubeflow projects.

cc @kubeflow/wg-training-leads @shravan-achar @akshaychitneni @bigsur0

Would love to hear your thoughts!

[franciscojavierarceo]

Coincidentally, I discussed this with @Crazyglue and @syntaxsdev and we also talked about the Feast and KFP registry so I agree @andreyvelich!

[rareddy]

@andreyvelich @franciscojavierarceo We recently initiated a similar proposal in Model Registry: kubeflow/hub#865.

Originally, the goal was to adopt MLMD from KFP into Model Registry, but that approach lost traction due to missing features like multi-tenancy. We considered contributing to MLMD to extend its capabilities, but that wasn’t feasible. Next, we explored rewriting the MLMD backend in Golang, but given Model Registry’s usage patterns, that didn’t seem like the right approach.

With evolving patterns like the Kubeflow SDK and Llama Stack API, we anticipate the need for new APIs to capture metadata across various subsystems.

As a first step, we plan to replace the Model Registry’s metadata layer with direct database access. Then, we’ll revisit the API layer to identify collaboration opportunities within Kubeflow SDK/Llama Stack to support systems like KFP.

I suggest we first assess the extent of MLMD usage in KFP, and then assess this issue along with work like Kubeflow SDK

[andreyvelich]

I think, this is very long standing task to have metadata layer across Kubeflow projects. For the context see this:

• Kubeflow component integration with ML Metadata community#783
• Create a kubeflow/metadata repository community#238

Maybe it is time to reveal the kubeflow/metadata repo: https://github.com/kubeflow/metadata

We should chat more about it.

[andreyvelich]

cc @johnugeorge

[rareddy]

We should chat more about it.

Since we do call out in architecture diagrams, we should evolve the Model Registry capabilities be this store IMO.

[andreyvelich]

cc @thesuperzapper

[thesuperzapper]

I have no current opinion on dropping support for MLMD, or adopting a fork of it, or something similar.

However, we will need to include it in the KFP manifests for as long as we intend to keep backwards compatibility with KFP's v1 API.

[HumairAK]

Thank you for all your inputs folks, I highly agree with @andreyvelich in that there needs to be planning around a broader metadata tracking solution. Decoupling this from KFP is the right architecture IMO for various reasons, one being that this allows users to create/pull/push artifacts from external sources and then utilize these in their KF Pipelines (and other KF components). Conversely it allows for surfacing api's that allow other services utilize and to consume the artifacts produced by their pipelines (or from wherever else, like notebooks). This would allow us to create robust UIs around our metadata data tracking capabilities and make integrations a heck of a lot easier across components.

I've generalized the title to be keep our discussion a bit more broader, in case the route we take is one that involves simply replacing mlmd with a KF-wide solution of which KFP is but one consumer.

[franciscojavierarceo]

Agree @HumairAK, we need a broader metadata tracking solution and we should look to MLFlow to learn what an endstate could look like (maybe even make our solution compatible with MLFlow).

CC @rareddy @tarilabs