Future of metadata development and deployment - standalone? Only as part of KFP

[jlewi]

/kind feature

What is the future of metadata deployment?

There are currently at least two variants of metadata

• There is one version of metadata that is tightly integrated and deployed as part of KFP
• There is a second version of metadata which can be deployed independently

I think the differences might pertain mostly to the UI. I think KFP ships a UI for metadata integrated into the KFP UI but I think the backend might be the same.

I think the net effect is that a lot of development is happening in the KFP UI and the generic metadata UI is lagging behind; e.g. #217 is tracking upstreaming changes for lineage that are in KFP UI but not metadata UI.

I think metadata is largely based on mlmd which is developed in google/ml-metadata

What's the path forward for providing a metadata story?

/cc @neuromage @Bobgy @rmgogogo @avdaredevil @zhenghuiwang

[rmgogogo]

quick question, here "metadata deployment" means google/ml-metadata or KF metadata?

If google/ml-metadata, then it's already in KFP (google/mlmd provided a gRPC server). I think we don't have plan/resource to continue the KF metadata, right? I may lack of knowledge/info on the context.

[Bobgy]

I think #217 (comment) is the only planned future feature. Other than that, keeping current status and asking for community help is the best I can forsee.

[animeshsingh]

If we don't plan to extend it, and it becomes redundant with google/ml-metadata, and that's whre KFP is focussed, best to bring it up in community meeting to decide on the future

[jlewi]

If #217 is the only planned feature then what does this mean for creating a generic metadata story?

When people deploy Kubeflow pipelines do they get:

1. A metadata backend that can be used to record metadata from arbitrary services (not just KFP)
2. A UI for displaying metadata even if it wasn't created by KFP.

/cc @paveldournov

[Bobgy]

With current status, both 1. and 2. mentioned above are true.

If we have bandwidth, it's better metadata UI is keeping maintained separately, but I don't think that's the case now.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/front-end	0.63

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[jlewi]

I believe when you install the full KFP there are two different URLs for the UI for the metadata store

1. https:/${KFENDPOINT}/_/metadata
2. https://${KFENDPOINT}/_/pipeline/#/artifacts

Are these both pointing at the same UI service or are they two different servers?

I suspect they are two different servers but I could be wrong.

[Bobgy]

They are two different servers. Some code is reused in kubeflow/frontend, but the codebase are built, distributed completely separatedly.

[avdaredevil]

Yes these are two different code bases (kubeflow/metadata) and (kubeflow/pipelines) both which import MLMD Lineage from (kubeflow/frontend). Ideally pipelines would not have showed the link to the artifact store when running in iframe mode (within central-dashboard). But currently both UIs display the same information

…

On Mon, May 11, 2020 at 4:28 PM Yuan (Bob) Gong ***@***.***> wrote: They are two different servers. Some code is reused in kubeflow/frontend, but the codebase are built, distributed completely separatedly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#225 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIOV2QLGUOWLJJH6VOFG43RRCC27ANCNFSM4MSMVWDA> .

-- — Yours Sincerely, Apoorv Verma.