Improve repo structure and delete outdated manifests

[thesuperzapper]

This repository is extremely messy and is a horrible user experience. The mess has caused us real issues, as we botched the 1.1 release due to a lack of visibility of what is even in the manifests (See: #1553).

When paired with the almost complete lack of docs relating to Kubflow 1.1 this leaves Kubeflow near impossible to install, just take a look at recent-issues (across all the repos), and our slack and you will see what I mean.

My proposal is to make a new manifests repository and make some changes as we do, we should:

1. seperate each of the 'distributions' into a seperate repository (similar to gcp-blueprints),
2. only have the actual kubeflow manifests in the core manifests repo, (that is, no Istio and cert-manger, knative, as they are ostensibly part of a distribution, not Kubeflow)
3. actually keep the docs up to date

EDIT: see this comment below for my proposal of a 'Generic' distribution (to replace most of the current ones)

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/feature	0.60
platform/gcp	0.92

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[jlewi]

I think this is relevant to the discussion in kubeflow/community#402; particular clarifying the responsibility between application owners, the proposed deployments WG and the platform owners.

I think the lack of structure, and code accumulation in kubeflow/manifests is an outcome of lack of clear ownership/responsibilities.

[thesuperzapper]

/priority p1

[thesuperzapper]

I also believe we longer need the concept of a 'distribution' (in the form we currently have it), as this was primarily made to support GCP deploying a full K8S stack with Kubeflow on top.

If you look at Slack/GitHub-Issues you will see most users are installing Kubeflow on existing clusters of all types: GKE, AKS, EKS, native-k8s. We need to focus on supporting them!

To facilitate this, I propose we:

1. Do the above migration to a new kubeflow/manifests for all ACTUAL Kubeflow manifests
   • INCLUDING: pipelines, jupyter-web-app, etc
   • NOT INCLUDING: authentication (like dex), istio, argo-workflows, cert-manger, knative, etc
2. Consolidate all distributions (that are willing) into a 'Generic' one:
   • this distribution would:
     • assume the user at least already has a K8S cluster running
     • not depend on any vendor specific feature
     • live in its own git repo (as proposed above)
     • include everything you would need to deploy Kubeflow on an empty cluster: auth (dex), istio, cert-manger, argo-workflows, knative, mysql, etc
       • NOTE: we would let users pick the pieces they need (potentially with kfctl)
       • NOTE: we would clearly document which versions/configs of these tools are supported (if they don't choose to install them with Kubeflow)
     • get its actual Kubeflow manifests from the new kubeflow/manifests repo (as proposed above)
     • be intended to be used in Git-Ops, (but not tied to any specific git-ops tool)
   • I believe this is good for EVERYBODY:
     • as currently, all distributions are pretty much the same
     • vendors can still validate this distribution for their customers
     • the overhead for releasing a new Kubeflow version is reduced to a single place
     • it means that Kubeflow is an OPEN STANDARD (which can be run anywhere)
     • users will actually be able to install/upgrade Kubeflow!

[mameshini]

Agile Stacks distribution of Kubeflow is available here: https://github.com/agilestacks/stack-ml-eks
It is open source and it can be merged with Kubeflow repository in the future. We have taken a path of GitOps and customized the Kubeflow manifests to get more control over all components that are part of Kubernetes platform: dex, istio, letsencrypt, cert-manager, mysql, knative, ceph, csi plugin. The above example is available for AWS EKS, and we are working on the similar distributions for GKE, AKS, on-prem.

[thesuperzapper]

Another option (instead of Part 1) is that we discontinue the manifests repo entirely, and store the manifests for each component in the same repository as the component code.

However, I see a few issues with that approach:

• Non-Kubeflow components, like seldon-operator, spark-operator will be homeless.
  • (But I guess you could argue those should just go in the 'Generic' distribution repo, as not all users will need/want them)
• The kubeflow/kubeflow git-repo has many parts of Kubeflow in the same repo:
  • Example: centraldashboard, jupyter-web-app, etc
  • (We could consider splitting them up, but that seems like a lot of work/overhead)
• There is a need for cross-component resources/configs, and this would force those to live in the 'Generic'
  distribution (where they probably don't belong)

[alfsuse]

What if we do split non-Kubeflow components and group them into a sort of "extension/plugin" repo? This way we would have the following:

• Main Core Repo: the core KF components
• Core Extension Repo: centraldashboard, jupyter-web-app
• Other Extension Repo: seldon-op,spark-op, vendor specific-ops

This way the experience would result in a simpler and lighter installation for the core components plus a specific customized part based on requirements.
Resources/configs would be part of the extension configuration as patches or overwrite to the core components.

[erikerlandson]

a bit of a nit, but what is the benefit of per-release repo, compared to just branching/tagging for releases?