Create OCP Stack in KF

[nakfour]

Testing done on OCP 4.3/Kub 1.16. Code is in this PR : https://github.com/kubeflow/manifests
Issues:

1. Profiles app is not setting $namespaces as seen in error from validating webhook. Looking at manifest/profiles/base_v3 it seems not to specify a variable $namespace as is specified in base/kustomization.yaml file. DONE-FIXED

WARN[0032] Encountered error applying application profiles:  (kubeflow.error): Code 500 with message: Apply.Run : error when creating "/tmp/kout859943078": admission webhook "pilot.validation.istio.io" denied the request: configuration is invalid: domain name "$(namespace).svc.cluster.local" invalid (label "$(namespace)" invalid)  filename="kustomize/kustomize.go:266"
WARN[0032] Will retry in 6 seconds.                      filename="kustomize/kustomize.go:267"

2. Cache-server pod crashing on "secret "webhook-server-tls" not found"
3. When creating a notebook server, fsgroup is set to 100. DONE-FIXED
4. When launching a notebook server we get error "no healthy upstream" DONE-FIXED
5. ml-pipeline pod crashes withe logs DONE-FIXED

 config.go:37] Please specify flag PROFILES_KFAM_SERVICE_HOST

This is due to the fact this needs profiles installed, and profiles only works when installed in top kustomization file
6. Getting to Pipeline page from Kubeflow dashboard gives error before: kubeflow/kubeflow#5271
upstream connect error or disconnect/reset before headers. reset reason: connection failure
to solve this set

tls:
      mode: DISABLE

in destination rule for ml-pipeline-ui. DONE-FIXED

• tf-jobs,pytorch,katib,metadata (Vasek)

• profiles, pipelines (Landon)

• Seldon (Juana)

• JH web (Maros)

• Notebook Controller

• istio (Juana)

• cert-manager

[vpavlin]

2. cache server failing for KFP multi-user installation kubeflow/community-distribution#1390

[vpavlin]

1. could it be because the params.yaml is not taken into account (https://github.com/kubeflow/manifests/blob/master/profiles/overlays/istio/params.yaml) and thus the virtual service is not templated?

[vpavlin]

TF operator, PyTorch operator and Katib work

metadata-db needs a small fix mentioned in kubeflow#1567 (comment)