kubeflow istio gateway should be in istio-namespace

[jlewi]

Right now the kubeflow ISTIO gateway is in namespace kubeflow. Based on the docs it looks like the gateway should be in the same namespace as the pods running the envoy servers at the edge providing ingress to the cluster.

Having the ISTIO gateway in the kubeflow namespace is currently working with Kubeflow 1.0 and ISTIO 1.1.6. I was told though that this might be due to a bug which got fixed in ISTIO (~1.5.X).

Either way we probably want to move the ISTIO gateway into the istio namespace. This likely means we will need to update all of our virtual services referencing the gateway to refer to it in the appropriate namespace.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/bug	0.79

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/bug	0.79

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[Jeffwan]

Any plan to upgrade istio? 1.1.x is too old and if isito gateway in kubeflow doesn't work in higher istio versions, we might want to upgrade istio first. Some other components depends on istio may need corresponding changes.

[jlewi]

@Jeffwan I think the answer about which version of ISTIO to install will vary from platfom to platform.

On GCP we are planning on switching to ASM (managed ISTIO) (#1024) . Right now this will install ISTIO 1.4. Really what it means is on GCP ISTIO versioning will be determined by ASM.

This will likely lead to version incompatible issues which we will have to surface upstream to the various applications in order to find suitable work arounds to support a suitable range of ISTIO versions.

I think this is most pronounced right now in KFServing since KFServing has very tight coupling between KFServing versions, knative, and ISTIO.

@johnugeorge @yanniszark @animeshsingh

[yanniszark]

Any plan to upgrade istio? 1.1.x is too old and if isito gateway in kubeflow doesn't work in higher istio versions, we might want to upgrade istio first. Some other components depends on istio may need corresponding changes.

@Jeffwan we already have Istio 1.3.1 committed in manifests, thanks to @krishnadurai.
The kfctl_istio_dex config already uses it successfully.

Having the ISTIO gateway in the kubeflow namespace is currently working with Kubeflow 1.0 and ISTIO 1.1.6

It still works for 1.3.1 (and maybe 1.4, haven't tried though).
However, I see indeed that the docs say:

the Gateway resource must reside in the same namespace as the gateway workload instance.

I wonder if this is a practical issue or a best-practice issue.
Agreed that it would be best to follow the Istio recommendation.
We also need to find out which controllers are affected. I know the notebook-controller is.

[nrchakradhar]

Thought will add references to the finer documentation and isito issue.
Documentation: Specifically in Gateway section

Istio issue
Comment

@yanniszark
In Kubeflow only one gateway is created as much as I have seen. However multiple virtual services are created as and when new notebooks are created. For VirtualServices this restriction does not seem to apply.
https://istio.io/docs/reference/config/networking/virtual-service/#VirtualService

Please correct me if my understanding is incorrect.

[jlewi]

THanks @nrchakradhar Virtual services can reside in any namespace. We already have virtual services in multiple namespaces. VS can refer to gateways in other namespaces.

[shawnzhu]

IMHO, there're at least 4 places need to change:

1. component istio/istio: remove hard coded namespace plus adding a variable named namespace:
   https://github.com/kubeflow/manifests/blob/299f4669c9e467ad1234f96a6192aca94fc09801/istio/istio/base/kustomization.yaml#L6
2. anywhere that refer to kubeflow/kubeflow-gatway no matter virtual-services or ConfigMap, need to use Istio-system/kubeflow-gateway instead.
3. in all existing kfdef depends on master branch, will need to add namespace variable for kustomize config istio

change is easy but takes time to verify them