Hub / Model Registry is being deployed to the namespace `default`

[christian-heusel]

Validation Checklist

• I confirm that this is a Kubeflow-related issue.
• I am reporting this in the appropriate repository.
• I have followed the Kubeflow installation guidelines.
• The issue report is detailed and includes version numbers where applicable.
• I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).
• This issue pertains to Kubeflow development.
• I am available to work on this issue.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Detailed Description

This issue was introduced in the recent PR #3318

$ kubectl get pod -n default
NAME                                        READY   STATUS    RESTARTS      AGE
model-catalog-postgres-0                    1/1     Running   0             22h
model-catalog-server-895b9c7bb-w9hdv        2/2     Running   0             22h
model-registry-db-bf6ff894f-m9vns           1/1     Running   0             22h
model-registry-deployment-fbfc6587f-rkvjv   2/2     Running   1 (22h ago)   22h

• As part of the fix it would also be good to create a check from that ever happening again. We could check for the namespaces that we know that we control and error if there are any leftover pods outside of that:
  https://github.com/kubeflow/manifests/blob/336ac873e50adcebae8cc35c9a5884c81fc11bff/tests/PSS_enable.sh#L11

---

First reported in https://cloud-native.slack.com/archives/C073W572LA2/p1778600481592009

Steps to Reproduce

1. Deploy latest master to kind
2. Check running pods in namespace default via kubectl get pod -n default

Screenshots or Videos (Optional)

No response

[christian-heusel]

Also when investigating a different issue I found the following:

$ istioctl analyze
2026-05-12T23:26:22.906243Z	error	kube	translation function for core/v1alpha1/MeshConfig not found	controller=analysis-controller
2026-05-12T23:26:22.906298Z	error	kube	translation function for core/v1alpha1/MeshNetworks not found	controller=analysis-controller
Error [IST0101] (VirtualService default/model-registry) Referenced gateway not found: "kubeflow-gateway"
Error [IST0101] (VirtualService default/model-registry) Referenced host not found: "model-registry-service.kubeflow.svc.cluster.local"
Warning [IST0132] (VirtualService default/model-registry) one or more host [*] defined in VirtualService default/model-registry not found in Gateway default/kubeflow-gateway.
Warning [IST0174] (DestinationRule default/model-registry-service) The host model-registry-service.kubeflow.svc.cluster.local defined in the DestinationRule does not match any services in the mesh.
Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run 'kubectl label namespace default istio-injection=enabled' to enable it, or 'kubectl label namespace default istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0118] (Service default/model-catalog-postgres) Port name postgres (port: 5432, targetPort: 5432) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service default/model-registry-db) Port name dbapi (port: 5432, targetPort: 5432) doesn't follow the naming convention of Istio port.
Error: Analyzers found issues when analyzing namespace: default.
See https://istio.io/v1.27/docs/reference/config/analysis for more information about causes and resolutions.

[Raakshass]

investigated and raised a fix in #3458.

root cause: 3 of the 4 model-registry kustomization overlays referenced by example/kustomization.yaml were missing namespace: kubeflow. the ci install script uses kubectl apply -n kubeflow which masked the bug, but the documented deployment path (kustomize build example/ | kubectl apply -f -) has no such override.

overlay	had namespace?
upstream/overlays/postgres/kustomization.yaml	no
upstream/options/istio/kustomization.yaml	no
upstream/options/catalog/overlays/demo/kustomization.yaml	no
upstream/options/ui/overlays/istio/kustomization.yaml	yes

the fix adds namespace: kubeflow to the 3 missing overlays. all model-registry and catalog ci tests pass. the only failing check is the katib grid trial timeout in the end-to-end integration test which is unrelated to these changes.

regarding the namespace guard check you mentioned — happy to look into that as a follow-up if needed.

[pboyd]

@Raakshass, what if we add an overlay to this repo that sets the namespace to the default profile namespace (kubeflow-user-example-com) instead? If we deploy it to kubeflow the dashboard won't work.

Something like this:

diff --git a/applications/hub/overlays/kustomization.yaml b/applications/hub/overlays/kustomization.yaml
new file mode 100644
index 00000000..c46137e6
--- /dev/null
+++ b/applications/hub/overlays/kustomization.yaml
@@ -0,0 +1,10 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: kubeflow-user-example-com
+
+resources:
+- ../upstream/overlays/postgres
+- ../upstream/options/istio
+- ../upstream/options/ui/overlays/istio
+- ../upstream/options/catalog/overlays/demo
diff --git a/example/kustomization.yaml b/example/kustomization.yaml
index 98085940..09f89008 100644
--- a/example/kustomization.yaml
+++ b/example/kustomization.yaml
@@ -85,14 +85,8 @@ resources:
 # Spark Operator
 - ../applications/spark/spark-operator/overlays/kubeflow

-# Model Registry
-- ../applications/hub/upstream/overlays/postgres
-# Model Registry Istio networking (VirtualService for /api/model_registry/)
-- ../applications/hub/upstream/options/istio
-# Model Registry UI
-- ../applications/hub/upstream/options/ui/overlays/istio
-# Model Catalog (demo)
-- ../applications/hub/upstream/options/catalog/overlays/demo
+# Model Registry (Hub)
+- ../applications/hub/overlays

 # Ray is an experimental integration
 # Here is the documentation for Ray: https://docs.ray.io/en/latest/

Although, we're going to need more than that. I can work up the rest of a PR, if you want hand this off.

[Raakshass]

understood, kubeflow-user-example-com is the correct namespace since model registry is per-user and deploying to kubeflow would break the dashboard.
one question i have an open upstream pr at kubeflow/hub#2709 that sets namespace: kubeflow in the hub repo overlays. should that be updated, closed, or left as is given that the manifests overlay you proposed would override it?
could you also share what the additional work beyond the basic overlay involves? happy to implement the overlay and namespace guard check on the manifests side.

[pboyd]

@Raakshass, we should probably close kubeflow/hub#2709. There isn't a single namespace name we can use there because the user namespace varies. kubeflow-example-user-com works as an example overlay because that's the name that's used in the other manifests.

I'm not sure exactly what else needs to be done. I tested the patch I sent above and found that the model-registry UI was not working. I suspect there are a couple resources which should be in the kubeflow namespace, probably something for istio. I can dig more into this, if it would help.

[Raakshass]

@pboyd i investigated the ui breakage from the source. the overlay sets namespace: kubeflow-user-example-com on all resources, but 6 fields in the upstream istio yamls have hardcoded references that break:

gateway references (2 fields):

• upstream/options/istio/virtual-service.yaml:7 → gateways: [kubeflow-gateway]
• upstream/options/ui/overlays/istio/virtual-service.yaml:9 → gateways: [kubeflow-gateway]

istio resolves gateway names relative to the virtualservice's namespace. with the overlay these virtualservices move to kubeflow-user-example-com, but kubeflow-gateway lives in kubeflow (defined in common/istio/kubeflow-istio-resources/base/). they need to be patched to kubeflow/kubeflow-gateway.

destination host fqdns (4 fields):

• upstream/options/istio/virtual-service.yaml:16 → model-registry-service.kubeflow.svc.cluster.local
• upstream/options/istio/destination-rule.yaml:6 → model-registry-service.kubeflow.svc.cluster.local
• upstream/options/ui/overlays/istio/virtual-service.yaml:24 → model-registry-ui-service.kubeflow.svc.cluster.local
• upstream/options/ui/overlays/istio/destination-rule-ui.yaml:8 → model-registry-ui-service.kubeflow.svc.cluster.local

these hardcode .kubeflow. in the fqdn, but the services move to kubeflow-user-example-com with the overlay. they need to be patched to .kubeflow-user-example-com.svc.cluster.local.

the authorizationpolicies and workload resources (postgres, model-registry, catalog) are fine in the user namespace — no changes needed there.

i can implement the overlay with these 6 kustomize patches if you'd like, or happy for you to take it since you have the test cluster.