fix(hub): deploy model registry to user profile namespace instead of default

[Raakshass]

summary

deploys hub (model registry + model catalog) to the user profile namespace (kubeflow-user-example-com) instead of default. creates a manifests-level overlay with namespace assignment and 6 istio patches. adds a default namespace isolation guard to the full integration test workflow.

fixes #3457

root cause

#3318 added 4 hub upstream references to example/kustomization.yaml. three of them had no namespace: directive. the continuous integration install script masked this by passing -n kubeflow to kubectl apply, but the documented single-command install (kustomize build example | kubectl apply -f -) deployed everything to default.

$ kubectl get pod -n default
model-catalog-postgres-0                    1/1     Running
model-catalog-server-895b9c7bb-w9hdv        2/2     Running
model-registry-db-bf6ff894f-m9vns           1/1     Running
model-registry-deployment-fbfc6587f-rkvjv   2/2     Running

why kubeflow-user-example-com and not kubeflow

per pboyd's direction: model registry must live in the user profile namespace for dashboard integration. deploying to kubeflow breaks the dashboard (hub#1045). hub upstream must stay namespace-agnostic (hub#2709 — closed).

namespace architecture

graph TB
    subgraph cluster["kubernetes cluster"]
        subgraph kf["namespace: kubeflow"]
            gw["kubeflow-gateway<br/>(istio ingress)"]
            dashboard["central dashboard"]
        end

        subgraph user["namespace: kubeflow-user-example-com"]
            subgraph registry["model registry"]
                mr_deploy["model-registry-deployment"]
                mr_db["model-registry-db"]
                mr_ui["model-registry-ui"]
                mr_vs["VirtualService: model-registry"]
                mr_ui_vs["VirtualService: model-registry-ui"]
                mr_dr["DestinationRule: model-registry-service"]
                mr_ui_dr["DestinationRule: model-registry-ui"]
                mr_authz["AuthorizationPolicy"]
            end

            subgraph catalog["model catalog"]
                mc_server["model-catalog-server"]
                mc_postgres["model-catalog-postgres"]
                mc_configmaps["ConfigMaps: sources, perf-data"]
            end
        end

        subgraph def["namespace: default"]
            empty["empty — no kubeflow components"]
        end

        gw -- "patched: kubeflow/kubeflow-gateway" --> mr_vs
        gw -- "patched: kubeflow/kubeflow-gateway" --> mr_ui_vs
        mr_vs --> mr_deploy
        mr_ui_vs --> mr_ui
        mr_deploy --> mr_db
        mc_server --> mc_postgres
    end

    style kf fill:#1a3a5c,stroke:#4a9eff,color:#ffffff
    style user fill:#2d4a1a,stroke:#6abf40,color:#ffffff
    style def fill:#4a1a1a,stroke:#ff4a4a,color:#ffffff
    style registry fill:#1a4a2d,stroke:#40bf6a,color:#ffffff
    style catalog fill:#4a3a1a,stroke:#bf9f40,color:#ffffff

Loading

note: the current upstream kubeflow/hub bundles model registry and model catalog as a single kustomize unit. this overlay deploys both to kubeflow-user-example-com. per the hub documentation, model catalog is a read-only discovery service that should logically be a cluster-wide singleton in kubeflow. splitting it into a separate kubeflow-namespace deployment is pending maintainer direction (discussed in this comment).

changes

file	change
applications/hub/overlays/kustomization.yaml	[NEW] overlay wrapping 4 upstream resources with namespace: kubeflow-user-example-com and 4 JSON6902 patch entries (6 operations)
example/kustomization.yaml	replace 4 upstream paths with single overlay reference
tests/model_registry_install.sh	use single overlay build, add namespace existence check, update all namespace references to kubeflow-user-example-com
tests/model_registry_test.sh	fix port-forward namespace, add trap-based cleanup
tests/model_catalog_install.sh	update namespace reference to kubeflow-user-example-com
tests/model_catalog_test.sh	update namespace reference to kubeflow-user-example-com
.github/workflows/model_registry_test.yaml	broaden continuous integration trigger path to applications/hub/**
.github/workflows/full_kubeflow_integration_test.yaml	add default namespace isolation guard that fails continuous integration if any pods exist in default

istio patches (6 operations across 4 entries)

setting namespace: kubeflow-user-example-com moves all resources out of kubeflow. six hardcoded istio references break and need patching:

gateway references (2): istio resolves gateway names relative to the virtualservice namespace. kubeflow-gateway lives in kubeflow (kf-istio-resources.yaml), so virtualservices in kubeflow-user-example-com must use kubeflow/kubeflow-gateway. this matches workspaces, kserve, and notebooks.

destination fully qualified domain names (4): services move to kubeflow-user-example-com with the overlay, so fully qualified domain names change from .kubeflow.svc.cluster.local to .kubeflow-user-example-com.svc.cluster.local.

#	resource	field	before	after
1	VirtualService/model-registry	gateways[0]	kubeflow-gateway	kubeflow/kubeflow-gateway
2	VirtualService/model-registry-ui	gateways[0]	kubeflow-gateway	kubeflow/kubeflow-gateway
3	VirtualService/model-registry	destination.host	...kubeflow.svc...	...kubeflow-user-example-com.svc...
4	DestinationRule/model-registry-service	host	...kubeflow.svc...	...kubeflow-user-example-com.svc...
5	VirtualService/model-registry-ui	destination.host	...kubeflow.svc...	...kubeflow-user-example-com.svc...
6	DestinationRule/model-registry-ui	host	...kubeflow.svc...	...kubeflow-user-example-com.svc...

what is not changed

• no upstream files in applications/hub/upstream/ modified (hub stays namespace-agnostic)
• no yaml manifests outside the overlay
• authorization policies reference istio-system service accounts — namespace-independent, no patch needed

continuous integration ordering verification

profile namespace kubeflow-user-example-com is created before hub install in both continuous integration paths:

• full integration: Create KF Profile (step 69) → Install Model Registry (step 91)
• standalone: Create KF Profile (step 51) → Install Model Registry (step 54)

cc @pboyd @christian-heusel

[github-actions]

Welcome to the Kubeflow Manifests Repository

Thanks for opening your first PR. Your contribution means a lot to the Kubeflow community.

Before making more PRs:
Please ensure your PR follows our Contributing Guide.
Please also be aware that many components are synchronizes from upstream via the scripts in /scripts.
So in some cases you have to fix the problem in the upstream repositories first, but you can use a PR against kubeflow/manifests to test the platform integration.

Community Resources:

• CNCF Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
• Community meetings: https://www.kubeflow.org/docs/about/community/

Thanks again for helping to improve Kubeflow.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR consolidates Hub-related deployments into a single Kustomize overlay and updates the local install + integration test scripts to operate in the profile namespace (kubeflow-user-example-com) instead of the shared kubeflow namespace.

Changes:

• Added applications/hub/overlays Kustomize overlay to deploy Hub components into kubeflow-user-example-com and patch Istio references accordingly.
• Updated install/test scripts to target kubeflow-user-example-com.
• Updated example kustomization and workflow path filters to reference the new overlay.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/model_registry_test.sh	Port-forward now targets the profile namespace service.
tests/model_registry_install.sh	Installs Hub via the new overlay and waits/logs in the profile namespace.
example/kustomization.yaml	Switches from listing individual Hub resources to the single overlay.
applications/hub/overlays/kustomization.yaml	New overlay setting namespace + Istio patching for cross-namespace gateway and service FQDNs.
.github/workflows/model_registry_test.yaml	Ensures CI reruns when the new overlay changes.

[juliusvonkohout]

1. Please add an asci diagram or so to explain what goes into which namespace. I thought we wanted to have a per customer namespace "kubeflow-user-example-com" model registry and cental catalog in "kubeflow". Why should we have multiple read only services that do exactly the same thing ? But as said please provide a diagram.
2. Where is the test in https://github.com/Raakshass/manifests/blob/fix/hub-default-namespace/.github/workflows/full_kubeflow_integration_test.yaml that fails if anyone added anything to the "default" namespace?

https://www.kubeflow.org/docs/components/hub/overview/#what-is-kubeflow-hub

[Raakshass]

hi julius, applied your workflow path suggestion in f7db495. addressing both points below.

1. namespace architecture

here is the exact deployment topology produced by this pr's overlay:

graph TB
    subgraph cluster["kubernetes cluster"]
        subgraph kf["namespace: kubeflow"]
            gw["kubeflow-gateway<br/>(istio ingress)"]
            dashboard["central dashboard"]
        end

        subgraph user["namespace: kubeflow-user-example-com"]
            subgraph registry["model registry"]
                mr_deploy["model-registry-deployment"]
                mr_db["model-registry-db"]
                mr_ui["model-registry-ui"]
                mr_vs["VirtualService: model-registry"]
                mr_ui_vs["VirtualService: model-registry-ui"]
                mr_dr["DestinationRule: model-registry-service"]
                mr_ui_dr["DestinationRule: model-registry-ui"]
                mr_authz["AuthorizationPolicy"]
            end

            subgraph catalog["model catalog"]
                mc_server["model-catalog-server"]
                mc_postgres["model-catalog-postgres"]
                mc_configmaps["ConfigMaps: sources, perf-data"]
            end
        end

        subgraph def["namespace: default"]
            empty["empty — no kubeflow components"]
        end

        gw -- "patched: kubeflow/kubeflow-gateway" --> mr_vs
        gw -- "patched: kubeflow/kubeflow-gateway" --> mr_ui_vs
        mr_vs --> mr_deploy
        mr_ui_vs --> mr_ui
        mr_deploy --> mr_db
        mc_server --> mc_postgres
    end

    style kf fill:#1a3a5c,stroke:#4a9eff,color:#ffffff
    style user fill:#2d4a1a,stroke:#6abf40,color:#ffffff
    style def fill:#4a1a1a,stroke:#ff4a4a,color:#ffffff
    style registry fill:#1a4a2d,stroke:#40bf6a,color:#ffffff
    style catalog fill:#4a3a1a,stroke:#bf9f40,color:#ffffff

Loading

key observations:

• the overlay sets namespace: kubeflow-user-example-com on all 4 upstream resources, so both model registry and model catalog deploy to the user profile namespace
• the 6 istio patches only target model registry and model registry ui — model catalog has no virtualservice, no destinationrule, and no gateway reference in this overlay. it is only reachable via direct cluster-internal service dns
• this matches the upstream hub bundling where catalog and registry ship as a single kustomize unit

**you are correct that model catalog should logically be a cluster-wide singleton in kubeflow, not duplicated per tenant. per the hub documentation, model catalog is a "read-only discovery service" that aggregates external sources, while model registry is a per-tenant metadata store for user-specific model versions and artifacts. they serve different purposes, but the current upstream packaging bundles them together.

to split model catalog into kubeflow without modifying upstream, i would need a second overlay in this repository. should i add that separation in this pr, or keep them bundled and split in a follow-up?

2. default namespace guard test

added in f7db495 the step Fail if there are resources in the "default" namespace runs kubectl get pods -n default --no-headers, dumps diagnostics on any findings, and exits non-zero to fail the build.

[juliusvonkohout]

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [juliusvonkohout]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment