Update kubeflow/trainer manifests from v2.2.0 (kubeflow#3413)

* Update kubeflow/trainer manifests from v2.2.0

Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>

* Update pull request paths in trainer_test.yaml

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* Remove patches for deployments in kustomization.yaml

Removed patches for jobset-controller-manager and kubeflow-trainer-controller-manager deployments.

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

* fixes

Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>

* fix(trainer): resolve webhook timeout in v2.2.0 CI (kubeflow#3415)

* fix(trainer): resolve webhook timeout in v2.2.0 CI

Root cause (proven from CI diagnostic logs):

1. Install: Redundant runtimes re-apply triggered unreachable webhook.
   Fix: Remove the step; runtimes already deployed before webhook
   registers.

2. Test: Istio sidecar intercepts outbound traffic from trainer
   controller to K8s API server on port 443. When API server calls
   jobset mutating webhook, the call times out because it routes
   through the trainer's Envoy sidecar.
   Fix: Add excludeOutboundPorts: 443 annotation to both trainer and
   jobset controller pod templates, alongside existing
   excludeInboundPorts: 9443.

3. Timing: kubectl rollout restart creates new jobset pod on a
   different node. kube-proxy endpoint propagation delay causes
   transient webhook unreachability.
   Fix: Add 30s sleep after restart for endpoint propagation. Add
   retry loop around TrainJob creation in test script.

Signed-off-by: Siddhant Jain <siddhantjainofficial26@gmail.com>

* fix(trainer): correct jobset-webhook podSelector labels

The jobset-webhook NetworkPolicy was incorrectly targeting pods with the label 'service.istio.io/canonical-name: jobset-controller-manager'. The jobset pods actually carry the label 'app.kubernetes.io/name: jobset'. This mismatch resulted in a default-deny rule dropping webhook validation requests, causing TrainJob deployments to time out. Patched the podSelector to match the correct standard labels.

Signed-off-by: Siddhant Jain <siddhantjainofficial26@gmail.com>

* fix: address review — remove non-essential changes

Per reviewer feedback, stripped all symptom patches that are not strictly needed now that the root cause (jobset-webhook NetworkPolicy podSelector mismatch) is fixed: - Removed excludeOutboundPorts annotations from both istio patches - Reverted trainer_test.sh to base (no duplicated cert waits, no retry loop, no timeout bump) - Restored runtimes build line in trainer_install.sh Only the NetworkPolicy fix and install.sh webhook cert waits remain.

Signed-off-by: Siddhant Jain <siddhantjainofficial26@gmail.com>

---------

Signed-off-by: Siddhant Jain <siddhantjainofficial26@gmail.com>

---------

Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>
Signed-off-by: Siddhant Jain <siddhantjainofficial26@gmail.com>
Co-authored-by: Siddhant Jain <149181251+Raakshass@users.noreply.github.com>

Author: Raakshass

.github/workflows/trainer_test.yaml

@@ -4,7 +4,7 @@ on:
     paths:
     - tests/install_KinD_create_KinD_cluster_install_kustomize.sh
     - .github/workflows/trainer_test.yaml
-    - applications/trainer/upstream/**
+    - applications/trainer/**
     - tests/trainer*
     - tests/istio*
     - tests/oauth2-proxy_install.sh

README.md

@@ -63,7 +63,7 @@ This repository periodically synchronizes all official Kubeflow components from
 | Component | Local Manifests Path | Upstream Revision | CPU (millicores) | Memory (Mi) |  PVC Storage (GB) |
 | - | - | - | - | - | - |
 | Training Operator | applications/training-operator/upstream | [v1.9.2](https://github.com/kubeflow/training-operator/tree/v1.9.2/manifests) | 3m | 25Mi | 0GB |
-| Trainer | applications/trainer/upstream | [v2.1.0](https://github.com/kubeflow/trainer/tree/v2.1.0/manifests) | 8m | 143Mi | 0GB |
+| Trainer | applications/trainer/upstream | [v2.2.0](https://github.com/kubeflow/trainer/tree/v2.2.0/manifests) | 8m | 143Mi | 0GB |
 | Notebook Controller | applications/jupyter/notebook-controller/upstream | [v1.10.0](https://github.com/kubeflow/kubeflow/tree/v1.10.0/components/notebook-controller/config) | 5m | 93Mi | 0GB |
 | PVC Viewer Controller | applications/pvcviewer-controller/upstream | [v1.10.0](https://github.com/kubeflow/kubeflow/tree/v1.10.0/components/pvcviewer-controller/config) | 15m | 128Mi | 0GB |
 | Tensorboard Controller | applications/tensorboard/tensorboard-controller/upstream | [v1.10.0](https://github.com/kubeflow/kubeflow/tree/v1.10.0/components/tensorboard-controller/config) | 15m | 128Mi | 0GB |

applications/trainer/overlays/kustomization.yaml

@@ -4,45 +4,3 @@ namespace: kubeflow-system
 
 resources:
 - ../upstream/overlays/kubeflow-platform
-
-patches:
-- target:
-    group: apps
-    version: v1
-    kind: Deployment
-    name: jobset-controller-manager
-  patch: |-
-    apiVersion: apps/v1
-    kind: Deployment
-    metadata:
-      name: jobset-controller-manager
-      namespace: kubeflow-system
-    spec:
-      template:
-        metadata:
-          annotations:
-            traffic.sidecar.istio.io/excludeInboundPorts: "9443"
-        spec:
-          securityContext:
-            seccompProfile:
-              type: RuntimeDefault
-- target:
-    group: apps
-    version: v1
-    kind: Deployment
-    name: kubeflow-trainer-controller-manager
-  patch: |-
-    apiVersion: apps/v1
-    kind: Deployment
-    metadata:
-      name: kubeflow-trainer-controller-manager
-      namespace: kubeflow-system
-    spec:
-      template:
-        metadata:
-          annotations:
-            traffic.sidecar.istio.io/excludeInboundPorts: "9443"
-        spec:
-          securityContext:
-            seccompProfile:
-              type: RuntimeDefault

applications/trainer/upstream/base/crds/trainer.kubeflow.org_clustertrainingruntimes.yaml

@@ -42,3 +42,8 @@ certManagement:
 clientConnection:
   qps: 50
   burst: 100
+
+statusServer:
+  port: 10443
+  qps: 5
+  burst: 10

applications/trainer/upstream/base/crds/trainer.kubeflow.org_trainingruntimes.yaml

@@ -24,31 +24,50 @@ spec:
         - name: manager
           image: ghcr.io/kubeflow/trainer/trainer-controller-manager
           securityContext:
+            readOnlyRootFilesystem: true
             allowPrivilegeEscalation: false
             runAsNonRoot: true
             capabilities:
               drop:
                 - ALL
             seccompProfile:
               type: RuntimeDefault
+
+          ports:
+            - name: health
+              containerPort: 8081
+              protocol: TCP
+            - name: metrics
+              containerPort: 8443
+              protocol: TCP
+            - name: webhook
+              containerPort: 9443
+              protocol: TCP
+            - name: status-server
+              containerPort: 10443
+              protocol: TCP
+
           volumeMounts:
             - mountPath: /tmp/k8s-webhook-server/serving-certs
               name: cert
               readOnly: true
+
           livenessProbe:
             httpGet:
               path: /healthz
-              port: 8081
+              port: health
             initialDelaySeconds: 15
             periodSeconds: 20
             timeoutSeconds: 3
+
           readinessProbe:
             httpGet:
               path: /readyz
-              port: 8081
+              port: health
             initialDelaySeconds: 10
             periodSeconds: 15
             timeoutSeconds: 3
+
       serviceAccountName: kubeflow-trainer-controller-manager
       volumes:
         - name: cert
@@ -63,11 +82,15 @@ metadata:
 spec:
   ports:
     - name: monitoring-port
-      port: 8080
-      targetPort: 8080
+      port: 8443
+      targetPort: metrics
     - name: webhook-server
       port: 443
       protocol: TCP
-      targetPort: 9443
+      targetPort: webhook
+    - name: status-server
+      port: 10443
+      protocol: TCP
+      targetPort: status-server
   selector:
     app.kubernetes.io/component: manager

applications/trainer/upstream/base/crds/trainer.kubeflow.org_trainjobs.yaml

@@ -2,3 +2,5 @@ resources:
   - role.yaml
   - role_binding.yaml
   - service_account.yaml
+  - public_configmap_role.yaml
+  - public_configmap_role_binding.yaml

applications/trainer/upstream/base/manager/controller_manager_config.yaml

@@ -0,0 +1,15 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: kubeflow-trainer-public
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - configmaps
+  resourceNames:
+  - kubeflow-trainer-public
+  verbs:
+  - get
+  - list
+  - watch