[feature] Enhance support and visualization for pod lifecycle failure in Kubeflow Pipelines

[alyssacgoins]

Feature Area

/area frontend
/area backend

What feature would you like to see?

A pipeline in Kubeflow Pipelines (KFP) fails for one of two reasons: the user-defined pipeline script returns an error, or a pod lifecycle failure occurs. The latter failure can be classified according to lifecycle stage; a failure is either at the provisioning level (e.g., ImagePullBackOff or Unschedulable), runtime level (e.g., CrashLoopBackOff or OOMKilled), or node level (e.g., NodeLost or Preempted).

KFP currently provides limited support when a pipeline run hits a pod lifecycle failure. While the Kubernetes CLI allows users to view the status of their pipeline pods in real time on the terminal, the KFP UI currently provides only minimal visualization. When a pod lifecycle failure occurs, the KFP UI displays a pipeline frozen at the current pod – not progressing, finishing or failing. The cause of failure is not displayed. This creates a confusing and frustrating user experience. KFP intends to function as a Kubernetes abstraction for data scientists and AI engineers, meaning that not all KFP users are experienced with Kubernetes. Requiring use of Kubernetes tools and skills leaks that abstraction.

The solution here proposes error timeouts and enhanced visualization. It would be helpful to parameterize time limits on the three categories of pod lifecycle failure outlined above, as environment variables on the API server. A default time limit of one hour would be provided, and users could optionally specify time limits per pod status (e.g. ImagePullBackOff). In addition to the time management component, it would enhance user experience to visually log these failures in the UI.
Pod lifecycle failures should be logged and displayed in the UI in the same way pipeline script failures are displayed, in order to provide users with real-time pod lifecycle updates.

What is the use case or pain point?

Users executing pipeline runs with pod lifecycle failures are unable to view pipeline failure in the UI, and unable to manage failure timeout.

Is there a workaround currently?

The current workaround to lifecycle failure timeout is to manually delete a pipeline run. The current workaround to the limited UI visualization is kubectl get pods -w using the Kubernetes CLI in the terminal. There is no workaround to failure visualization directly within the UI.

---

Love this idea? Give it a 👍.

[alyssacgoins]

/assign for gsoc project above ^

[CREVIOS]

Hi @alyssacgoins, I'm interested in this as a potential GSoC project.

I went through the KFP frontend code and the current pipeline run visualization. As described, when a pod hits something like ImagePullBackOff or OOMKilled, the UI just shows the step frozen with no indication of what went wrong. Users have to switch to kubectl to figure out the failure reason.

My initial thinking on the implementation:

Backend side (Go):

• The backend already watches pod status via the Kubernetes API. The gap is surfacing pod-level failure reasons (from pod.status.containerStatuses[].state.waiting.reason and state.terminated.reason) as structured data in the run detail response
• Parameterized timeouts per pod status would need a config map or run-level setting — e.g. ImagePullBackOff: 5m, Unschedulable: 10m, with sensible defaults

Frontend side (TypeScript):

• The run detail view can render pod failure reasons similarly to how script errors are shown today — a red status badge with the failure reason and a collapsible section showing the pod events timeline
• For the timeout behavior, a progress indicator showing "waiting for pod scheduling (2m/10m)" would give users visibility

I'm comfortable with both TypeScript and Go. Would it help to start with a small scoping PR for just the backend changes to surface pod failure reasons first?

[Ashish241]

Hi @alyssacgoins, I've been working on this issue and have opened PR #12989 as a starting point, it defines the API server environment variables for the three failure categories (Provisioning, Runtime, and Node). Would love your feedback when you get a chance!

[Harshdhall01]

Hi @alyssacgoins! I'm a GSoC applicant interested in this project. I have React/TypeScript experience and I'm building an MVP demonstrating pod failure visualization on the frontend with configurable timeout support on the backend. Quick question — for the frontend visualization, should pod failure reasons be displayed inline within the pipeline graph, or in a separate side panel similar to how script errors are shown currently?

[Karman580]

Hi @alyssacgoins ,

I’m Karman Singh Talwar, a Computer Engineering student interested in contributing to Kubeflow for GSoC 2026, and this project on improving pod lifecycle failure handling and visualization in Kubeflow Pipelines particularly caught my interest.

From reviewing the issue and exploring parts of the KFP codebase, my understanding is that the current gap lies mainly in surfacing pod-level lifecycle failures from Kubernetes into the KFP abstraction layer and then reflecting those states meaningfully in the UI. Specifically:

Backend (Go / API server)
• Extract lifecycle failure reasons from pod.status.containerStatuses[].state.waiting.reason and state.terminated.reason.
• Classify them into the proposed categories: provisioning (e.g., ImagePullBackOff, Unschedulable), runtime (e.g., CrashLoopBackOff, OOMKilled), and node-level (e.g., NodeLost, Preempted).
• Surface these states through the pipeline run / task status APIs, allowing the frontend to display structured failure information instead of a stalled pipeline node.
• Introduce configurable timeout controls (likely via API server environment variables or config maps) to prevent runs from remaining indefinitely in lifecycle wait states.

Frontend (TypeScript / UI)
• Extend the run detail visualization so lifecycle failures are surfaced similarly to script-level errors, with clear status badges and failure reasons.
• Optionally display pod lifecycle progress indicators (e.g., scheduling wait time vs timeout threshold) to improve observability for users unfamiliar with Kubernetes.

I’m currently setting up the Kubeflow Pipelines development environment and reviewing how pod state propagates through the backend services and into the UI run graph.

As an initial step, I was considering starting with a backend-focused PR to expose structured pod lifecycle failure reasons through the run/task APIs, which would then enable the frontend visualization layer.

Would this be a reasonable first scope for contribution?

Thanks for proposing and mentoring this feature , it seems like a valuable improvement to the KFP user experience.

[wayngo]

Hi @alyssacgoins! I’m Wayne, a student developer interested in contributing to this project for GSoC 2026.

I’ve spent time working on internal codebases for my school using Go and TypeScript, so I’m particularly interested in how we can bridge the gap between the Kubernetes backend and the user experience. The "abstraction leak" mentioned in the description is a great point users shouldn't need to use kubectl just to see why a pod failed.

I see that @Ashish241 has already opened a PR (#12989) regarding environment variables for timeouts. To build on that, I’m interested in focusing my proposal on the UI visualization and the API data contract needed to ensure the failure reasons are surfaced clearly in the KFP graph view.

I’m currently setting up my local environment with kind. Before I finalize my proposal, do you think it’s better to focus on a centralized "Error Dashboard" within KFP, or should the priority be inline notifications within the pipeline graph itself?