kserve enforce authentication by madmecodes · Pull Request #3180 · kubeflow/community-distribution

madmecodes · 2025-07-01T13:54:14Z

✏️ Summary of Changes

KServe JWT Authentication PR Analysis

Core Security Features

JWT-based authentication for cluster-local-gateway: Added RequestAuthentication and AuthorizationPolicy to secure the gateway that KServe uses by default.
Two authentication overlays:
- m2m-auth: Basic JWT authentication requiring valid Kubernetes service account tokens.
- m2m-auth-strict: Enhanced version with namespace-level isolation.
Comprehensive test suite: Multiple test scripts to validate authentication scenarios.
Documentation: Added KSERVE_JWT_AUTHENTICATION.md with a detailed implementation guide.

Technical Implementation Details

RequestAuthentication: Validates JWT tokens from Kubernetes API server with proper issuer configuration.
AuthorizationPolicy (DENY): Blocks all requests without valid JWT principals, except health checks.
AuthorizationPolicy (ALLOW): Permits requests with valid JWT principals and exempts health check endpoints.
External access support: Example configurations for secure external access via istio-ingressgateway.
Namespace isolation examples: Templates for restricting access to same-namespace or explicit cross-namespace.

2. How It Addresses the Core Security Issue

The implementation directly addresses issue #2811 by:

Closing the authentication gap: Previously, cluster-local-gateway had no authentication while istio-ingressgateway had oauth2-proxy.
Consistent security model: Both gateways now require authentication (JWT for cluster-local, oauth2-proxy for ingress).
Default secure configuration: Authentication is enforced by default in the knative-cni install script.
Flexible authorization: Supports both permissive (any valid JWT) and strict (namespace-isolated) modes.

The PR represents a significant security improvement but leaves room for further enhancements based on community feedback and production usage patterns.

📦 Dependencies

None

🐛 Related Issues

#2811

✅ Contributor Checklist

I have tested these changes with kustomize. See Installation Prerequisites.
All commits are signed-off to satisfy the DCO check.
I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).

You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

juliusvonkohout · 2025-07-08T14:11:16Z

@@ -0,0 +1,66 @@
+# KServe External Access Configuration


lets move this to the other kserve test files. Do not create a new /examples folder.

juliusvonkohout · 2025-07-08T14:13:09Z

+# Apply cluster-local-gateway with JWT authentication (with retry)
+echo "Applying cluster-local-gateway JWT authentication policies..."
+for ((i=1; i<=3; i++)); do
+    if kustomize build common/istio/cluster-local-gateway/overlays/m2m-auth | kubectl apply -f -; then
+        echo "cluster-local-gateway JWT auth applied successfully"
+        break
+    else
+        echo "Attempt $i failed to apply cluster-local-gateway JWT auth, retrying..."
+        sleep 5
+    fi
+done
+


why are you adding this ?
kustomize build common/istio/cluster-local-gateway/overlays/m2m-auth | kubectl apply -f - should be enough instead of 12 lines

juliusvonkohout · 2025-07-08T14:14:14Z

@@ -0,0 +1,209 @@
+#!/bin/bash


tests/knative_authENTICATION_test.sh

juliusvonkohout · 2025-07-08T14:16:37Z

+    # Wait for InferenceService to be ready
+    log_info "Waiting for InferenceService to be ready..."
+    kubectl wait --for=condition=Ready inferenceservice/secure-sklearn -n $PRIMARY_NAMESPACE --timeout=300s || {
+        log_info "InferenceService not ready, continuing with tests..."


Lets remove most of the logging stuff. We just want to fail directly if something is wrong.

juliusvonkohout · 2025-07-08T14:16:59Z

+
+    # Wait for InferenceService to be ready
+    log_info "Waiting for InferenceService to be ready..."
+    kubectl wait --for=condition=Ready inferenceservice/secure-sklearn -n $PRIMARY_NAMESPACE --timeout=300s || {


300s is a bit much, lets use 180s

juliusvonkohout · 2025-07-08T14:21:03Z

@@ -0,0 +1,257 @@
+#!/bin/bash


Please fix the filename here as well

juliusvonkohout · 2025-07-08T14:22:21Z

@@ -0,0 +1,148 @@
+#!/bin/bash


what is the difference to the other tests? I think also the filename is not clear enough to explain what you are doing.

juliusvonkohout · 2025-07-08T14:23:21Z

@@ -0,0 +1,58 @@
+#!/bin/bash


tests/kserve_setup_external_access.sh

juliusvonkohout · 2025-07-08T14:26:01Z

Please also fix the tests.

orfeas-k · 2025-07-18T10:35:28Z

Does this also secure from this issue mentioned in KServe docs?

Currently when activator is on the request path, it is not able to check the originated namespace or original identity due to the net-istio issue.

My understanding is that this is actually not feasible to tackle on kubeflow's side, so users will still be able to access ISVCs in other users' namespaces, when activator intercepts them.

madmecodes · 2025-07-18T10:43:01Z

Does this also secure from this issue mentioned in KServe docs?

Currently when activator is on the request path, it is not able to check the originated namespace or original identity due to the net-istio issue.

My understanding is that this is actually not feasible to tackle on kubeflow's side, so users will still be able to access ISVCs in other users' namespaces, when activator intercepts them.

I agree, the core problem: When Knative's activator is in the request path (during cold starts or scale-to-zero), it bypasses namespace-based JWT authentication because the activator has broad access permissions and the original request identity is lost.

From the issue: knative-extensions/net-istio#554 "services in different namespace will be able to communicate with each other whenever activator is in path, because activator is allowed to access any service"

I also thinnk this is not fixable at the manifests level - it's an architectural limitation that requires changes to Knative/KServe itself.

madmecodes · 2025-07-18T10:45:54Z

Does this also secure from this issue mentioned in KServe docs?

Currently when activator is on the request path, it is not able to check the originated namespace or original identity due to the net-istio issue.

My understanding is that this is actually not feasible to tackle on kubeflow's side, so users will still be able to access ISVCs in other users' namespaces, when activator intercepts them.

So namespace level wont be possible, but we can try enforcing JWT, atleast that is doable, but im facing errors hard to find whats causing the tests failing.

orfeas-k · 2025-07-18T10:51:48Z

IMO It 'd be good to mention this architectural limitation in the new setup/overlay provided, given that people using it should be aware of this quite important limitation.

madmecodes · 2025-07-18T14:53:49Z

IMO It 'd be good to mention this architectural limitation in the new setup/overlay provided, given that people using it should be aware of this quite important limitation.

Like in the manifest Readme for Istio, we should update that?

madmecodes · 2025-07-18T17:21:48Z

@juliusvonkohout can you please re run these 2 tests?

juliusvonkohout · 2025-07-21T08:54:39Z

@juliusvonkohout can you please re run these 2 tests?

You should be able to do so with

/retest

if not we have to add you as member to Kubeflow. Do you mind creating a PR to add the 4 GSOC students as member to the kubeflow organization @madmecodes ?

orfeas-k · 2025-07-21T09:51:45Z

Like in the manifest Readme for Istio, we should update that?

Yep exactly, preferably with sth like a warning box if it's aligned with the style of readmes in this repo

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

juliusvonkohout · 2025-07-28T09:34:28Z

Thank you, that significantly improves the status quo.

/lgtm
/approve

google-oss-prow · 2025-07-28T09:34:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [juliusvonkohout]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow Bot added do-not-merge/work-in-progress size/XXL labels Jul 1, 2025

google-oss-prow Bot requested review from juliusvonkohout and kimwnasptd July 1, 2025 13:54

juliusvonkohout linked an issue Jul 1, 2025 that may be closed by this pull request

Enable and document for Kubeflow 1.10 Kserve secure inferencing from inside and outside the cluster with tokens #2811

Closed

7 tasks

juliusvonkohout assigned madmecodes Jul 1, 2025

madmecodes force-pushed the kserve-auth-new branch 2 times, most recently from 2416e85 to a2b976e Compare July 1, 2025 18:28

juliusvonkohout reviewed Jul 8, 2025

View reviewed changes

Comment thread tests/setup_external_access.sh

@@ -0,0 +1,58 @@

#!/bin/bash

juliusvonkohout Jul 8, 2025

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests/kserve_setup_external_access.sh

juliusvonkohout marked this pull request as ready for review July 15, 2025 15:09

google-oss-prow Bot removed the do-not-merge/work-in-progress label Jul 15, 2025

google-oss-prow Bot requested a review from juliusvonkohout July 15, 2025 15:09

google-oss-prow Bot added size/XL and removed size/XXL labels Jul 18, 2025

madmecodes and others added 21 commits July 28, 2025 10:46

ủpdate: sleep time to 60sec

4b30184

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

ủpdate: failing policy installing with retry

2d23e08

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

update: added debug cluster things

22cdc50

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

update: debugging in CI

fd9ecf7

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

update: kserve tests

aa75c55

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

remove wrong file pushed

6037fad

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

test: kserve rename and workflow update

00ccc6e

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

test: fix correct host

2114be0

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

test: if not 403 with valid token its a success

7b077b6

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

remove: debugging echos

8e4740d

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

update: readme added cross namespace limitation

d050fd1

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

update: gitignore changes reverted

965ffea

Signed-off-by: madmecodes <ayushguptadev1@gmail.com>

Rename kserve_jwt_auth_test.yaml to kserve_jwt_authentication_test.yaml

a6e8912

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Update kserve_secure_test.yaml

27514fe

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Update kserve_secure_test.yaml

a73a3b1

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Rename knative-cni_install.sh to knative_install.sh

a5bca06

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Update full_kubeflow_integration_test.yaml

41e90ce

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Update kserve_test.yaml

32ec9af

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Update kserve_jwt_authentication_test.yaml

4f65f43

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Update kserve_secure_test.yaml

0c45921

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

Update kserve_test.yaml

fb53c49

Signed-off-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com>

juliusvonkohout force-pushed the kserve-auth-new branch from 3298b84 to fb53c49 Compare July 28, 2025 08:46

google-oss-prow Bot assigned juliusvonkohout Jul 28, 2025

google-oss-prow Bot added the lgtm label Jul 28, 2025

google-oss-prow Bot added the approved label Jul 28, 2025

google-oss-prow Bot merged commit 560c60a into kubeflow:master Jul 28, 2025
29 checks passed

madmecodes mentioned this pull request Oct 2, 2025

Feature: Add Istio Ambient Mode Support via Overlay Method #3246

Merged

3 tasks

grudloffev mentioned this pull request Feb 12, 2026

Update VirtualService conflict troubleshooting for KServe path-based Routing #3263

Closed

8 tasks

Conversation

madmecodes commented Jul 1, 2025

✏️ Summary of Changes

KServe JWT Authentication PR Analysis

2. How It Addresses the Core Security Issue

📦 Dependencies

🐛 Related Issues

✅ Contributor Checklist

Uh oh!

juliusvonkohout Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout commented Jul 8, 2025

Uh oh!

orfeas-k commented Jul 18, 2025

Uh oh!

madmecodes commented Jul 18, 2025

Uh oh!

madmecodes commented Jul 18, 2025

Uh oh!

orfeas-k commented Jul 18, 2025

Uh oh!

madmecodes commented Jul 18, 2025

Uh oh!

madmecodes commented Jul 18, 2025

Uh oh!

juliusvonkohout commented Jul 21, 2025

Uh oh!

orfeas-k commented Jul 21, 2025

Uh oh!

juliusvonkohout commented Jul 28, 2025

Uh oh!

google-oss-prow Bot commented Jul 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

juliusvonkohout Jul 8, 2025 •

edited

Loading

juliusvonkohout Jul 8, 2025 •

edited

Loading