test(kserve): add raw deployment test, trim authentication and authorization documentation, print prediction output

[Raakshass]

✏️ Summary of Changes

Addresses the three follow-up items from PR #3352 as requested by @juliusvonkohout in this comment:

1. Trim KServe authentication documentation (common/oauth2-proxy/README.md)

• Remove verbose subsections: CI fallback YAML block, path-based/host-based routing table, Models Web App Authentication section, CI Test Coverage table (~70 lines removed)
• Remove speculative Architecture Analysis section and orphaned [^6] footnote (review)
• Add concise 4-line note about the CI rules: [{}] workaround
• Add CI workflow and test script links to the KServe Authentication heading (review)
• Add sentence about host-based and path-based routing support with kserve/kserve#5090 reference
• Preserve core traffic flow diagram and AuthorizationPolicy configuration

2. Add test for raw deployment mode (tests/kserve_test.sh)

• Test 7: Deploy an sklearn model with serving.kserve.io/deploymentMode: RawDeployment annotation
• AuthorizationPolicy uses the universal serving.kserve.io/inferenceservice label (works for both serverless and raw deployment modes, unlike Knative-specific labels)
• Host-based routing only — path-based routing for raw deployment requires ingressPathTemplate which is not yet merged upstream
• Verifies authentication enforcement: unauthenticated requests are rejected (403/302), authenticated requests succeed
• Reference: inferenceservice.yaml ingressClassName config

3. Improve test output and CI log readability

• Added prediction result logging in kserve_sklearn_test.py before assertion (review)
• Truncated response body to 200 characters in all echo statements to prevent HTML pages from flooding CI logs (review)
• All 13 curl test outputs now show HTTP <code> | <truncated body> format

4. Security hardening and cleanup

• Added securityContext block to secure-model-predictor Knative Service fixture (allowPrivilegeEscalation: false, runAsNonRoot: true, seccompProfile: RuntimeDefault, capabilities: drop: ["ALL"]) (review)
• Fix pre-existing em dashes in comments (Unicode to ASCII)
• Add missing kubectl delete for allow-isvc-sklearn AuthorizationPolicy cleanup (pre-existing gap from refactor(tests): consolidate KServe test infrastructure into two-file pattern #3352)

📦 Dependencies / Related Issues

• Follow-up to: refactor(tests): consolidate KServe test infrastructure into two-file pattern #3352
• Raw deployment reference: kserve/kserve#5090 (open, not yet merged)
• Config reference: inferenceservice.yaml#L389
• Reviewer comments addressed:
  • Merge comment requesting all 3 items
  • Raw deployment follow-up comment
  • Documentation update request

✅ Contributor Checklist

• I have tested these changes with kustomize. See Installation Prerequisites.
• All commits are signed-off to satisfy the DCO check.
• I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).

[github-actions]

Welcome to the Kubeflow Manifests Repository

Thanks for opening your first PR. Your contribution means a lot to the Kubeflow community.

Before making more PRs:
Please ensure your PR follows our Contributing Guide.
Please also be aware that many components are synchronizes from upstream via the scripts in /scripts.
So in some cases you have to fix the problem in the upstream repositories first, but you can use a PR against kubeflow/manifests to test the platform integration.

Community Resources:

• CNCF Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
• Community meetings: https://www.kubeflow.org/docs/about/community/

Thanks again for helping to improve Kubeflow.

[juliusvonkohout]

Also for pytest a bit more than

+ python -m pytest /home/runner/work/manifests/manifests/tests/kserve_sklearn_test.py -vs --log-level info
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0 -- /opt/hostedtoolcache/Python/3.12.12/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/manifests/manifests
Notice:  A new release of pip is available: 26.0 -> 26.0.1
Notice:  To update, run: pip install --upgrade pip
collecting ... collected 1 item
tests/kserve_sklearn_test.py::test_sklearn_kserve PASSED
======================== 1 passed in 109.96s (0:01:49) =========================

could be useful. Can you log the prediction results ?

[juliusvonkohout]

Also

Test 2a path-based (with token): HTTP 404 | 
+ '[' 404 '!=' 200 ']'
+ '[' 404 '!=' 404 ']'
+ HOST_HEADER='Host: isvc-sklearn.kubeflow-user-example-com.example.com'
++ curl -s -w '\n%{http_code}' -H 'Host: isvc-sklearn.kubeflow-user-example-com.example.com' -H 'Content-Type: application/json' http://localhost:8080/v1/models/isvc-sklearn:predict -d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'
+ RESPONSE='
<!DOCTYPE html>
<html lang="en" charset="utf-8">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <title>Sign In</title>
    <link rel="stylesheet" href="/oauth2/static/css/bulma.min.css">
    <style>
      body {
...

looks like its failing or outputting the oauth2-proxy login page for some reason.

[juliusvonkohout]

And i see

Test 3 models web application (unauth): HTTP 403 | {"log":"User 'system:serviceaccount:default:default' is not authorized to list serving.kserve.io/v1beta1/inferenceservices in namespace 'kubeflow-user-example-com'","status":403,"success":false,"user":"system:serviceaccount:default:default"}
Models Web Application: Token from unauthorized ServiceAccount cannot list InferenceServices in kubeflow-user-example-com namespace.
+ kubectl apply -f -
Warning: Kubernetes default value is insecure, Knative may default this to secure in a future release: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation, spec.template.spec.containers[0].securityContext.capabilities, spec.template.spec.containers[0].securityContext.runAsNonRoot, spec.template.spec.containers[0].securityContext.seccompProfile
service.serving.knative.dev/secure-model-predictor created

so the inference service might be missing a proper security context.

[juliusvonkohout]

Thank you for the PR, I added comments

[Raakshass]

Also

Test 2a path-based (with token): HTTP 404 | 
+ '[' 404 '!=' 200 ']'
+ '[' 404 '!=' 404 ']'
+ HOST_HEADER='Host: isvc-sklearn.kubeflow-user-example-com.example.com'
++ curl -s -w '\n%{http_code}' -H 'Host: isvc-sklearn.kubeflow-user-example-com.example.com' -H 'Content-Type: application/json' http://localhost:8080/v1/models/isvc-sklearn:predict -d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'
+ RESPONSE='
<!DOCTYPE html>
<html lang="en" charset="utf-8">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <title>Sign In</title>
    <link rel="stylesheet" href="/oauth2/static/css/bulma.min.css">
    <style>
      body {
...

looks like its failing or outputting the oauth2-proxy login page for some reason.

This is expected behavior. The Sign In HTML is the oauth2-proxy response for
unauthenticated requests -- oauth2-proxy intercepts at the ingress gateway
and returns a 302 redirect to its login page. The test checks the HTTP
status code (302 = blocked), not the response body, so the test passed
correctly.

The full HTML body in the logs can be reduced by truncating the response
body to the first 200 characters in the echo statements. I'll include this
in the next commit to keep the logs readable.

[Raakshass]

And i see

Test 3 models web application (unauth): HTTP 403 | {"log":"User 'system:serviceaccount:default:default' is not authorized to list serving.kserve.io/v1beta1/inferenceservices in namespace 'kubeflow-user-example-com'","status":403,"success":false,"user":"system:serviceaccount:default:default"}
Models Web Application: Token from unauthorized ServiceAccount cannot list InferenceServices in kubeflow-user-example-com namespace.
+ kubectl apply -f -
Warning: Kubernetes default value is insecure, Knative may default this to secure in a future release: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation, spec.template.spec.containers[0].securityContext.capabilities, spec.template.spec.containers[0].securityContext.runAsNonRoot, spec.template.spec.containers[0].securityContext.seccompProfile
service.serving.knative.dev/secure-model-predictor created

so the inference service might be missing a proper security context.

Yes. The secure-model-predictor is a test-only Knative Service
(using gcr.io/knative-samples/helloworld-go), not a real InferenceService,
but it should still follow security best practices.

I'll add the proper securityContext block in the next commit:

securityContext:
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop: ["ALL"]

[Raakshass]

Also for pytest a bit more than

+ python -m pytest /home/runner/work/manifests/manifests/tests/kserve_sklearn_test.py -vs --log-level info
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0 -- /opt/hostedtoolcache/Python/3.12.12/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/manifests/manifests
Notice:  A new release of pip is available: 26.0 -> 26.0.1
Notice:  To update, run: pip install --upgrade pip
collecting ... collected 1 item
tests/kserve_sklearn_test.py::test_sklearn_kserve PASSED
======================== 1 passed in 109.96s (0:01:49) =========================

could be useful. Can you log the prediction results ?

Done -- added explicit prediction logging in kserve_sklearn_test.py
before the assertion. CI output will now show:

INFO Prediction result: [1, 1] (expected [1, 1])

[Copilot]

Pull request overview

This PR addresses follow-up items from PR #3352, focusing on improving KServe test infrastructure, documentation, and security hardening. The changes include adding a raw deployment mode test, improving test output readability, trimming verbose documentation, and hardening security contexts.

Changes:

• Added Test 7 to verify KServe raw deployment mode with host-based routing and authentication enforcement
• Improved CI log readability by truncating curl response bodies to 200 characters and logging HTTP codes for all 13 test requests
• Trimmed ~70 lines of verbose content from oauth2-proxy README while preserving essential information
• Added security context to Knative Service test fixture and fixed cleanup gaps

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
tests/kserve_test.sh	Added raw deployment test (Test 7), unified curl response handling with body truncation for all tests, added security context to Knative Service fixture, fixed missing cleanup commands
tests/kserve_sklearn_test.py	Added prediction result logging before assertion for better debugging, fixed pytest log level flag
common/oauth2-proxy/README.md	Trimmed verbose sections, added CI workflow links, added concise note about routing modes and CI workarounds

[juliusvonkohout]

Thank you
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [juliusvonkohout]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment