Protect KFServing endpoint

[Jeffwan]

/kind feature

Currently, a kfserving-gateway is used for kfserving inference endpoints. It uses LoadBalancer and all endpoints are publicly available without protection.

This doesn't meet requirement for production grade cluster. We need a way to project these endpoints. Seems currently probe is a blocker to inference behind authn layer.

Describe the solution you'd like

Long term, I think there're two options. (assume we can probe issue fixed)

1. Remove kfserving-gateway, reuse istio-ingressgateway which means requests from external need to get authenticated. I think both IAP (GCP), Coginito/OIDC (AWS) supports programmatic authentication. I am not sure about Dex. The advantage is this solution reuse AuthN and AuthZ from existing infra.

2. Still have separate gateway for kfserving. Leave implementation to different vendors. User could build authentication on top of it. For example, AWS can replace service with ingress and LB level authentication for it. The reason we don't reuse istio-gateway is because we can have other authentication strategy for kfserving. For example, each user can request different APIKeys for different models. etc.

3. Have a middleware for kfserving to manage API Keys. Not sure if there's existing solution on Kubernetes. This sounds like very common use case.

Anything else you would like to add:
Pipeline SDK will have similar issue, we can consider this together.

The user experience should be simple enough, client can get clientId and secret to refresh token or just use an assigned token to make call directly.

Solution needs to be latency optimized.

Related Issue:
kubeflow/kfctl#140
kubeflow/kubeflow#4912

@yuzisun @ellis-bigelow @animeshsingh @jlewi @cliveseldon

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.88

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[ellistarn]

I would definitely lean option 1. Our default install should provide a single gateway that is configured correctly for kubeflow. Multiple gateways creates confusion and potential security holes. Option 3 seems like a non-starter. I'd prefer that we aren't in the business of authentication. Curious to hear what others thing.

[jlewi]

+1 to option 1.

We should try to follow the pattern of "being secure" by default; i.e. we should not be exposing services publicly without authentication by default.

[yuzisun]

/cc @krishnadurai
KFServing ingress gateway is a temporary walk around for the knative probing issue knative/serving#6962, now they changed to the best effort probe instead of hard fail when getting HTTP 401/403 since probes are not backed up by credentials.

In order to use Kubeflow ingress gateway(should be simple config change), we first need to wait for above knative bug fix released and also we should find a solution to support programming credentials which would be the real use cases for KFServing, for some scenarios we may need api keys for different inference services.

[krishnadurai]

Adding to @yuzisun 's explanation, I'll link the history of issues to provide some context around our current implementation and how it is linked to Knative's probing mechanism.

Since Istio ingressgateway is auth enabled, it was leading to the following issue:
#668

This was due to the probing mechanism of KNative istio-networking probe requests through the gateway which didn't have credentials for it.
Highlighted here:
knative/serving#6829

This lead to a workaround implementation with a separate gateway for 1.0 Kubeflow release.

Marked by issue: kubeflow/community-distribution#924

As soon as KNative releases its next version with a non-request based probing mechanism, we plan to change the gateway back to istio-ingressgateway.