Distributions readiness for KF 1.5

[kimwnasptd]

Prev issue #2038

Distribution testing phase - handbook

The goal of this issue is to track the progress of distributions alongside the 1.5 release, and coordinate our communications. First goal is to expose any issues we will be bumping into here, so that all distros can keep an eye on issues that arise.

While we hope all distros would manage to be ready when the KF 1.5 release is out, this is sometimes impossible to achieve. In this issue we want to both keep track of the progress of distributions, towards the KF 1.5 release, but also which of the distros will be working on KF 1.5 even if they can't meet the KF 1.5 deadline.

Without further ado, here's the list of distros we have in mind:

Distribution	Representatives	State
Arrikto EKF	@kimwnasptd	❌
Arrikto MiniKF	@kimwnasptd	❌
Azure	❓	❌
AWS	@surajkota	❌
Charmed Kubeflow	@DomFleischmann	❌
Google Cloud	@zijianjoy	✅
IBM	@yhwang	✅
Nutanix	@johnugeorge	✅
Kubeflow with Argo CD	@davidspek	❌
Openshift	@nakfour @LaVLaS	❌

So lets use this issue to expose our state while testing the KF 1.5 release, and also give heads up to users about the progress of distros with the KF 1.5 release

[kimwnasptd]

We urge everyone to start their testing from the latest v1.5.0-rc.1 manifests tag. If anyone bumps into a problem, please open an issue and add a comment here as well so that we can all by in sync.

Regarding Arrikto's plans for the KF 1.5 release, we are targeting to also have our products ready for the deadline. But even if we don't manage, we will still be testing these following weeks and reporting bugs.

[kimwnasptd]

And also
cc @kubeflow/release-team

[zijianjoy]

Hello @kimwnasptd , which model-web-app does central dashboard integrate with? There are KFserving and KServe. I am curious how to configure between these two web apps in the central dashboard.

[surajkota]

Created a tracking issue for AWS distribution work - awslabs/kubeflow-manifests#91

We are targeting to complete Generic/Vanilla Kubeflow i.e. as-is from this repository working on EKS as part of distribution testing phase. Other features and release will follow

[zijianjoy]

Hello Kimonas, I would like to provide an update which requires changes to manifests as we are validating the Google Cloud distribution.

1. Update KFP to v1.8.1-rc.0: https://github.com/kubeflow/pipelines/releases/tag/1.8.1-rc.0. This includes only fixes and no feature.
2. I encountered issues when deploying kfserving endpoint using mnist sample. The way I resolved this issue is by running the following command:

kubectl patch mutatingwebhookconfiguration inferenceservice.serving.kubeflow.org --patch '{"webhooks":[{"name": "inferenceservice.kfserving-webhook-server.v1beta1.defaulter","objectSelector":{"matchExpressions":[{"key":"serving.kubeflow.org/inferenceservice", "operator": "Exists"}]}}]}'
 
kubectl patch ValidatingWebhookConfiguration inferenceservice.serving.kubeflow.org --patch '{"webhooks":[{"name": "inferenceservice.kfserving-webhook-server.v1beta1.validator","objectSelector":{"matchExpressions":[{"key":"serving.kubeflow.org/inferenceservice", "operator": "Exists"}]}}]}'

I think it is related to kserve/kserve#568 (comment). My testing environment is GKE v1.20.12. The error message is:

Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Audit-Id': '48360d9d-9621-43e8-a580-f40d74568b19', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '3e136267-4e52-4e29-9aa1-764e7dadc339', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b7e6649d-ea7f-442f-b7b5-7ea82514ebd3', 'Date': 'Wed, 23 Feb 2022 22:54:10 GMT', 'Content-Length': '717'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"inferenceservice.kfserving-webhook-server.v1beta1.validator\": Post \"
[https://kfserving-webhook-server-service.kubeflow.svc:443/validate-serving-kubeflow-org-v1beta1-inferenceservice?timeout=30s](https://kfserving-webhook-server-service.kubeflow.svc/validate-serving-kubeflow-org-v1beta1-inferenceservice?timeout=30s)
\": x509: certificate signed by unknown authority","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"inferenceservice.kfserving-webhook-server.v1beta1.validator\": Post \"[https://kfserving-webhook-server-service.kubeflow.svc:443/validate-serving-kubeflow-org-v1beta1-inferenceservice?timeout=30s](https://kfserving-webhook-server-service.kubeflow.svc/validate-serving-kubeflow-org-v1beta1-inferenceservice?timeout=30s)\": x509: certificate signed by unknown authority"}]},"code":500}
https://kfserving-webhook-server-service.kubeflow.svc:443/validate-serving-kubeflow-org-v1beta1-inferenceservice?timeout=30s
\": x509: certificate signed by unknown authority"}]},"code":500}

3. I encountered the following issue for saving data during the Kubeflow - Serve Model using KFServing step:
   https://github.com/kubeflow/pipelines/blob/master/samples/contrib/kubeflow-e2e-mnist/kubeflow-e2e-mnist.ipynb

                                                                                                                
Traceback (most recent call last):
  File "kfservingdeployer.py", line 437, in <module>
    main()
{'apiVersion': 'serving.kubeflow.org/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'sidecar.istio.io/inject': 'false'}, 'creationTimestamp': '2022-02-23T23:16:40Z', 'finalizers': ['inferenceservice.finalizers'], 'generation': 2, 'managedFields': [{'apiVersion': 'serving.kubeflow.org/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:sidecar.istio.io/inject': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:tensorflow': {'.': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2022-02-23T23:16:40Z'}, {'apiVersion': 'serving.kubeflow.org/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {}}, 'f:spec': {'f:predictor': {'f:tensorflow': {'f:name': {}, 'f:resources': {}}}}, 'f:status': {}}, 'manager': 'manager', 'operation': 'Update', 'time': '2022-02-23T23:16:40Z'}], 'name': 'mnist-e2e-v1beta1-validator', 'namespace': 'jamxl', 'resourceVersion': '110397', 'uid': '706c317d-dc10-4f6d-94c4-307e42a5d7be'}, 'spec': {'predictor': {'tensorflow': {'name': '', 'resources': {}, 'storageUri': 'pvc://end-to-end-pipeline-6wmv9-model-volume/'}}}}
  File "kfservingdeployer.py", line 394, in main
    for condition in model_status["status"]["conditions"]:
KeyError: 'status'
time="2022-02-23T23:21:43.263Z" level=error msg="cannot save artifact /tmp/outputs/InferenceService_Status/data" argo=true error="stat /tmp/outputs/InferenceService_Status/data: no such file or directory"
Error: exit status 1

Do you know how to resolve the last issue? @kimwnasptd @andreyvelich

[ryansteakley]

@kimwnasptd Checking in here from AWS, attempting to do a vanilla installation into a fresh EKS cluster on kubernetes version 1.19 installed the manifests using the single-line command. Cannot connect to port-fowarding and looks like the issue is down to the cache-deployer-deployment pod being stuck in an error state.

echo 'ERROR: After approving csr cache-server.kubeflow, the signed certificate did not appear on the resource. Giving up after 10 attempts.'
ERROR: After approving csr cache-server.kubeflow, the signed certificate did not appear on the resource. Giving up after 10 attempts.

When changing the image back to 1.5.0 from 1.8.0 in cache-deployer , it is working properly again. I saw you had run into the same issue do you know how to resolve it? kubeflow/pipelines#7093 (comment)

images:
name: gcr.io/ml-pipeline/cache-deployer
newTag: 1.8.0

[zijianjoy]

After discussion and debugging, we found that the issues 2 and 3 in #2146 (comment) are because I deploy KFServing and KServe together. My current suggestion is to deploy only one of them (which is kfserving), until we figure out how to migrate to kserve successfully and validate it using an updated mnist E2E script.

[kimwnasptd]

@zijianjoy @ryansteakley thank you very much for exposing your progress!

Hello @kimwnasptd , which model-web-app does central dashboard integrate with? There are KFserving and KServe. I am curious how to configure between these two web apps in the central dashboard.

I'll provide some instructions for this very soon, on how someone will be able to use the KServe app. I'll also make this the default app that will be used by the dashboard, but there are some rough edges right now. I'll create the issues accordingly and give a heads up here again.

1. Update KFP to v1.8.1-rc.0: https://github.com/kubeflow/pipelines/releases/tag/1.8.1-rc.0. This includes only fixes and no feature.

I'll also make a PR to update our manifests with this latest RC.

2. I encountered issues when deploying kfserving endpoint using mnist sample. The way I resolved this issue is by running the following command:

I haven't bumped into this while testing the manifests. It's also not clear to me yet why this error happened now, since we had the same KFServing 0.6.1 manifests from the KF 1.4 release. In any case, thank you James for providing instructions for handling it. I'll look more into it.

3. I encountered the following issue for saving data during the Kubeflow - Serve Model using KFServing step:
   https://github.com/kubeflow/pipelines/blob/master/samples/contrib/kubeflow-e2e-mnist/kubeflow-e2e-mnist.ipynb

I hadn't bumped into this as well. It looks like the InferenceService fails to get a status field. I'll open a new issue for this to track it independently and ping you there as well to further debug

[kimwnasptd]

@ryansteakley regarding #2146 (comment) can you double check you are using the v1.5.0-rc.1 of the manifests?

That RC includes KFP 1.8.0, which in turn includes the fix for the cache-deployer AFAIK kubeflow/pipelines#7273

[ryansteakley]

@kimwnasptd Yes, I'm checking out the v1.5.0-rc.1 tag of the manifests to test the vanilla kubeflow on EKS 1.19 using 1.20 kubectl locally.

[zijianjoy]

@kimwnasptd Google Cloud distribution is ready. 🚀

[yhwang]

@kimwnasptd IBM IKS is ready for k8s 1.21.
However, I am waiting for the knative 0.22.3 and going to try it out on k8s 1.22

[pwzhong]

@kimwnasptd Kubeflow Azure distribution has remained in v1.2 for two years, while v1.5 is coming up soon. Do you know if there is any plan to release Azure distribution with a more recent version? Who is the point of contact/representative?

I believe this is blocking Azure users from using kubeflow. v1.2 is on old k8s and istio versions, and v1.4 has no clear documents for Azure, it is quite hard to make it run on k8s 1.20+, which AKS only supports.