Seldon: Creating seldondeployment from KF 1.2 gives error: x509: certificate signed by unknown authority

[nakfour]

Hi
I am following the example for seldon listed here: https://www.kubeflow.org/docs/components/serving/seldon/ after installing seldon on Kubernetes 1.17 using this kfdef: https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_k8s_istio.v1.2.0.yaml

When creating the SeldonDeployment in the seldon namespace, I get this error

error when creating "../test-seldon.yaml": Internal error occurred: failed calling webhook "v1.vseldondeployment.kb.io": Post https://seldon-webhook-service.kubeflow.svc:443/validate-machinelearning-seldon-io-v1-seldondeployment?timeout=30s: x509: certificate signed by unknown authority ```

[Jeffwan]

/cc @kubeflow/wg-serving-leads

[ukclivecox]

We have seen this error when a reinstall happens and the mutatingwebhookconfiguration and validatingwebhookconfiguration are not updated so they don't get the new certificate.

[ukclivecox]

I just tried installing kubeflow with https://github.com/kubeflow/manifests/blob/v1.2-rc.1/kfdef/kfctl_k8s_istio.yaml and changed the uri to https://github.com/kubeflow/manifests/archive/v1.2-rc.1.tar.gz

This worked.

However, https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_k8s_istio.v1.2.0.yaml doesn't see to contain Seldon @Jeffwan ?

[Jeffwan]

@cliveseldon I don't think there're maintainers for this manifest. we probably like to deprecate in the future. platform owners will own what packages they like to integrate into the manifests.

/cc @kubeflow/wg-manifests-leads

[ukclivecox]

@Jeffwan But I'm just wondering why it can't stay like the generic one with all components until that time.
@nakfour I'm not sure given above how Seldon was installed if its not in this manifest

[PatrickXYS]

But I'm just wondering why it can't stay like the generic one with all components until that time.

It can, but there's no one to help maintain/test the manifest, then we're not sure of its availability.

[nakfour]

@cliveseldon its in the stack

- kustomizeConfig:
      repoRef:
        name: manifests
        path: stacks/kubernetes
    name: kubeflow-apps

https://github.com/kubeflow/manifests/blob/master/stacks/kubernetes/kustomization.yaml#L26

[ukclivecox]

I've replicated the issue with https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_k8s_istio.v1.2.0.yaml while https://github.com/kubeflow/manifests/blob/v1.2-rc.1/kfdef/kfctl_k8s_istio.yaml worked.

[ukclivecox]

The problem is the kustomize variables are not being updated when called from the stacks/kubernetes folder.
In particualar in the webhook there is

cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)

These variables are not updated.

They are defined in
https://github.com/kubeflow/manifests/blob/master/seldon/seldon-core-operator/base/kustomizeconfig.yaml

The file https://github.com/kubeflow/manifests/blob/master/stacks/kubernetes/kustomization.yaml would need to be fixed

[PatrickXYS]

Let's try to avoid using master branch's manifest, since they are not stable and vulnerable.

Stick with Release branch 1.0/1.1/1.2 is a good solution.

[ukclivecox]

https://github.com/kubeflow/manifests/blob/v1.2-rc.2/kfdef/kfctl_k8s_istio.v1.2.0.yaml will have the same issue

[Jeffwan]

@cliveseldon https://github.com/kubeflow/manifests/blob/338b1241a2214e898a0f9cd5a70ee11e9556c272/stacks/kubernetes/kustomization.yaml#L26

it uses ../../seldon/seldon-core-operator/overlays/application which defines https://github.com/kubeflow/manifests/blob/338b1241a2214e898a0f9cd5a70ee11e9556c272/seldon/seldon-core-operator/base/kustomization.yaml#L6-L7

[Jeffwan]

@cliveseldon https://github.com/kubeflow/manifests/blob/v1.2-rc.1/kfdef/kfctl_k8s_istio.yaml this is not well maintained.

It's not kustomize v3 compatible any more. Does https://github.com/kubeflow/manifests/blob/v1.2-rc.2/kfdef/kfctl_k8s_istio.v1.2.0.yaml work?

[ukclivecox]

It seems the ValidatingWebhookConfiguration does not have its variables converted while the MutatingWebhookConfiguration does. I can replicate this with kfctl build -d locally which produces

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: kubeflow/seldon-serving-cert

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)

Using kustomize directly seems to work but I was using kustomize v3. If I can't work out why its happening I will hardwire these values as they can be set without variables as kubeflow is always installed into a set namespace with a set certificate name as shown in the first of above.