KFServing fail with "Bad response from application: 503" when sending a prediction request through ingress and IAP

[jal06]

When trying KFServing GCP/IAP example , I receive an error Bad response from application: 503

After some investigations (more details in issue 1199 ), we discovered that the prediction request cannot reach the KFServing endpoint because the Istio local gateway is missing

[Bobgy]

Quoted from #177

Hi @jal06, I think the PR is not the idiomatic way to install cluster-local-gateway with istio/asm (anthos service mesh).

Because looking at https://github.com/knative/docs/blob/master/docs/install/installing-istio.md#installing-istio-without-sidecar-injection (strange that it's different from https://knative.dev/docs/install/installing-istio/#installing-istio-without-sidecar-injection),
there's a declarative configuration in IstioOperator that allows installing cluster-local-gateway:

  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          service:
            type: ClusterIP
            ports:
            - port: 15020
              name: status-port
            - port: 80
              targetPort: 8080
              name: http2
            - port: 443
              targetPort: 8443
              name: https

This seems to be the correct approach when installing istio using operator. Similarly, we are installing asm using the operator per definition in https://github.com/kubeflow/manifests/blob/master/gcp/v2/asm/istio-operator.yaml.
Therefore, we should include a similar ingressGateway definition in that operator. This will ensure the added cluster-local-gateway work with the entire service mesh.

and I suspect that installing cluster-local-gateway using kubeflow/manifests installed a gateway with old istio versions not designed for ASM, that's why it fails IAP authentication in your later experimentation.

[Bobgy]

I think next steps would be

1. upgrade ASM to 1.6+ as per Upgrade ASM from 1.4 to 1.6+ #144
2. add the cluster-local-gateway definition during installation
3. verify whether kfserving works e2e