Istio Authorization for Knative Service to Service Communication

[evankanderson]

@muthurajr

knative/serving#10940

I am having trouble in enabling Istio Authorization for Knative service to service communication.

Steps taken:

• Installed Istio and enabled mTLS with STRICT mode
• Installed Knative and enabled mTLS PERMISSIVE [https://knative.dev/docs/serving/istio-authorization/] as requests might be forwarded by activator based on TargetBurstCapacity.
• Created two namespaces serving-test1 & serving-test2
• Enabled istio sidecar injection and deployed Hello service in both serving-test1 & serving-test2

** Ignore the IP used in the code as I have tried to use Internal Load Balancer IP.

Expectation:

• Services in serving-test1 should get RBAC access denied when accessing services in serving-test2
• vice versa of previous scenario as well

Actual Result:

Services in serving-test1 is able to communicate with services in serving-test2 & vice versa.
I have tried on adding destination rules as well with host="*.local" and tls mode=ISTIO_MUTUAL, but no luck. Something I'm missing here but unable to identify. Any help is appreciated.

Complete installation and test scripts below.

export ISTIO_VERSION="1.8.2"
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} TARGET_ARCH=x86_64 sh -
export PATH="$PATH:istio-${ISTIO_VERSION}/bin"
export ISTIO_HOME=istio-${ISTIO_VERSION}/
cat <<EOF > istio-minimal-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        autoInject: enabled
      useMCP: false
      jwtPolicy: first-party-jwt
    meshConfig:
      enableAutoMtls: true
  addonComponents:
    pilot:
      enabled: true
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          service:
            type: LoadBalancer
            loadBalancerIP: "10.173.128.70"
EOF

istioctl install -y -f istio-minimal-operator.yaml
kubectl -n istio-system annotate service istio-ingressgateway cloud.google.com/load-balancer-type=Internal --overwrite
kubectl ${OPERATION:-apply} -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
EOF


kubectl create namespace serving-test1
kubectl create namespace serving-test2
kubectl create namespace knative-serving
kubectl label namespace serving-test1 istio-injection=enabled
kubectl label namespace serving-test2 istio-injection=enabled
kubectl label namespace knative-serving istio-injection=enabled


export KNATIVE_VERSION="v0.20.0"
kubectl apply -f https://github.com/knative/serving/releases/download/${KNATIVE_VERSION}/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/${KNATIVE_VERSION}/serving-core.yaml
kubectl apply -f https://github.com/knative/net-istio/releases/download/${KNATIVE_VERSION}/release.yaml

kubectl ${OPERATION:-apply} -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "knative-serving"
spec:
  mtls:
    mode: PERMISSIVE
EOF

cat <<EOF | kubectl ${OPERATION:-apply} -f -
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  labels:
    app: hello
  name: hello
  namespace: serving-test1
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
      - env:
        - name: TARGET
          value: Go Sample v1
        image: gcr.io/knative-samples/helloworld-go
        name: user-container
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  labels:
    app: hello
  name: hello
  namespace: serving-test2
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
      - env:
        - name: TARGET
          value: Go Sample v2
        image: gcr.io/knative-samples/helloworld-go
        name: user-container
EOF

kubectl -n serving-test1 exec -it $(kubectl -n serving-test1 get pod -o jsonpath='{.items[0].metadata.name}') -- bash
curl -H "Host: hello.serving-test2.example.com" http://10.173.128.70
curl -H "Host: hello.serving-test2.example.com" http://istio-ingressgateway.istio-system.svc.cluster.local
curl -H "Host: hello.serving-test2.example.com" http://knative-local-gateway.istio-system.svc.cluster.local
curl http://hello.serving-test2.svc.cluster.local

kubectl -n serving-test2 exec -it $(kubectl -n serving-test2 get pod -o jsonpath='{.items[0].metadata.name}') -- bash
curl -H "Host: hello.serving-test1.example.com" http://10.173.128.70
curl -H "Host: hello.serving-test1.example.com" http://istio-ingressgateway.istio-system.svc.cluster.local
curl -H "Host: hello.serving-test1.example.com" http://knative-local-gateway.istio-system.svc.cluster.local
curl http://hello.serving-test1.svc.cluster.local


kubectl ${OPERATION:-apply} -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: serving-test1
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: serving-test1
spec:
  {}
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: allow-serving-tests
 namespace: serving-test1
spec:
 action: ALLOW
 rules:
 - from:
   - source:
      namespaces: ["serving-test1", "knative-serving"]
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allowlist-by-paths
  namespace: serving-test1
spec:
  action: ALLOW
  rules:
  - to:
    - operation:
        paths:
        - /metrics
        - /healthz
---
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "serving-test2"
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: serving-test2
spec:
  {}
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: allow-serving-tests
 namespace: serving-test2
spec:
 action: ALLOW
 rules:
 - from:
   - source:
      namespaces: ["serving-test2", "knative-serving"]
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allowlist-by-paths
  namespace: serving-test2
spec:
  action: ALLOW
  rules:
  - to:
    - operation:
        paths:
        - /metrics
        - /healthz
EOF

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[davidspek]

I believe I am also running into this while investigation problems with KFServing for Kubeflow.

[julz]

I assume the problem is "Installed Knative and enabled mTLS PERMISSIVE [https://knative.dev/docs/serving/istio-authorization/] as requests might be forwarded by activator based on TargetBurstCapacity." -- the services in different namespace will be able to communicate with each other whenever activator is in path, because activator is allowed to access any service.

Unfortunately I think this is currently "working as designed", because requests can go via activator and not directly between namespaces. In order to prevent access via AuthorizationPolicy you would have to also add AuthorizationPolicy to block activator requests for other namespaces based on incoming namespace, e.g. you could enforce that requests from namespaceA may only access activator when providing the K_SERVING_NAMESPACE=namespaceA header, which would prevent namespaceA apps using activator to access other namespaces.

[davidspek]

@julz I've noticed that the Activator seems to try to access the service pods directly, rather than their service. Is that correct? This is currently preventing me from implementing an authorization rule that allows access to /metrics, /ready and other paths KNative uses only from the knative-serving namespace or the service account KNative uses. In a situation like Kubeflow where users are isolated by namespace, if the selector for the knative-serving namespace or the service account is omitted from the authorization policy, any user can access those paths in another user's namespace which breaks the isolation.

I go into more details about what I've found here

[markusthoemmes]

The knative activator will indeed try to directly address the pods, if it can. It should not be able to do that with mTLS strict and thelike enabled. It'll fall back to service-communication if it finds it can't address the pods directly.

[davidspek]

@markusthoemmes Thanks for the response. I'll try enabling strict mode and see if that helps.

[yanniszark]

Thanks a lot @markusthoemmes! Could you expand a little on what you mean by:

try to directly address the pods, if it can. It should not be able to do that with mTLS strict and thelike enabled.

What does Knative check to decide that it can't address a Pod directly?
What is the minimum Knative version that this should be working in?

P.S. We have also found this issue from you (istio/istio#23494) which seems relevant to the discussion at hand.

[markusthoemmes]

It tries to send a request to the pod directly. If that doesn't work, it falls back to the Service. This should be working since like ever. Which version would you be bound to?

[davidspek]

@markusthoemmes We are using Istio 1.9.5 or higher at the moment. Personally, I am using the latest KNative. Kubeflow is using KNative 0.17.3.

[markusthoemmes]

Though I'd strongly suggest to think about updating (we've just released 0.24), 0.17 should have this fallback methodology in. I wonder why the activator can even reach the pods directly in your system. Are you injecting sidecars into both the activator and the service pods?

[davidspek]

I'll update to 0.24 now or very soon in any case. Both the activator and the service pods have sidecars injected yes.