Failed Validation Resource Tests When using Kustomize patches and patchesJson6902

[nakfour]

As part of the Openshift stack in #1567 we are using patches and patchesJson6902. Running the tests from the manifest test folder fails with error

=== RUN   TestCheckWebhookSelector
--- FAIL: TestCheckWebhookSelector (0.74s)
    validate_resources_test.go:314: Error unmarshaling ../stacks/openshift/application/katib/katib-controller-deployment.yaml; error: wrong Node Kind for  expected: MappingNode was SequenceNode: value: {- op: add
          path: /spec/template/spec/containers/0/args/-
          value: '--webhook-inject-securitycontext=true'}

Our patches in the kustomize file looks like this

patchesJson6902:
 - target:
   group: apps
   version: v1
   kind: Deployment
   name: katib-controller
 path: katib-controller-deployment.yaml

katib-controller-deployment.yaml

- op: add
  path: /spec/template/spec/containers/0/args/-
  value: '--webhook-inject-securitycontext=true'

This could be related to kpt issue that was never resolved here: kptdev/kpt#506
What is the workaround for this? The inline suggested workaround in the kpt issue did not work

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/bug	0.90
area/katib	0.81
area/kfctl	0.56

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[berndverst]

I'm seeing something potentially related (also getting an error with MappingNode vs SequenceNode) even just generating the tests as I am trying to add the Azure stack to the master branch in #1633

INFO|2020-11-11T18:06:59|../hack/generate_tests.py|142| Regenerating test /home/bverst/src/work/kubeflow/manifests/tests/stacks/azure/kustomize_test.go for /home/bverst/src/work/kubeflow/manifests/stacks/azure
INFO|2020-11-11T18:06:59|../hack/generate_tests.py|52| Removing directory /home/bverst/src/work/kubeflow/manifests/tests/stacks/azure/test_data/expected
INFO|2020-11-11T18:06:59|../hack/generate_tests.py|55| Creating directory /home/bverst/src/work/kubeflow/manifests/tests/stacks/azure/test_data/expected
2020/11/11 18:07:02 well-defined vars that were never replaced: artifactRepositoryAccessKeySecretKey,metadata-db-service,podDefaultsCertName,artifactRepositoryEndpoint,artifactRepositorySecretKeySecretKey,artifactRepositorySecretKeySecretName,executorImage,artifactRepositoryAccessKeySecretName,kfp-app-version,profile-name,metadata-service,ui-namespace,lock-namespace,policy,prefix,user,artifactRepositoryKeyPrefix,containerRuntimeExecutor,kfp-ui-namespace,podDefaultsDeploymentName,kubectl-delivery-image,ui-clusterDomain,artifactRepositoryBucket,artifactRepositoryInsecure,katib-ui-namespace,metadata-envoy-service
Error: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "admission-webhook",
      "app.kubernetes.io/component": "poddefaults", "app.kubernetes.io/name": "poddefaults",
      "kustomize.component": "admission-webhook"}, "name": "admission-webhook-deployment",
    "namespace": "kubeflow"}, "spec": {"selector": {"matchLabels": {"app": "admission-webhook",
        "app.kubernetes.io/component": "poddefaults", "app.kubernetes.io/name": "poddefaults",
        "kustomize.component": "admission-webhook"}}, "template": {"metadata": {"annotations": {
          "sidecar.istio.io/inject": "false"}, "labels": {"app": "admission-webhook",
          "app.kubernetes.io/component": "poddefaults", "app.kubernetes.io/name": "poddefaults",
          "kustomize.component": "admission-webhook"}}, "spec": {"containers": [{
            "args": ["--tlsCertFile=/etc/webhook/certs/tls.crt", "--tlsKeyFile=/etc/webhook/certs/tls.key"],
            "image": "gcr.io/kubeflow-images-public/admission-webhook:vmaster-gaf96e4e3",
            "name": "admission-webhook", "volumeMounts": [{"mountPath": "/etc/webhook/certs",
                "name": "webhook-cert", "readOnly": true}]}], "serviceAccountName": "admission-webhook-service-account",
        "volumes": [{"name": "webhook-cert", "secret": {"secretName": "webhook-certs"}}]}}}}
' at path 'spec/template/spec/containers/0/args/1': visit traversal on path: [0 args 1]: fieldName: 0: wrong Node Kind for  expected: SequenceNode was MappingNode: value: {{"args": ["--tlsCertFile=/etc/webhook/certs/tls.crt", "--tlsKeyFile=/etc/webhook/certs/tls.key"],
  "image": "gcr.io/kubeflow-images-public/admission-webhook:vmaster-gaf96e4e3", "name": "admission-webhook",
  "volumeMounts": [{"mountPath": "/etc/webhook/certs", "name": "webhook-cert", "readOnly": true}]}}
Traceback (most recent call last):
  File "../hack/generate_tests.py", line 145, in <module>
    run_kustomize_build(repo_root, full_dir)
  File "../hack/generate_tests.py", line 58, in run_kustomize_build
    subprocess.check_call([os.environ.get("KUSTOMIZE_BIN", "kustomize"), "build", "--load_restrictor", "none",
  File "/home/bverst/.pyenv/versions/3.8.5/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['kustomize', 'build', '--load_restrictor', 'none', '-o', '/home/bverst/src/work/kubeflow/manifests/tests/stacks/azure/test_data/expected']' returned non-zero exit status 1.
make: *** [Makefile:53: generate-changed-only] Error 1

[berndverst]

@pvaneck @Jeffwan FYI - perhaps you know what to do here.

[Jeffwan]

What's the kustomize version you are using?

[pvaneck]

@berndverst Yea, double check the kustomize version, as you might be using a later version. If so, just download this one for your system. Test generation is a bit finicky if not on the v3.2.1 version right now.

[berndverst]

I'm definitely on a newer version. Thanks for pointing that out!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to inactivity.

[edi-bice-by]

I ran into this and tried all sorts of kustomize versions including the one suggested above

`+ mkdir -p apps/jupyter/notebook-controller

• kpt pkg get https://github.com/kubeflow/manifests.git/apps/jupyter/notebook-controller/upstream@v1.3.1 apps/jupyter/notebook-controller
  Package "upstream":
  Fetching https://github.com/kubeflow/manifests@v1.3.1
  From https://github.com/kubeflow/manifests

• tag v1.3.1 -> FETCH_HEAD
  Adding package "apps/jupyter/notebook-controller/upstream".

Fetched 1 package(s).
Error: /Users/edibice/gcp-blueprints/kubeflow/apps/jupyter/notebook-controller/upstream/crd/patches/old_crd.yaml: wrong Node Kind for expected: MappingNode was SequenceNode: value: {# Use the old CRD because of the quantity validation issue:

https://github.com/kubeflow/kubeflow/issues/5722`