Required arrays with null value

[mbohlool]

Looks like there is a problem with 1.8 spec going forward [TODO:ref issue] that an array that marked as required can be null. Assuming an empty array is an acceptable array in these situation (and it look like it is as kubernetes return empty array in some of those cases, TODO: add ref issues) we should consider fixing it in the main repo (TODO: create an issue) and also here as a quick fix until main repo issue has been fixed. This is my brain dump on the issue in case of anyone wants to work on this:

• we should first research the problem and see if that is the case for all required arrays or a subset of them
• as a quick fix, we can remove those arrays (or subset of them based on previous findings) from required list in a preprocessing step in this repo.
• A regeneration of python (and other languages) clients should fix the issues (TODO: ref those issues)
• Next, there should be a more permanent fix in the main repo.

[roycaihw]

Gathering known required API fields issues in python client:

• io.k8s.apimachinery.pkg.apis.meta.v1.APIVersions requires serverAddressByClientCIDRs (Error when trying to retrieve api versions python#418, get_api_group() raises ValueError: Invalid value for server_address_by_client_cid_rs, must not be None python#394, kubernetes.client.apis.apps_v1beta1_api.AppsV1beta1Api.get_api_resources() crashes on OpenShift 3.6 because of deserialization failure python#464)

  • serverAddressByClientCIDRs is required by server may return null as a valid response https://github.com/kubernetes/kubernetes/blob/9bd4f12c336af339c35e773e1d71a37cad342f57/staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go#L407-L414
  • Fixed in Make serverAddressByClientCIDRs in discovery API optional kubernetes/kubernetes#61963

• v1beta1.CustomResourceDefinitionStatus requires conditions (allows null conditions, fixes #415 python#447)

• DisruptedPods in policy.v1beta1.PodDisruptionBudgetStatus (V1beta1PodDisruptionBudget deserialization should not throw exception on V1beta1PodDisruptionBudgetStatus python#466)

• extensions_v1beta1_deployment_rollback. Unrelated. There is a k8s pr fixing the wrong openapi spec. (v1beta1/create_namespaced_deployment_rollback python#491, Req: Example for deployment rollback python#427)

• involved_object in v1.Event. Unrelated. The bug is in watch client not handling HTTP error before unmarshaling Watch stream should handle HTTP error before unmarshaling event python-base#57 (Compatibility issue between K8s v1.9 and python client v5.0 python#484)

The issue started in release-4.0. Although we had the required-field validation generated since release-1.0, the api_client would assign [] for the null fields before deserialize the model (before release-3.0). Since release-4.0 the generated api_client doesn't convert null to empty array anymore, and the ValueError gets raised.

Agree that the problem is in our API convention (can an required array be null?), and there should be a permanent fix in the main repo.

[ceridwen]

There are a number of other API fields with this problem. From watching the issues in the two repositories in question, I have the following list. (This is just straight links, I don't have time right now to make the formatting pretty. I tried to avoid duplicates.)

Kubernetes

kubernetes-client/python#491
kubernetes-client/python#484
kubernetes-client/python#466
kubernetes-client/python#427
kubernetes-client/python#424

OpenShift

kubernetes-client/python#464
openshift/openshift-restclient-python#138

There are probably more that I haven't run into and that no one else has bothered to post an issue for yet.

[smiller171]

Looking at related issues (specifically #394) This has been an issue since at least November.

Is there any word on a fix or a workaround? As someone new to Kubernetes and even newer to the Python client, I have absolutely no idea how to move past this and make my application work.

[StephanX]

@smiller171 For good or ill, I've managed to work around this in my kubernetes-python module by commenting out the subject enforcement check at lines 184 & 185 of my /usr/local/lib/python3.6/site-packages/kubernetes/client/models/v1beta1_cluster_role_binding.py file (the content being):

        if subjects is None:
            raise ValueError("Invalid value for `subjects`, must not be `None`")

[djmckinley]

I just came across this issue with the "limits" array in the LimitRangeSpec resource. If I write a LimitRange object with an empty array, when reading it back, I get the exception: "Invalid value for limits, must not be None". I worked around it by catching those exceptions and dealing with them - but I'm wondering if removing the test raising the exception like StephanX did would be better.

This seems to be a serious, pervasive issue with at least the 5.0.0 library. Is that accurate, or am I misunderstanding something?

Is there any fix in the works? Or suggested workaround other than dealing with each and every occurrence? So far, in my project, we've had to implement workarounds for this limits case, and the PodDisruptionBudget case (kubernetes-client/python#466). This issue (#52) makes it appear there could be many more problems lurking out there. Do we need to give up on 5.0.0, and go back to an earlier stable library release?

[roycaihw]

Upstream PR fixing serverAddressByClientCIDRs: kubernetes/kubernetes#61963

[roycaihw]

Fixes for Conditions in CustomResourceDefinitionStatus kubernetes/kubernetes#64996
DisruptedPods in policy.v1beta1.PodDisruptionBudgetStatus kubernetes/kubernetes#65041