Support Kubernetes extended resources for container components through attributes that override container and pod specification

[l0rd]

/area api

Which functionality do you think we should add?

Why is this needed? Is your feature request related to a problem?

Devfiles don't allow to specify request/limits for resources other than memory and CPU. But there are cases where developers workloads require extended resources (e.g. nvidia.com/gpu).

Detailed description:

Kubernetes nodes can be assigned with extended resources (besides memory and CPU) and workloads can request and limit the use of those resources.

For example NVIDIA GPU operator advertises nvidia.com/gpu resources on the cluster nodes so that a Pod that requires them can specify the request and limit quantities:

  apiVersion: v1
  kind: Pod
  metadata:
    name: python
  spec:
    containers:
      - name: python
        image: tensorflow/tensorflow:latest-gpu-jupyter
        resources:
          limits:
            memory: 4Gi
            nvidia.com/gpu: 1 # limiting to 1 GPU
          requests:
            nvidia.com/gpu: 1 # requesting 1 GPU

Describe the solution you'd like

Use devfile and components attributes to override pod and container spec:

JSON container overrides

schemaVersion: 2.1.0
metadata:
    name: python
components:
  - name: python
+   attributes:
+     container-overrides: {"resources": {"limits": {"nvidia.com/gpu": 1"}}}
    container:
      image: tensorflow/tensorflow:latest-gpu-jupyter
      (...)

YAML container overrides

schemaVersion: 2.1.0
metadata:
    name: python
components:
  - name: python
+   attributes:
+     container-overrides:
+       resources:
+         limits:
+           nvidia.com/gpu: 1
    container:
      image: tensorflow/tensorflow:latest-gpu-jupyter
      (...)

pod overrides at devfile attributes level (both YAML and JSON are supported)

schemaVersion: 2.1.0
metadata:
    name: python
+ attributes:
+   pod-overrides: {"spec": {"securityContext": {"allowPrivilegeEscalation": false}}}
components:
  - name: python
    container:
      image: tensorflow/tensorflow:latest-gpu-jupyter
      (...)

pod overrides at container attributes level (both YAML and JSON are supported)

schemaVersion: 2.1.0
metadata:
    name: python
components:
  - name: python
+   attributes:
+      pod-overrides: {"spec": {"schedulerName": "stork"}}
    container:
      image: tensorflow/tensorflow:latest-gpu-jupyter
      (...)

Additional context

The request comes from OpenShift Dev Spaces customers that were able to achieve that with devfile v1 kubernetes components. With devfile v2 kubernetes components run in their own pod and may not have access to the source code (that is mounted in the main pod).

[Javatar81]

A drawback of the approach to support only a specific subset of the resources field is that the Devfile spec needs to follow the evolution of the Kubernetes pod resources field specification. This wouldn't be a big issue if it was stable and clearly defined. However, the spec defines limits and requests as type Object (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#resourcerequirements-v1-core). Hence, the definition of limits and requests is open which is also demonstrated by the variety of fields interpreted by Kubernetes as documented here: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/. Examples are:

• spec.containers[].resources.limits.cpu
• spec.containers[].resources.limits.memory
• spec.containers[].resources.limits.hugepages-
• spec.containers[].resources.requests.cpu
• spec.containers[].resources.requests.memory
• spec.containers[].resources.requests.hugepages-

When it comes to gpus, the docs https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/ introduces .com/gpu (e.g. amd.com/gpu or nvidia.com/gpu) for device plugins.

I just wanted to point this out to discuss whether it might make sense to just pass-through all the fields under the resources field. The devfile spec could just follow the Kubernetes spec and define it as object as well to leave open what is inside.

[kadel]

This makes me again think about the purpose of the container component.
Do we want to cover everything possible configuration from Kubernetes container definition? It looks like we are slowly heading this way :-( And if this is not intentional, we should draw a line somewhere.

We already a several issues about adding more fields from k8s container definition to devfile container definition.

• Readiness probes for endpoints #168
• ServiceAccount for a container component #903
• imagePullPolicy of Always leads to unnecessary pulls against Docker Hub rate limit, spec should specify pull policy and possibly provide other options #89

I think that need to decide between the following options and stick to it:

• Define that Devfile component is a simplified version of the Kubernetes component and stop adding new fields to it, unless there is a really good reason and 80% of users will benefit from it. If someone wants to use more advanced features, they will have to use kubernetes component and provide a full definition. In this case will have to make sure that kubernetes components can be used as a replacement for containers in the rest fo the Devfile spec (for example in exec command)
• Define that Devfile component can be used as a replacement for Kubernetes component. In this case we should map all fields from Kubernetes component to Devfile component definition and do it in some systematic effort rather than doing it one by one.

[miboettc]

From a user perspective container component and kubernetes component are quite different, at least from an Eclipse Che perspective.

• container components run within the workspace pod. In the Devfile spec they have the mountSources flag which allows for easy access to the source code.
• kubernetes components do not have the mountSources flag (it was removed in the Devfile 2.x spec, but existed in Devfile 1.x). So there is no convenient way to access sources from a container. This means, the container could request GPUs but would not have access to any source code to execute on them.

Overall, I prefer the idea of @Javatar81 of making the definition of ressources in the container component more flexible. Being able to request GPUs would open Eclipse Che, odo and other tools to MLOps, ML Engineering use cases. So I assume there is a sufficiently large number of users who would benefit from such a feature.

[kadel]

Overall, I prefer the idea of @Javatar81 of making the definition of ressources in the container component more flexible. Being able to request GPUs would open Eclipse Che, odo and other tools to MLOps, ML Engineering use cases. So I assume there is a sufficiently large number of users who would benefit from such a feature.

I'm ok with that, that means that we will be taking the first approach that I mentioned in #920 (comment)

But that also means that we should add the other things like ServiceAccount, ImagePullPolicy, and another important one that just came up is SecurityContext (#923) to the container definition.

[l0rd]

After discussions during today's devfile cabal we agreed on:

• not including this into the devfile spec
• adopt a convention for tools like odo and che to use components attributes to override pods and containers specification.

Example n.1: add nvidia.com/gpu: 1 to a container resources requirements limits

components:
  - name: go
+   attributes:
+     container-overrides: {"resources": {"limits": "nvidia.com/gpu": 1"}}
    container:
      image: ...

Example n.2: specify the pod securityContext

components:
  - name: go
+   attributes:
+     pod-overrides: {"spec":{"securityContext":{"runAsUser":1000,"runAsGroup":3000,"fsGroup":2000}}}
    container:
      image: ...
   ...

[miboettc]

This sounds like a very flexible solution to address the problem. :-)

Would this be part of the Devfile 2.2 spec?

How does it relate to #860 and devfile/devworkspace-operator#852 ?

[amisevsk]

I've opened DevWorkspace Operator PR devfile/devworkspace-operator#944 that implements support for the pod-overrides attribute as described above. Any comments are welcome.

Some additional assumptions this implementation makes, at the moment:

1. Pod overrides apply to the workspace deployment, but in the sample are defined on a container component. To allow specifying the attribute in either global attributes or on a component, I've defined an order of precedence for processing multiple instances of the attribute, with later definitions overriding earlier definitions
2. The format for the pod-overrides attribute matches the definition in Allow DevWorkspace CR to specify pod spec overrides #860, i.e. pod-overrides are specified either as yaml:

   components:
     - name: go
       attributes:
         pod-overrides: 
           spec:
             securityContext: 
               runAsUser: 1000
               runAsGroup: 3000
               fsGroup: 2000
       container:
         image: ...
     ...

   or as json (but not a string):

   components:
     - name: go
       attributes:
         pod-overrides: {"spec":{"securityContext":{"runAsUser":1000,"runAsGroup":3000,"fsGroup":2000}}}
       container:
         image: ...
     ...

   This is a departure from the sample as it's not possible to concatenate fields (spec.securityContext:). The definition for how to support the concatenated form is unclear -- e.g. how would we set annotation

   io.kubernetes.cri-o.userns-mode: "auto:size=65536;map-to-root=true"
   

   in pod-overrides?