Skip to content

Commit 72107fd

Browse files
npgretzChenYi015
npgretz
and
ChenYi015
authored
Allow the Controller and Webhook Containers to run with the securityContext: readOnlyRootfilesystem: true (#2282)
* create a tmp dir for the controller to write Spark artifacts to and set the controller to readOnlyRootFilesystem Signed-off-by: Nick Gretzon <[email protected]> * mount a dir for the webhook container to generate its certificates in and set readOnlyRootFilesystem: true for the webhook pod Signed-off-by: Nick Gretzon <[email protected]> * update the securityContext in the controller deployment test Signed-off-by: Nick Gretzon <[email protected]> * update securityContext of the webhook container in the deployment_test Signed-off-by: Nick Gretzon <[email protected]> * update README Signed-off-by: Nick Gretzon <[email protected]> * remove -- so comments are not rendered in the README.md Signed-off-by: Nick Gretzon <[email protected]> * recreate README.md after removal of comments for volumes and volumeMounts Signed-off-by: Nick Gretzon <[email protected]> * make indentation for volumes and volumeMounts consistent with rest of values.yaml Signed-off-by: Nick Gretzon <[email protected]> * Revert "make indentation for volumes and volumeMounts consistent with rest of values.yaml" This reverts commit dba97fc. Signed-off-by: Nick Gretzon <[email protected]> * fix indentation in webhook and controller deployment templates for volumes and volumeMounts Signed-off-by: Nick Gretzon <[email protected]> * Update charts/spark-operator-chart/values.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/values.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/values.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/values.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/templates/controller/deployment.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/templates/controller/deployment.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/templates/webhook/deployment.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/templates/webhook/deployment.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * add additional securityContext to the controller deployment_test.yaml Signed-off-by: Nick Gretzon <[email protected]> --------- Signed-off-by: Nick Gretzon <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> Co-authored-by: Yi Chen <[email protected]>
1 parent 763682d commit 72107fd

File tree

5 files changed

+50
-12
lines changed

5 files changed

+50
-12
lines changed

‎charts/spark-operator-chart/README.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
102102
| controller.rbac.annotations | object | `{}` | Extra annotations for the controller RBAC resources. |
103103
| controller.labels | object | `{}` | Extra labels for controller pods. |
104104
| controller.annotations | object | `{}` | Extra annotations for controller pods. |
105-
| controller.volumes | list | `[]` | Volumes for controller pods. |
105+
| controller.volumes | list | `[{"emptyDir":{"sizeLimit":"1Gi"},"name":"tmp"}]` | Volumes for controller pods. |
106106
| controller.nodeSelector | object | `{}` | Node selector for controller pods. |
107107
| controller.affinity | object | `{}` | Affinity for controller pods. |
108108
| controller.tolerations | list | `[]` | List of node taints to tolerate for controller pods. |
@@ -111,9 +111,9 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
111111
| controller.topologySpreadConstraints | list | `[]` | Topology spread constraints rely on node labels to identify the topology domain(s) that each Node is in. Ref: [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/). The labelSelector field in topology spread constraint will be set to the selector labels for controller pods if not specified. |
112112
| controller.env | list | `[]` | Environment variables for controller containers. |
113113
| controller.envFrom | list | `[]` | Environment variable sources for controller containers. |
114-
| controller.volumeMounts | list | `[]` | Volume mounts for controller containers. |
114+
| controller.volumeMounts | list | `[{"mountPath":"/tmp","name":"tmp","readOnly":false}]` | Volume mounts for controller containers. |
115115
| controller.resources | object | `{}` | Pod resource requests and limits for controller containers. Note, that each job submission will spawn a JVM within the controller pods using "/usr/local/openjdk-11/bin/java -Xmx128m". Kubernetes may kill these Java processes at will to enforce resource limits. When that happens, you will see the following error: 'failed to run spark-submit for SparkApplication [...]: signal: killed' - when this happens, you may want to increase memory limits. |
116-
| controller.securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true}` | Security context for controller containers. |
116+
| controller.securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"readOnlyRootFilesystem":true,"runAsNonRoot":true}` | Security context for controller containers. |
117117
| controller.sidecars | list | `[]` | Sidecar containers for controller pods. |
118118
| controller.podDisruptionBudget.enable | bool | `false` | Specifies whether to create pod disruption budget for controller. Ref: [Specifying a Disruption Budget for your Application](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) |
119119
| controller.podDisruptionBudget.minAvailable | int | `1` | The number of pods that must be available. Require `controller.replicas` to be greater than 1 |
@@ -141,7 +141,7 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
141141
| webhook.labels | object | `{}` | Extra labels for webhook pods. |
142142
| webhook.annotations | object | `{}` | Extra annotations for webhook pods. |
143143
| webhook.sidecars | list | `[]` | Sidecar containers for webhook pods. |
144-
| webhook.volumes | list | `[]` | Volumes for webhook pods. |
144+
| webhook.volumes | list | `[{"emptyDir":{"sizeLimit":"500Mi"},"name":"serving-certs"}]` | Volumes for webhook pods. |
145145
| webhook.nodeSelector | object | `{}` | Node selector for webhook pods. |
146146
| webhook.affinity | object | `{}` | Affinity for webhook pods. |
147147
| webhook.tolerations | list | `[]` | List of node taints to tolerate for webhook pods. |
@@ -150,9 +150,9 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
150150
| webhook.topologySpreadConstraints | list | `[]` | Topology spread constraints rely on node labels to identify the topology domain(s) that each Node is in. Ref: [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/). The labelSelector field in topology spread constraint will be set to the selector labels for webhook pods if not specified. |
151151
| webhook.env | list | `[]` | Environment variables for webhook containers. |
152152
| webhook.envFrom | list | `[]` | Environment variable sources for webhook containers. |
153-
| webhook.volumeMounts | list | `[]` | Volume mounts for webhook containers. |
153+
| webhook.volumeMounts | list | `[{"mountPath":"/etc/k8s-webhook-server/serving-certs","name":"serving-certs","readOnly":false,"subPath":"serving-certs"}]` | Volume mounts for webhook containers. |
154154
| webhook.resources | object | `{}` | Pod resource requests and limits for webhook pods. |
155-
| webhook.securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true}` | Security context for webhook containers. |
155+
| webhook.securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"readOnlyRootFilesystem":true,"runAsNonRoot":true}` | Security context for webhook containers. |
156156
| webhook.podDisruptionBudget.enable | bool | `false` | Specifies whether to create pod disruption budget for webhook. Ref: [Specifying a Disruption Budget for your Application](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) |
157157
| webhook.podDisruptionBudget.minAvailable | int | `1` | The number of pods that must be available. Require `webhook.replicas` to be greater than 1 |
158158
| spark.jobNamespaces | list | `["default"]` | List of namespaces where to run spark jobs. If empty string is included, all namespaces will be allowed. Make sure the namespaces have already existed. |

‎charts/spark-operator-chart/templates/webhook/deployment.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ spec:
9494
{{- end }}
9595
{{- with .Values.webhook.volumeMounts }}
9696
volumeMounts:
97-
{{- toYaml . | nindent 10 }}
97+
{{- toYaml . | nindent 8 }}
9898
{{- end }}
9999
{{- with .Values.webhook.resources }}
100100
resources:
@@ -123,7 +123,7 @@ spec:
123123
{{- end }}
124124
{{- with .Values.webhook.volumes }}
125125
volumes:
126-
{{- toYaml . | nindent 8 }}
126+
{{- toYaml . | nindent 6 }}
127127
{{- end }}
128128
{{- with .Values.webhook.nodeSelector }}
129129
nodeSelector:

‎charts/spark-operator-chart/tests/controller/deployment_test.yaml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -355,16 +355,30 @@ tests:
355355
set:
356356
controller:
357357
securityContext:
358+
readOnlyRootFilesystem: true
358359
runAsUser: 1000
359360
runAsGroup: 2000
360361
fsGroup: 3000
362+
allowPrivilegeEscalation: false
363+
capabilities:
364+
drop:
365+
- ALL
366+
runAsNonRoot: true
367+
privileged: false
361368
asserts:
362369
- equal:
363370
path: spec.template.spec.containers[0].securityContext
364371
value:
372+
readOnlyRootFilesystem: true
365373
runAsUser: 1000
366374
runAsGroup: 2000
367375
fsGroup: 3000
376+
allowPrivilegeEscalation: false
377+
capabilities:
378+
drop:
379+
- ALL
380+
runAsNonRoot: true
381+
privileged: false
368382

369383
- it: Should add sidecars if `controller.sidecars` is set
370384
set:

‎charts/spark-operator-chart/tests/webhook/deployment_test.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,10 +299,14 @@ tests:
299299
set:
300300
webhook:
301301
securityContext:
302+
readOnlyRootFilesystem: true
302303
runAsUser: 1000
303304
runAsGroup: 2000
304305
fsGroup: 3000
305306
asserts:
307+
- equal:
308+
path: spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem
309+
value: true
306310
- equal:
307311
path: spec.template.spec.containers[0].securityContext.runAsUser
308312
value: 1000

‎charts/spark-operator-chart/values.yaml

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,11 @@ controller:
107107
# key2: value2
108108

109109
# -- Volumes for controller pods.
110-
volumes: []
110+
volumes:
111+
# Create a tmp directory to write Spark artifacts to for deployed Spark apps.
112+
- name: tmp
113+
emptyDir:
114+
sizeLimit: 1Gi
111115

112116
# -- Node selector for controller pods.
113117
nodeSelector: {}
@@ -143,7 +147,11 @@ controller:
143147
envFrom: []
144148

145149
# -- Volume mounts for controller containers.
146-
volumeMounts: []
150+
volumeMounts:
151+
# Mount a tmp directory to write Spark artifacts to for deployed Spark apps.
152+
- name: tmp
153+
mountPath: "/tmp"
154+
readOnly: false
147155

148156
# -- Pod resource requests and limits for controller containers.
149157
# Note, that each job submission will spawn a JVM within the controller pods using "/usr/local/openjdk-11/bin/java -Xmx128m".
@@ -159,6 +167,7 @@ controller:
159167

160168
# -- Security context for controller containers.
161169
securityContext:
170+
readOnlyRootFilesystem: true
162171
privileged: false
163172
allowPrivilegeEscalation: false
164173
runAsNonRoot: true
@@ -256,7 +265,11 @@ webhook:
256265
sidecars: []
257266

258267
# -- Volumes for webhook pods.
259-
volumes: []
268+
volumes:
269+
# Create a dir for the webhook to generate its certificates in.
270+
- name: serving-certs
271+
emptyDir:
272+
sizeLimit: 500Mi
260273

261274
# -- Node selector for webhook pods.
262275
nodeSelector: {}
@@ -292,7 +305,13 @@ webhook:
292305
envFrom: []
293306

294307
# -- Volume mounts for webhook containers.
295-
volumeMounts: []
308+
volumeMounts:
309+
# Mount a dir for the webhook to generate its certificates in.
310+
- name: serving-certs
311+
mountPath: /etc/k8s-webhook-server/serving-certs
312+
subPath: serving-certs
313+
readOnly: false
314+
296315

297316
# -- Pod resource requests and limits for webhook pods.
298317
resources: {}
@@ -305,6 +324,7 @@ webhook:
305324

306325
# -- Security context for webhook containers.
307326
securityContext:
327+
readOnlyRootFilesystem: true
308328
privileged: false
309329
allowPrivilegeEscalation: false
310330
runAsNonRoot: true

0 commit comments

Comments
 (0)