v2.3.0

Highlights

• Support Spark v4.
• Add support for Spark Connect by adding a new CRD called SparkConnect. One example can be found here.
• Upgrade CRDs automatically when running helm upgrade by setting hook.upgradeCrd=true. This will create a Helm pre-install/pre-upgrade Job to run kubectl apply --server-side to update CRDs.
• Configure logging format by setting {controller,webhook}.logEncoder to json or console.

Features

• Add support for Spark Connect (#2569 by @ChenYi015)
• upgrade to Spark 4.0.0 (#2564 by @nabuskey)
• Make logging encoder configurable (#2580 by @ChenYi015)
• Include pod.Status.Message in recordExecutorEvent (#2589 by @matschaffer-roblox)
• Add print columns for Spark Connect (#2592 by @nabuskey
• chore: update prometheus pattern and labels for structured streaming driver (#2581 by @yahwang)
• Add Helm hook to upgrade CRDs (#2371 by @ChenYi015)

Bug Fixes

• Splat recordExecutorEvent args for cleaner event messages (#2582 by @matschaffer-roblox)
• fix: should add executor env when driver env is empty (#2586 by @ChenYi015)
• Add web UI configurations when enabling UI service and ingress (#2599 by @ChenYi015)
• Grant get/patch permissions for SparkConnect CRD to hook ClusterRole (#2605 by @ChenYi015)

Dependencies

• Bump aquasecurity/trivy-action from 0.31.0 to 0.32.0 (#2585 by @dependabot[bot])
• Bump github.com/go-viper/mapstructure/v2 from 2.2.1 to 2.3.0 (#2572 by @dependabot[bot])
• Bump sigs.k8s.io/yaml from 1.4.0 to 1.5.0 (#2577 by @dependabot[bot])
• Bump helm.sh/helm/v3 from 3.17.3 to 3.18.4 (#2587 by @dependabot[bot])
• Bump golang.org/x/mod from 0.25.0 to 0.26.0 (#2608 by @dependabot[bot])
• Bump github.com/onsi/gomega from 1.36.1 to 1.37.0 (#2607 by @dependabot[bot])

Misc

• Add SparkConnect e2e test (#2578 by @ChenYi015)
• feat(docs): Guide to report security vulnerabilities (#2593 by @andreyvelich)
• Read Helm version from go.mod file (#2598 by @ChenYi015)

Full Changelog

v2.3.0-rc.0

Official Release v2.3.0-rc.0

v2.2.1

Features

• Customize ingress URL with Spark application ID (#2554 by @ChenYi015)
• Make default ingress tls and annotations congurable in the helm config (#2513 by @Tom-Newton)
• Use code-generator for clientset, informers, listers (#2563 by @jbhalodia-slack)

Misc

• add driver ingress unit tests (#2552 by @nabuskey)
• Get logger from context (#2551 by @ChenYi015)
• Update golangci lint (#2560 by @joshuacuellar1)

Dependencies

• Bump aquasecurity/trivy-action from 0.30.0 to 0.31.0 (#2557 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.21.1 to 1.22.0 (#2548 by @dependabot[bot])
• Bump sigs.k8s.io/scheduler-plugins from 0.30.6 to 0.31.8 (#2549 by @dependabot[bot])
• Bump golang.org/x/mod from 0.24.0 to 0.25.0 (#2566 by @dependabot[bot])
• Bump github.com/go-logr/logr from 1.4.2 to 1.4.3 (#2567 by @dependabot[bot])

Full Changelog

v2.2.0

Features

• Upgrade to Spark 3.5.5 (#2490 by @jacobsalway)
• Add timeZone to ScheduledSparkApplication (#2471 by @jacobsalway)
• Enable the override of MemoryLimit through webhook (#2478 by @danielrsfreitas)
• Add ShuffleTrackingEnabled to DynamicAllocation struct to allow disabling shuffle tracking (#2511 by @jbhalodia-slack)
• Define SparkApplicationSubmitter interface to allow customizing submitting mechanism (#2500 by @ChenYi015)
• Add support for using cert manager to generate webhook certificates (#2373 by @ChenYi015)

Bug Fixes

• fix: add webhook cert validity checking (#2489 by @teejaded)
• fix and add back unit tests (#2532 by @nabuskey)
• fix volcano tests (#2533 by @nabuskey)
• Add v2 to module path (#2515 by @ChenYi015)
• #2525 spark metrics in depends on prometheus (#2529 by @blcksrx)

Misc

• Add APRA AMCOS to adopters (#2485 by @shuch3ng)
• Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 (#2488 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.20.5 to 1.21.1 (#2487 by @dependabot[bot])
• Bump sigs.k8s.io/controller-runtime from 0.20.1 to 0.20.4 (#2486 by @dependabot[bot])
• Deprecating sparkctl (#2484 by @vikas-saxena02)
• Changing image repo from docker.io to ghcr.io (#2483 by @vikas-saxena02)
• Upgrade Golang to 1.24.1 and golangci-lint to 1.64.8 (#2494 by @jacobsalway)
• Bump helm.sh/helm/v3 from 3.16.2 to 3.17.3 (#2503 by @dependabot[bot])
• Add changelog for v2.1.1 (#2504 by @ChenYi015)
• Remove sparkctl (#2466 by @ChenYi015)
• Bump github.com/spf13/viper from 1.19.0 to 1.20.1 (#2496 by @dependabot[bot])
• Bump golang.org/x/net from 0.37.0 to 0.38.0 (#2505 by @dependabot[bot])
• Remove clientset, informer and listers generated by code-generator (#2506 by @ChenYi015)
• Remove v1beta1 API (#2516 by @ChenYi015)
• add unit tests for driver and executor configs (#2521 by @nabuskey)
• Adding securityContext to spark examples (#2530 by @tarekabouzeid)
• Bump github.com/spf13/cobra from 1.8.1 to 1.9.1 (#2497 by @dependabot[bot])
• Bump golang.org/x/mod from 0.23.0 to 0.24.0 (#2495 by @dependabot[bot])
• Adding Manabu to the reviewers (#2522 by @vara-bonthu)
• Bump manusa/actions-setup-minikube from 2.13.1 to 2.14.0 (#2523 by @dependabot[bot])
• Bump k8s.io dependencies to v0.32.5 (#2540 by @ChenYi015)
• Pass the correct LDFLAGS when building the operator image (#2541 by @ChenYi015)

v2.2.0-rc.1

Spark Operator Official Release v2.2.0-rc.1

v2.2.0-rc.0

Spark Operator Official Release v2.2.0-rc.0

v2.1.1

Annocuments

• We plan to deprecate sparkctl in the future releases since it is no longer actively maintained, see #2465. Please consider using kubectl or other tools to manage Spark applications.

Features

• Adding seccompProfile RuntimeDefault (#2397 by @tarekabouzeid)
• Add option for disabling leader election (#2423 by @ChenYi015)
• Controller should only be granted event permissions in spark job namespaces (#2426 by @ChenYi015)
• Make image optional (#2439 by @jbhalodia-slack)
• Support non-standard Spark container names (#2441 by @jbhalodia-slack)
• add support for metrics-job-start-latency-buckets flag in helm (#2450 by @nabuskey)

Bug Fixes

• fix: webhook fail to add lifecycle to Spark3 executor pods (#2458 by @pvbouwel)
• change env in executorSecretOption (#2467 by @TQJADE)

Misc

• Move sparkctl to cmd directory (#2347 by @ChenYi015)
• Bump golang.org/x/net from 0.30.0 to 0.32.0 (#2350 by @dependabot[bot])
• Bump golang.org/x/crypto from 0.30.0 to 0.31.0 (#2365 by @dependabot[bot])
• add an example of using prometheus servlet (#2403 by @nabuskey)
• Remove dependency on k8s.io/kubernetes (#2398 by @jacobsalway)
• fix make deploy and install (#2412 by @nabuskey)
• Add helm unittest step to integration test workflow (#2424 by @ChenYi015)
• ensure passed context is used (#2432 by @nabuskey)
• Bump manusa/actions-setup-minikube from 2.13.0 to 2.13.1 (#2390 by @dependabot[bot])
• Bump helm/chart-testing-action from 2.6.1 to 2.7.0 (#2391 by @dependabot[bot])
• Bump golang.org/x/mod from 0.21.0 to 0.23.0 (#2427 by @dependabot[bot])
• Bump github.com/golang/glog from 1.2.2 to 1.2.4 (#2411 by @dependabot[bot])
• Bump golang.org/x/net from 0.32.0 to 0.35.0 (#2428 by @dependabot[bot])
• Support Kubernetes 1.32 (#2416 by @jacobsalway)
• use cmd context in sparkctl (#2447 by @nabuskey)
• Bump golang.org/x/net from 0.35.0 to 0.36.0 (#2470 by @dependabot[bot])
• Bump aquasecurity/trivy-action from 0.29.0 to 0.30.0 (#2475 by @dependabot[bot])
• Bump golang.org/x/net from 0.35.0 to 0.37.0 (#2472 by @dependabot[bot])
• Bump github.com/containerd/containerd from 1.7.19 to 1.7.27 (#2476 by @dependabot[bot])
• Bump k8s.io/apimachinery from 0.32.0 to 0.32.3 (#2474 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.66.0 to 1.78.2 (#2473 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/config from 1.28.0 to 1.29.9 (#2463 by @dependabot[bot])
• Bump sigs.k8s.io/scheduler-plugins from 0.29.8 to 0.30.6 (#2444 by @dependabot[bot])

v2.1.0

New Features

• Upgrade to Spark 3.5.3 (#2202 by @jacobsalway)
• feat: support archives param for spark-submit (#2256 by @kaka-zb)
• Allow --ingress-class-name to be specified in chart (#2278 by @jacobsalway)
• Update default container security context (#2265 by @ChenYi015)
• Support pod template for Spark 3.x applications (#2141 by @ChenYi015)
• Allow setting automountServiceAccountToken (#2298 by @Aranch)
• Allow the Controller and Webhook Containers to run with the securityContext: readOnlyRootfilesystem: true (#2282 by @npgretz)
• Use NSS_WRAPPER_PASSWD instead of /etc/passwd as in spark-operator image entrypoint.sh (#2312 by @Aakcht)

Bug Fixes

• Minor fixes to e2e test make targets (#2242 by @Tom-Newton)
• Added off heap memory to calculation for YuniKorn gang scheduling (#2209 by @guangyu-yang-rokt)
• Add permissions to controller serviceaccount to list and watch ingresses (#2246 by @tcassaert)
• Make sure enable-ui-service flag is set to false when controller.uiService.enable is set to false (#2261 by @Roberdvs)
• omitempty corrections (#2255 by @Tom-Newton)
• Fix retries (#2241 by @Tom-Newton)
• Fix: executor container security context does not work (#2306 by @ChenYi015)
• Fix: should not add emptyDir sizeLimit conf if it is nil (#2305 by @ChenYi015)
• Fix: should not add emptyDir sizeLimit conf on executor pods if it is nil (#2316 by @Cian911)
• Truncate UI service name if over 63 characters (#2311 by @jacobsalway)
• The webhook-key-name command-line param isn't taking effect (#2344 by @c-h-afzal)
• Robustness to driver pod taking time to create (#2315 by @Tom-Newton)

Misc

• remove redundant test.sh file (#2243 by @ChenYi015)
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.42 to 1.27.43 (#2252 by @dependabot[bot])
• Bump manusa/actions-setup-minikube from 2.12.0 to 2.13.0 (#2247 by @dependabot[bot])
• Bump golang.org/x/net from 0.29.0 to 0.30.0 (#2251 by @dependabot[bot])
• Bump aquasecurity/trivy-action from 0.24.0 to 0.27.0 (#2248 by @dependabot[bot])
• Bump gocloud.dev from 0.39.0 to 0.40.0 (#2250 by @dependabot[bot])
• Add Quick Start guide to README (#2259 by @jacobsalway)
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.63.3 to 1.65.3 (#2249 by @dependabot[bot])
• Add release badge to README (#2263 by @jacobsalway)
• Bump helm.sh/helm/v3 from 3.16.1 to 3.16.2 (#2275 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.20.4 to 1.20.5 (#2274 by @dependabot[bot])
• Bump cloud.google.com/go/storage from 1.44.0 to 1.45.0 (#2273 by @dependabot[bot])
• Run e2e tests with Kubernetes version matrix (#2266 by @jacobsalway)
• Bump aquasecurity/trivy-action from 0.27.0 to 0.28.0 (#2270 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.65.3 to 1.66.0 (#2271 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.43 to 1.28.0 (#2272 by @dependabot[bot])
• Add workflow for releasing sparkctl binary (#2264 by @ChenYi015)
• Bump volcano.sh/apis to 1.10.0 (#2320 by @jacobsalway)
• Bump aquasecurity/trivy-action from 0.28.0 to 0.29.0 (#2332 by @dependabot[bot])
• Bump github.com/onsi/ginkgo/v2 from 2.20.2 to 2.22.0 (#2335 by @dependabot[bot])
• Move sparkctl to cmd directory (#2347 by @ChenYi015)

Full Changelog

v2.1.0-rc.0

New Features

• Upgrade to Spark 3.5.3 (#2202 by @jacobsalway)
• feat: support archives param for spark-submit (#2256 by @kaka-zb)
• Allow --ingress-class-name to be specified in chart (#2278 by @jacobsalway)
• Update default container security context (#2265 by @ChenYi015)
• Support pod template for Spark 3.x applications (#2141 by @ChenYi015)

Bug Fixes

• Minor fixes to e2e test make targets (#2242 by @Tom-Newton)
• Added off heap memory to calculation for YuniKorn gang scheduling (#2209 by @guangyu-yang-rokt)
• Add permissions to controller serviceaccount to list and watch ingresses (#2246 by @tcassaert)
• Make sure enable-ui-service flag is set to false when controller.uiService.enable is set to false (#2261 by @Roberdvs)
• omitempty corrections (#2255 by @Tom-Newton)
• Fix retries (#2241 by @Tom-Newton)

Misc

• remove redundant test.sh file (#2243 by @ChenYi015)
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.42 to 1.27.43 (#2252 by @dependabot[bot])
• Bump manusa/actions-setup-minikube from 2.12.0 to 2.13.0 (#2247 by @dependabot[bot])
• Bump golang.org/x/net from 0.29.0 to 0.30.0 (#2251 by @dependabot[bot])
• Bump aquasecurity/trivy-action from 0.24.0 to 0.27.0 (#2248 by @dependabot[bot])
• Bump gocloud.dev from 0.39.0 to 0.40.0 (#2250 by @dependabot[bot])
• Add Quick Start guide to README (#2259 by @jacobsalway)
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.63.3 to 1.65.3 (#2249 by @dependabot[bot])
• Add release badge to README (#2263 by @jacobsalway)
• Bump helm.sh/helm/v3 from 3.16.1 to 3.16.2 (#2275 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.20.4 to 1.20.5 (#2274 by @dependabot[bot])
• Bump cloud.google.com/go/storage from 1.44.0 to 1.45.0 (#2273 by @dependabot[bot])
• Run e2e tests with Kubernetes version matrix (#2266 by @jacobsalway)
• Bump aquasecurity/trivy-action from 0.27.0 to 0.28.0 (#2270 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.65.3 to 1.66.0 (#2271 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.43 to 1.28.0 (#2272 by @dependabot[bot])
• Add workflow for releasing sparkctl binary (#2264 by @ChenYi015)

Full Changelog

v2.0.2

Bug Fixes

• Fix ingress capability discovery (#2201 by @jacobsalway)
• fix: imagePullPolicy was ignored (#2222 by @missedone)
• fix: spark-submission failed due to lack of permission by user spark (#2223 by @missedone)
• Remove cap_net_bind_service from image (#2216 by @jacobsalway)
• fix: webhook panics due to logging (#2232 by @ChenYi015)

Misc

• Bump github.com/aws/aws-sdk-go-v2 from 1.30.5 to 1.31.0 (#2207 by @dependabot[bot])
• Bump golang.org/x/net from 0.28.0 to 0.29.0 (#2205 by @dependabot[bot])
• Bump github.com/docker/docker from 27.0.3+incompatible to 27.1.1+incompatible (#2125 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.58.3 to 1.63.3 (#2206 by @dependabot[bot])
• Update integration test workflow and add golangci lint check (#2197 by @ChenYi015)
• Bump github.com/aws/aws-sdk-go-v2 from 1.31.0 to 1.32.0 (#2229 by @dependabot[bot])
• Bump cloud.google.com/go/storage from 1.43.0 to 1.44.0 (#2228 by @dependabot[bot])
• Bump manusa/actions-setup-minikube from 2.11.0 to 2.12.0 (#2226 by @dependabot[bot])
• Bump golang.org/x/time from 0.6.0 to 0.7.0 (#2227 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.33 to 1.27.42 (#2231 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.19.1 to 1.20.4 (#2204 by @dependabot[bot])
• Add check for generating manifests and code (#2234 by @ChenYi015)

What's Changed

• Release v2.0.2 by @ChenYi015 in #2233

Full Changelog

More Details

• By removing setcap 'cap_net_bind_service=+ep' from the docker build, the container can run with all capabilities dropped on a non-privileged port. If you want to listen on port <1024, you could either run as root or add back the NET_BIND_SERVICE capability or build your own image with this flag on the binary.