Add support for Spark Connect

[ChenYi015]

Purpose of this PR

Close #1801.

Proposed changes:

• Add a new CRD named SparkConnect
• Implements controller for SparkConnect
• Add example SparkConnect manifest
• Update Helm chart

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[ChenYi015]

/hold for review

[nabuskey]

Do we want to ensure this is applied every reconciliation loop? Not just the first time it's created.

[ChenYi015]

We should set immutable fields when creating the server pod. For mutable fields, we can try to update it in every reconciliation.

[hiboyang]

This is great! Thanks @ChenYi015 for the PR!

A quick question, Spark Connect will need a GRPC ingress to expose the driver side Spark Connect server endpoint, similar like the HTTP ingress to expose Spark UI. Does this PR contain code to create such GRPC ingress?

[ChenYi015]

A quick question, Spark Connect will need a GRPC ingress to expose the driver side Spark Connect server endpoint, similar like the HTTP ingress to expose Spark UI. Does this PR contain code to create such GRPC ingress?

@hiboyang Have not included it yet. We will implement this feature in this or following PRs.

[nabuskey]

LGTM!!

[MonkeyCanCode]

A quick question, Spark Connect will need a GRPC ingress to expose the driver side Spark Connect server endpoint, similar like the HTTP ingress to expose Spark UI. Does this PR contain code to create such GRPC ingress?

I think it will just be HTTP/2. You can port forward the pod/svc on 15002 then connect from your local.

[ChenYi015]

Will merge this PR and improve it in the following PRs.
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

/unhold

[kydim]

I am very happy to see the adding spark-connect, thank you

When will be release spark-connect?

[wkd-woo]

+1

[airkhin]

+1

[rafagsiqueira]

@ChenYi015 Thank you for your work on this!
I created a SparkConnect object using the example you provided, but it doesn't seem to have any effect. I do not see any pods created and the status is of the spark-connect object never changes:

NAMESPACE   NAME            AGE   STATUS   PODNAME
spark       spark-connect   18m

Am I missing something?

[ChenYi015]

@rafagsiqueira Please check whether the namespace of SparkConnect is included in spark.jobNamespaces, otherwise it will not be processed by the operater.

[rafagsiqueira]

@ChenYi015 I suspected that, but since I used the same namespace as the spark operator, I thought it wouldn't be an issue. Let me try with a different namespace.

[rafagsiqueira]

@ChenYi015 that was indeed the problem. Thank you very much for clarifying! Looking forward to using my newly deployed spark connect.

[torsol]

This is great! The example yaml specifies spark 4.0.0, but I'm limited to using spark 3.5 to use sedona 1.7.2. What versions of spark is this CRD limited to?

edit:
Found the sparkConf-setting, where you can specify the missing spark-connect-jar. I had to resolve an error related to the ivy-cache directory not found aswell.

spec:
  sparkVersion: 3.5.4
  sparkConf: 
    spark.jars.packages: org.apache.spark:spark-connect_2.12:3.5.4
    spark.driver.extraJavaOptions: "-Divy.cache.dir=/tmp -Divy.home=/tmp"
    spark.jars.ivy: /tmp/.ivy2

[ChenYi015]

The example yaml specifies spark 4.0.0, but I'm limited to using spark 3.5 to use sedona 1.7.2. What versions of spark is this CRD limited to?

@torsol For Spark v4, the spark connect jar is included by default. But for Spark 3.5, you will need to build a image which contains the spark connect jar or use ivy to specify it as a dependency.