[SPARK-46912] Use worker JAVA_HOME and SPARK_HOME instead of from submitter

[thanhdanh1803]

What changes were proposed in this pull request?

Replace JAVA_HOME and SPARK_HOME from submitter by value of worker when building localCommand.

Why are the changes needed?

There is a problem when submit a job in cluster mode to a standalone cluster. The worker start a java job using value from submitter JAVA_HOME instead of itself.

Does this PR introduce any user-facing change?

No

Was this patch authored or co-authored using generative AI tooling?

No

[srowen]

Hm, how does JAVA_HOME get from the 'submitter' - what do you mean, the application submitter? but the worker is already running by that point

[thanhdanh1803]

Hm, how does JAVA_HOME get from the 'submitter' - what do you mean, the application submitter? but the worker is already running by that point

• The submitter is the client machine which run then command spark-submit (with deploy-mode = cluster).
• The worker is already running at this point but the driver does not. When master received a submit request, it starts creating a driver on a worker, at that point, the driver copy the command and environment variable from submit command and use in its session. It sounds weird but that what I am facing in my case.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[jywjyw]

It's a bug. Try this command on machine A:

bin/spark-submit --master spark://{REMOTE}:7077 --deploy-mode cluster  --class org.apache.spark.examples.SparkPi  file:///opt/spark/examples/jars/spark-examples_2.12-3.5.3.jar 

it will submit application to standalone cluster(important: deploy-mode=cluster), then error occurs:

Exception from cluster was: java.io.IOException: Cannot run program "/usr/java/default//bin/java" (in directory "/opt/bitnami/spark/work/driver-20241206013526-0001"): error=2, No such file or directory

Because the worker node searched java from "/usr/java/default//bin/java", but "/usr/java/default//bin/java" is machine A's java path, not worker's java path

[MeltonSmith]

I've opened a new PR: #51314