[SPARK-2960] Support executing Spark from symlinks

[roji]

The current scripts (e.g. pyspark) fail to run when they are executed via symlinks. A common Linux scenario would be to have Spark installed somewhere (e.g. /opt) and have a symlink to it in /usr/bin.

Fixed the scripts to traverse symlinks until reaching the actual binary.

[AmplabJenkins]

Can one of the admins verify this patch?

[mateiz]

Jenkins, this is ok to test

[mateiz]

@roji mind opening a JIRA issue for this on https://issues.apache.org/jira/browse/SPARK and adding it in the pull request's title?

[SparkQA]

QA tests have started for PR 1875. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18290/consoleFull

[SparkQA]

QA results for PR 1875:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18290/consoleFull

[roji]

@mateiz, thanks for the attention. Have opened an issue for this.

May I suggest you guys update the contribution guide to specify that an issue be opened alongside pull requests...

[roji]

Have just noticed the additional scripts under sbin, which also require treatment. Just before I go ahead and work on that, can you confirm this is a desirable PR?

[mateiz]

Yup, please fix those too. I think this is useful to have.

[roji]

@mateiz, added a commit for the sbin scripts.

In general the scripts could use a bit of cleanup - the bin and sbin scripts work a bit differently, the sbin-specific spark-config.sh doesn't do much anymore, SPARK_HOME and SPARK_PREFIX are used interchangeably... But it isn't very important.

[SparkQA]

QA tests have started for PR 1875. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18953/consoleFull

[SparkQA]

QA results for PR 1875:
- This patch FAILED unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18953/consoleFull

[roji]

Um, am not sure, did something actually go wrong here?

[mateiz]

Yes, your patch no longer merges on master. Please rebase it onto the master branch.

[roji]

Rebased on master and squashed to a single commit, hope all is well now.

[mateiz]

Jenkins, test this please

[SparkQA]

QA tests have started for PR 1875 at commit ccbc6e5.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1875 at commit ccbc6e5.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mateiz]

Hey @roji, actually I looked at this and I notice that -h on the [ command is deprecated and replaced by -L. Mind changing that in the code? Otherwise it looks good.

[mateiz]

Actually I spoke too soon, this fix doesn't seem to work with symlinks to relative paths. You can do

cd spark
ln -s bin/spark-shell spark-shell

Then it will fail to run, thinking that the path is just "bin" or "/". Maybe it's because you add a / in front.

[mateiz]

It would be nice if you found some best practice way of doing this.

[mateiz]

Actually I notice that there's code for this in our sbt/sbt script, though it may be specific for finding directories. Take a look at that.

[SparkQA]

Can one of the admins verify this patch?

[marmbrus]

Any update here? I think this would be a great feature to have, but perhaps we should close this issue until it is ready to review (to make the size of the PR queue a little more manageable).

[roji]

Sorry for dropping out, was involved in other things. @mateiz, I'll take a look at your suggestions in the coming week.

[mateiz]

Alright, let me know when you've had a chance.