-
Notifications
You must be signed in to change notification settings - Fork 28.7k
[SPARK-5088] Use spark-class for running executors directly #3897
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Test build #25051 has started for PR 3897 at commit
|
Test build #25051 has finished for PR 3897 at commit
|
Test PASSed. |
What's the motivation for this change? |
@tnachen On the contrary, why mesos has two different launcher? I couldn't find any idea in a history of that codes, and spark-executor calls spark-class which has a case statement internally. I think that merging two similar codes is a good idea in maintaining codes. |
I'm not part of that history either, I just want to know the motivation since the PR doesn't say anything. |
@JoshRosen @tgravescs @andrewor14 Could anyone review this PR? That makes mesos codes clean |
/cc @mateiz How about this patch? I don't know who maintains mesos' codes mainly. Because you are one of committers of mesos, I ask you to review this simple PR, which is about removing redundant script file. I've already tested this PR in my cluster |
Please review this PR |
This looks okay, but is there any motivation for it other than getting rid of that source file? |
if (uri == null) { | ||
val executorPath = new File(executorSparkHome, "/sbin/spark-executor").getCanonicalPath | ||
val executorPath = new File(executorSparkHome, s"/bin/spark-class $executorBackendName") | ||
.getCanonicalPath |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is actually weird, why are we including $executorBackendName
in the filename? We should just look for spark-class. I'm not even sure getCanonicalPath will work on a file that doesn't exist
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's my mistake. I'll fix it right now. Because I always set spark.executor.uri
, That's not tested.
@mateiz That's all, but I think it is meaningful to remove some redundant codes and helps understand codes easy. And I wonder why that code exists. I think |
Coarse and fine-grained mode are different on purpose, I don't think it makes sense to merge those. BTW have you tested both of them on this? |
@mateiz I've test it both mode. |
Test build #25494 has started for PR 3897 at commit
|
Test build #25495 has started for PR 3897 at commit
|
@mateiz I've fixed weird codes by your comments - thanks - and added a test case whether |
@mateiz Could you please explain the purposes of coarse and fine-grained mode shortly? It looks similar. The only difference between two modes is the way to offer resources. I have no idea of history of existing two modes. |
Test build #25497 has started for PR 3897 at commit
|
Test build #25497 has finished for PR 3897 at commit
|
Test FAILed. |
Test build #25495 has finished for PR 3897 at commit
|
Test PASSed. |
retest this please.I don't know why that simple test fails. |
Test build #25502 has started for PR 3897 at commit
|
Test build #25494 has finished for PR 3897 at commit
|
Test PASSed. |
Test build #25502 has finished for PR 3897 at commit
|
Test PASSed. |
@mateiz Passed tests. Please review it again. |
Test build #25533 has started for PR 3897 at commit
|
Test build #25533 has finished for PR 3897 at commit
|
Test PASSed. |
@mateiz Check this PR finally. I think It would be good by your review. |
@jongyoul sorry for the delayed reply, but this looks good. Mind rebasing it on master though? It's now fallen behind. |
@mateiz I'll rebase this PR from master. |
Test build #25732 has started for PR 3897 at commit
|
Rebase is not finished. |
- Changed command for using spark-class directly - Delete sbin/spark-executor and moved some codes into spark-class' case statement
- Adjusted orders of import
- Fixed code if spark.executor.uri doesn't have any value - Added test cases
- Rebased from master Conflicts: core/src/test/scala/org/apache/spark/scheduler/mesos/MesosSchedulerBackendSuite.scala
Test build #25734 has started for PR 3897 at commit
|
Test build #25732 has finished for PR 3897 at commit
|
Test PASSed. |
Test build #25734 has finished for PR 3897 at commit
|
Test FAILed. |
- Added a listenerBus for fixing test cases
Test build #25738 has started for PR 3897 at commit
|
Test build #25738 has finished for PR 3897 at commit
|
Test PASSed. |
@mateiz I've rebased this PR and finished tests successfully. Merge this, please. |
LGTM I will pull it in. |
Thanks, looks good! |
No description provided.