[SPARK-9352][SPARK-9353] Add tests for standalone scheduling code #7668
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, if cores per executor is not set, because we assign 1 core at a time we end up requiring the worker to have enough memory for N executors, where N is the number of cores assigned. This is incorrect because in this mode we should have at most 1 executor per worker. This is fixed in this commit with a regression test.
|
FYI @nishkamravi2 there was another small issue with your patch. |
|
Test build #38453 has finished for PR 7668 at commit
|
|
retest this please |
|
Test build #38478 has finished for PR 7668 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Jul 26, 2015
This also fixes a small issue in the standalone Master that was uncovered by the new tests. For more detail, read the description of SPARK-9353. Author: Andrew Or <[email protected]> Closes #7668 from andrewor14/standalone-scheduling-tests and squashes the following commits: d852faf [Andrew Or] Add tests + fix scheduling with memory limits (cherry picked from commit 1cf1976) Signed-off-by: Andrew Or <[email protected]> Conflicts: core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
|
Master 1.4 |
asfgit
pushed a commit
that referenced
this pull request
Jul 27, 2015
`MasterSuite.makeMaster()` doesn't compile under 1.4. `Master` is still an actor in branch-1.4, and can only be created via `ActorSystem.actorOf`, or with `TestActorRef`. Author: Cheng Lian <[email protected]> Closes #7686 from liancheng/hotfix-revert-pr-7668 and squashes the following commits: 089a1a8 [Cheng Lian] Revert "[SPARK-9352] [SPARK-9353] Add tests for standalone scheduling code" 54e8ab0 [Cheng Lian] Revert "[HOTFIX] Fix compile in MasterSuite"
asfgit
pushed a commit
that referenced
this pull request
Jul 28, 2015
This also fixes a small issue in the standalone Master that was uncovered by the new tests. For more detail, read the description of SPARK-9353. Author: Andrew Or <[email protected]> Closes #7668 from andrewor14/standalone-scheduling-tests and squashes the following commits: d852faf [Andrew Or] Add tests + fix scheduling with memory limits Conflicts: core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This also fixes a small issue in the standalone Master that was uncovered by the new tests. For more detail, read the description of SPARK-9353.