Skip to content

[SPARK-4646] Replace Scala.util.Sorting.quickSort with Sorter(TimSort) in Spark #3507

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-4646] Replace Scala.util.Sorting.quickSort with Sorter(TimSort) in Spark #3507

wants to merge 2 commits into from

Conversation

maropu
Copy link
Member

@maropu maropu commented Nov 28, 2014

This patch just replaces a native quick sorter with Sorter(TimSort) in Spark.
It could get performance gains by ~8% in my quick experiments.

@ankurdave
Copy link
Contributor

ok to test

@SparkQA
Copy link

SparkQA commented Nov 28, 2014

Test build #23937 has finished for PR 3507 at commit 3527e00.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@oza
Copy link
Contributor

oza commented Nov 29, 2014 

org.apache.spark.streaming.CheckpointSuite.recovery with file input stream

Confirmed that the test failure looks not related to the patch.

@maropu
Copy link
Member Author

maropu commented Dec 1, 2014

Ok, I fixed it. If no issue, please merge it. Thanks.

@SparkQA
Copy link

SparkQA commented Dec 1, 2014

Test build #23964 has finished for PR 3507 at commit 8d4e5d2.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Dec 8, 2014
@maropu @ankurdave
[SPARK-4646] Replace Scala.util.Sorting.quickSort with Sorter(TimSort…
a4ae7c8 
…) in Spark

This patch just replaces a native quick sorter with Sorter(TimSort) in Spark.
It could get performance gains by ~8% in my quick experiments.

Author: Takeshi Yamamuro <[email protected]>

Closes #3507 from maropu/TimSortInEdgePartitionBuilderSpike and squashes the following commits:

8d4e5d2 [Takeshi Yamamuro] Remove a wildcard import
3527e00 [Takeshi Yamamuro] Replace Scala.util.Sorting.quickSort with Sorter(TimSort) in Spark

(cherry picked from commit 2e6b736)
Signed-off-by: Ankur Dave <[email protected]>
@asfgit asfgit closed this in 2e6b736 Dec 8, 2014
@ankurdave
Copy link
Contributor

Thanks! Merged into master & branch-1.2.

@maropu maropu deleted the TimSortInEdgePartitionBuilderSpike branch July 5, 2017 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@maropu @ankurdave @SparkQA @oza