Skip to content

[SPARK-6943][SPARK-6944] DAG visualization on SparkUI #5729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 47 commits into from
Closed

[SPARK-6943][SPARK-6944] DAG visualization on SparkUI #5729

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
6b3403b
Scope all RDD methods
Apr 17, 2015
a9ed4f9
Add a few missing scopes to certain RDD methods
Apr 17, 2015
5143523
Expose the necessary information in RDDInfo
Apr 17, 2015
2184348
Translate RDD information to dot file
Apr 17, 2015
f22f337
First working implementation of visualization with vis.js
Apr 17, 2015
9fac6f3
Re-implement scopes through annotations instead
Apr 22, 2015
494d5c2
Revert a few unintended style changes
Apr 22, 2015
6a7cdca
Move RDD scope util methods and logic to its own file
Apr 22, 2015
5e22946
Merge branch 'master' of github.com:apache/spark into viz
Apr 22, 2015
205f838
Reimplement rendering with dagre-d3 instead of viz.js
Apr 23, 2015
fe7816f
Merge branch 'master' of github.com:apache/spark into viz
Apr 27, 2015
8dd5af2
Fill in documentation + miscellaneous minor changes
Apr 27, 2015
71281fa
Embed the viz in the UI in a toggleable manner
Apr 27, 2015
09d361e
Add ID to node label (minor)
Apr 27, 2015
52187fc
Rat excludes
Apr 28, 2015
c3bfcae
Re-implement scopes using closures instead of annotations
Apr 27, 2015
aa868a9
Ensure that HadoopRDD is actually serializable
Apr 27, 2015
4310271
Merge branch 'master' of github.com:apache/spark into viz2
Apr 28, 2015
7ef957c
Fix scala style
Apr 28, 2015
d19c4da
Merge branch 'master' of github.com:apache/spark into viz2
Apr 28, 2015
6e2cfea
Remove all return statements in `withScope`
Apr 29, 2015
43de96e
Add parent IDs to StageInfo
Apr 29, 2015
5e388ea
Fix line too long
Apr 29, 2015
5f07e9c
Remove more return statements from scopes
Apr 29, 2015
ab91416
Introduce visualization to the Job Page
May 1, 2015
5c7ce16
Connect RDDs across stages + update style
May 1, 2015
deb48a0
Translate stage boxes taking into account the width
May 1, 2015
0706992
Add link from jobs to stages
May 1, 2015
b80cc52
Merge branch 'master' of github.com:apache/spark into viz2
May 3, 2015
f9830a2
Refactor + clean up + document JS visualization code
May 3, 2015
ee33d52
Separate HTML generating code from listener
May 3, 2015
5a7faf4
Rename references to viz scopes to viz clusters
May 4, 2015
83f9c58
Implement a programmatic representation of operator scopes
May 4, 2015
31aae06
Extract visualization logic from listener
May 4, 2015
b1f0fd1
Rename OperatorScope -> RDDOperationScope
May 4, 2015
429e9e1
Display cached RDDs on the viz
May 4, 2015
832443c
Merge branch 'master' of github.com:apache/spark into viz2
May 4, 2015
3459ab2
Fix tests
May 4, 2015
0d7aa32
Fix python tests
May 4, 2015
afb98e2
Merge branch 'master' of github.com:apache/spark into viz2
May 4, 2015
0d07a84
Fix python style
May 4, 2015
5fdd89d
children -> child (minor)
May 4, 2015
3ffe566
Restore "null" as default for RDD name
May 4, 2015
1c310e4
Wrap a few more RDD functions in an operation scope
May 4, 2015
6f9574a
Add tests for RDDOperationScope
May 4, 2015
01ba336
Change RDD cache color to red (minor)
May 4, 2015
666c03b
Round corners of RDD boxes on stage page (minor)
May 4, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .rat-excludes
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ spark-env.sh.template
log4j-defaults.properties
bootstrap-tooltip.js
jquery-1.11.1.min.js
d3.min.js
dagre-d3.min.js
graphlib-dot.min.js
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add dagre-d3 and graphlib-dot to LICENSE? I have already added d3 in #5533.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes good catch - we should do that.

On Tue, May 5, 2015 at 10:14 PM, Shixiong Zhu [email protected]
wrote:

In .rat-excludes
#5729 (comment):

@@ -30,6 +30,9 @@ spark-env.sh.template
log4j-defaults.properties
bootstrap-tooltip.js
jquery-1.11.1.min.js
+d3.min.js
+dagre-d3.min.js
+graphlib-dot.min.js

Do we need to add dagre-d3 and graphlib-dot to LICENSE? I have already
added d3 in #5533 #5533.

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/5729/files#r29713742.

sorttable.js
vis.min.js
vis.min.css
Expand Down
5 changes: 5 additions & 0 deletions core/src/main/resources/org/apache/spark/ui/static/d3.min.js
View file
Open in desktop

Large diffs are not rendered by default.

29 changes: 29 additions & 0 deletions core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js
View file
Open in desktop

Large diffs are not rendered by default.

Loading