Skip to content

Commit a5f7b3b

Browse files
sarutakkayousterhout
authored andcommitted
[SPARK-7296] Add timeline visualization for stages in the UI.
This PR builds on #2342 by adding a timeline view for the Stage page, showing how tasks spend their time. With this timeline, we can understand following things of a Stage. * When/where each task ran * Total duration of each task * Proportion of the time each task spends Also, this timeline view can scrollable and zoomable. Author: Kousuke Saruta <[email protected]> Closes #5843 from sarutak/stage-page-timeline and squashes the following commits: 4ba9604 [Kousuke Saruta] Fixed the order of legends 16bb552 [Kousuke Saruta] Removed border of legend area 2e5d605 [Kousuke Saruta] Modified warning message 16cb2e6 [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into stage-page-timeline 7ae328f [Kousuke Saruta] Modified code style d5f794a [Kousuke Saruta] Fixed performance issues more 64e6642 [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into stage-page-timeline e4a3354 [Kousuke Saruta] minor code style change 878e3b8 [Kousuke Saruta] Fixed a bug that tooltip remains b9d8f1b [Kousuke Saruta] Fixed performance issue ac8842b [Kousuke Saruta] Fixed layout 2319739 [Kousuke Saruta] Modified appearances more 81903ab [Kousuke Saruta] Modified appearances a79dcc3 [Kousuke Saruta] Modified appearance 55a390c [Kousuke Saruta] Ignored scalastyle for a line-comment 29eae3e [Kousuke Saruta] limited to longest 1000 tasks 2a9e376 [Kousuke Saruta] Minor cleanup 385b6d2 [Kousuke Saruta] Added link feature ba1ac3e [Kousuke Saruta] Fixed style 2ae8520 [Kousuke Saruta] Updated bootstrap-tooltip.js from 2.2.2 to 2.3.2 af430f1 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into stage-page-timeline e694b8e [Kousuke Saruta] Added timeline view to StagePage 8f6610c [Kousuke Saruta] Fixed conflict b587cf2 [Kousuke Saruta] initial commit 11fe67d [Kousuke Saruta] Fixed conflict 79ac03d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature a91abd3 [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature ef34a5b [Kousuke Saruta] Implement tooltip using bootstrap b09d0c5 [Kousuke Saruta] Move `stroke` and `fill` attribute of rect elements to css d3c63c8 [Kousuke Saruta] Fixed a little bit bugs a36291b [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature 28714b6 [Kousuke Saruta] Fixed highlight issue 0dc4278 [Kousuke Saruta] Addressed most of Patrics's feedbacks 8110acf [Kousuke Saruta] Added scroll limit to Job timeline 974a64a [Kousuke Saruta] Removed unused function ee7a7f0 [Kousuke Saruta] Refactored 6a91872 [Kousuke Saruta] Temporary commit 6693f34 [Kousuke Saruta] Added link to job/stage box in the timeline in order to move to corresponding row when we click 8f88222 [Kousuke Saruta] Added job/stage description aeed4b1 [Kousuke Saruta] Removed stage timeline fc1696c [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature 999ccd4 [Kousuke Saruta] Improved scalability 0fc6a31 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature 19815ae [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature 68b7540 [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature 52b5f0b [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature dec85db [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature fcdab7d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature dab7cc1 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature 09cce97 [Kousuke Saruta] Cleanuped 16f82cf [Kousuke Saruta] Cleanuped 9fb522e [Kousuke Saruta] Cleanuped d05f2c2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature e85e9aa [Kousuke Saruta] Cleanup: Added TimelineViewUtils.scala a76e569 [Kousuke Saruta] Removed unused setting in timeline-view.css 5ce1b21 [Kousuke Saruta] Added vis.min.js, vis.min.css and vis.map to .rat-exclude 082f709 [Kousuke Saruta] Added Timeline-View feature for Applications, Jobs and Stages (cherry picked from commit 9b6cf28) Signed-off-by: Kay Ousterhout <[email protected]>
1 parent 7dc0ff3 commit a5f7b3b

File tree

4 files changed

+348
-10
lines changed

4 files changed

+348
-10
lines changed

core/src/main/resources/org/apache/spark/ui/static/timeline-view.css

Lines changed: 65 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,65 @@ div#application-timeline, div#job-timeline {
2424
margin-top: 5px;
2525
}
2626

27+
#task-assignment-timeline div.legend-area {
28+
width: 574px;
29+
}
30+
31+
#task-assignment-timeline .legend-area > svg {
32+
width: 100%;
33+
height: 55px;
34+
}
35+
36+
#task-assignment-timeline div.item.range {
37+
padding: 0px;
38+
height: 26px;
39+
border-width: 0;
40+
}
41+
42+
.task-assignment-timeline-content {
43+
width: 100%;
44+
}
45+
46+
.task-assignment-timeline-duration-bar {
47+
width: 100%;
48+
height: 26px;
49+
}
50+
51+
rect.scheduler-delay-proportion {
52+
fill: #80B1D3;
53+
stroke: #6B94B0;
54+
}
55+
56+
rect.deserialization-time-proportion {
57+
fill: #FB8072;
58+
stroke: #D26B5F;
59+
}
60+
61+
rect.shuffle-read-time-proportion {
62+
fill: #FDB462;
63+
stroke: #D39651;
64+
}
65+
66+
rect.executor-runtime-proportion {
67+
fill: #B3DE69;
68+
stroke: #95B957;
69+
}
70+
71+
rect.shuffle-write-time-proportion {
72+
fill: #FFED6F;
73+
stroke: #D5C65C;
74+
}
75+
76+
rect.serialization-time-proportion {
77+
fill: #BC80BD;
78+
stroke: #9D6B9E;
79+
}
80+
81+
rect.getting-result-time-proportion {
82+
fill: #8DD3C7;
83+
stroke: #75B0A6;
84+
}
85+
2786
.vis.timeline {
2887
line-height: 14px;
2988
}
@@ -178,6 +237,10 @@ tr.corresponding-item-hover > td, tr.corresponding-item-hover > th {
178237
display: none;
179238
}
180239

240+
#task-assignment-timeline.collapsed {
241+
display: none;
242+
}
243+
181244
.control-panel {
182245
margin-bottom: 5px;
183246
}
@@ -186,7 +249,8 @@ tr.corresponding-item-hover > td, tr.corresponding-item-hover > th {
186249
margin: 0;
187250
}
188251

189-
span.expand-application-timeline, span.expand-job-timeline {
252+
span.expand-application-timeline, span.expand-job-timeline,
253+
span.expand-task-assignment-timeline {
190254
cursor: pointer;
191255
}
192256

core/src/main/resources/org/apache/spark/ui/static/timeline-view.js

Lines changed: 69 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,73 @@ function drawJobTimeline(groupArray, eventObjArray, startTime) {
133133
});
134134
}
135135

136+
function drawTaskAssignmentTimeline(groupArray, eventObjArray, minLaunchTime, zoomMax) {
137+
var groups = new vis.DataSet(groupArray);
138+
var items = new vis.DataSet(eventObjArray);
139+
var container = $("#task-assignment-timeline")[0]
140+
var options = {
141+
groupOrder: function(a, b) {
142+
return a.value - b.value
143+
},
144+
editable: false,
145+
align: 'left',
146+
selectable: false,
147+
showCurrentTime: false,
148+
min: minLaunchTime,
149+
zoomable: false,
150+
zoomMax: zoomMax
151+
};
152+
153+
var taskTimeline = new vis.Timeline(container)
154+
taskTimeline.setOptions(options);
155+
taskTimeline.setGroups(groups);
156+
taskTimeline.setItems(items);
157+
158+
taskTimeline.on("rangechange", function(prop) {
159+
if (currentDisplayedTooltip !== null) {
160+
$(currentDisplayedTooltip).tooltip("hide");
161+
}
162+
});
163+
164+
function getTaskIdxAndAttempt(selector) {
165+
var taskIdxText = $(selector).attr("data-title");
166+
var taskIdxAndAttempt = taskIdxText.match("Task (\\d+) \\(attempt (\\d+)");
167+
var taskIdx = taskIdxAndAttempt[1];
168+
var taskAttempt = taskIdxAndAttempt[2];
169+
return taskIdx + "-" + taskAttempt;
170+
}
171+
172+
// If we zoom up and a box moves away when the corresponding tooltip is shown,
173+
// the tooltip can be remain.
174+
// So, we need to hide tooltips using another mechanism.
175+
var currentDisplayedTooltip = null;
176+
177+
$("#task-assignment-timeline").on({
178+
"mouseenter": function() {
179+
var taskIdxAndAttempt = getTaskIdxAndAttempt(this);
180+
$("#task-" + taskIdxAndAttempt).addClass("corresponding-item-hover");
181+
$(this).tooltip("show");
182+
currentDisplayedTooltip = this;
183+
},
184+
"mouseleave" : function() {
185+
var taskIdxAndAttempt = getTaskIdxAndAttempt(this);
186+
$("#task-" + taskIdxAndAttempt).removeClass("corresponding-item-hover");
187+
$(this).tooltip("hide");
188+
currentDisplayedTooltip = null;
189+
}
190+
}, ".task-assignment-timeline-content");
191+
192+
setupZoomable('#task-assignment-timeline-zoom-lock', taskTimeline);
193+
194+
$("span.expand-task-assignment-timeline").click(function() {
195+
$("#task-assignment-timeline").toggleClass('collapsed');
196+
197+
// Switch the class of the arrow from open to closed.
198+
$(this).find('.expand-task-assignment-timeline-arrow').toggleClass('arrow-open');
199+
$(this).find('.expand-task-assignment-timeline-arrow').toggleClass('arrow-closed');
200+
});
201+
}
202+
136203
function setupExecutorEventAction() {
137204
$(".item.box.executor").each(function () {
138205
$(this).hover(
@@ -147,15 +214,15 @@ function setupExecutorEventAction() {
147214
}
148215

149216
function setupZoomable(id, timeline) {
150-
$(id + '>input[type="checkbox"]').click(function() {
217+
$(id + ' > input[type="checkbox"]').click(function() {
151218
if (this.checked) {
152219
timeline.setOptions({zoomable: true});
153220
} else {
154221
timeline.setOptions({zoomable: false});
155222
}
156223
});
157224

158-
$(id + ">span").click(function() {
225+
$(id + " > span").click(function() {
159226
$(this).parent().find('input:checkbox').trigger('click');
160227
});
161228
}

0 commit comments

Comments
 (0)