Skip to content

Commit cf097a5

Browse files
Maxime XuMridul Muralidharan
authored andcommitted
[SPARK-52776][CORE] Do not split the comm field in ProcfsMetricsGetter
### What changes were proposed in this pull request? We are fixing an issue in `ProcfsMetricsGetter` when parsing the `/proc/<pid>/stat` file. The current implementation will split the comm field by spaces if it contains them, thereby causing subsequent numbers to be shifted. The comm field, and only the comm field, is in parentheses so we can resolve this issue by ignoring everything between the first open parenthesis and last closing parenthesis when splitting the stat file. ### Why are the changes needed? These changes are needed to prevent a comm field with spaces from causing incorrect calculations for vmem/rssmem metrics. Please see [JIRA](https://issues.apache.org/jira/projects/SPARK/issues/SPARK-52776) for details. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added a unit test to test for irregular characters in the comm field ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#51457 from max2718281/procfs. Authored-by: Maxime Xu <[email protected]> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
1 parent 64f1c60 commit cf097a5

File tree

3 files changed

+21
-1
lines changed

3 files changed

+21
-1
lines changed

core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,14 @@ private[spark] class ProcfsMetricsGetter(procfsDir: String = "/proc/") extends L
9696
}
9797
Utils.tryWithResource(openReader()) { in =>
9898
val procInfo = in.readLine
99-
val procInfoSplit = procInfo.split(" ")
99+
// The comm field, which is inside parentheses, could contain spaces. We should not split
100+
// by those spaces as doing so could cause the numbers after it to be shifted.
101+
val commStartIndex = procInfo.indexOf('(')
102+
val commEndIndex = procInfo.lastIndexOf(')') + 1
103+
val pidArray = Array(procInfo.substring(0, commStartIndex).trim)
104+
val commArray = Array(procInfo.substring(commStartIndex, commEndIndex))
105+
val splitAfterComm = procInfo.substring(commEndIndex).trim.split(" ")
106+
val procInfoSplit = pidArray ++ commArray ++ splitAfterComm
100107
val vmem = procInfoSplit(22).toLong
101108
val rssMem = procInfoSplit(23).toLong * pageSize
102109
if (procInfoSplit(1).toLowerCase(Locale.US).contains("java")) {
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
487713 ((Executor) task l)) D 474416 474398 474398 0 -1 4194368 5 0 0 0 0 0 0 0 25 5 1 0 1542745216 7469137920 120815 18446744073709551615 104424108929024 104424108932808 140734257079632 0 0 0 4 3 553671884 1 0 0 17 58 0 0 0 0 0 104424108940536 104424108941336 104424532111360 140734257083781 140734257085131 140734257085131 140734257102797 0

core/src/test/scala/org/apache/spark/executor/ProcfsMetricsGetterSuite.scala

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,18 @@ class ProcfsMetricsGetterSuite extends SparkFunSuite {
9393
SparkEnv.set(originalSparkEnv)
9494
}
9595
}
96+
97+
test("SPARK-52776: Whitespace and parentheses in the comm field") {
98+
val p = new ProcfsMetricsGetter(getTestResourcePath("ProcfsMetrics"))
99+
var r = ProcfsMetrics(0, 0, 0, 0, 0, 0)
100+
r = p.addProcfsMetricsFromOneProcess(r, 487713)
101+
assert(r.jvmVmemTotal == 0)
102+
assert(r.jvmRSSTotal == 0)
103+
assert(r.pythonVmemTotal == 0)
104+
assert(r.pythonRSSTotal == 0)
105+
assert(r.otherVmemTotal == 7469137920L)
106+
assert(r.otherRSSTotal == 494858240)
107+
}
96108
}
97109

98110
object ProcfsMetricsGetterSuite {

0 commit comments

Comments
 (0)