Skip to content

Commit a131031

Browse files
MaxGekkdongjoon-hyun
authored andcommitted
[SPARK-30599][CORE][TESTS] Increase the maximum number of log events in LogAppender
### What changes were proposed in this pull request? Increased the limit for log events that could be stored in `SparkFunSuite.LogAppender` from 100 to 1000. ### Why are the changes needed? Sometimes (see traces in SPARK-30599) additional info is logged via log4j, and appended to `LogAppender`. For example, unusual log entries are: ``` [36] Removed broadcast_214_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.8 MiB) [37] Removed broadcast_204_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.9 MiB) [38] Removed broadcast_200_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2003.9 MiB) [39] Removed broadcast_207_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2003.9 MiB) [40] Removed broadcast_208_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2003.9 MiB) ``` and a test which uses `LogAppender` can fail with the exception: ``` java.lang.IllegalStateException: Number of events reached the limit of 100 while logging CSV header matches to schema w/ enforceSchema. ``` ### Does this PR introduce any user-facing change? No ### How was this patch tested? By re-running `"SPARK-23786: warning should be printed if CSV header doesn't conform to schema"` in a loop. Closes #27312 from MaxGekk/log-appender-filter. Authored-by: Maxim Gekk <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent cfb1706 commit a131031

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

core/src/test/scala/org/apache/spark/SparkFunSuite.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,7 @@ abstract class SparkFunSuite
189189
}
190190
}
191191

192-
class LogAppender(msg: String = "", maxEvents: Int = 100) extends AppenderSkeleton {
192+
class LogAppender(msg: String = "", maxEvents: Int = 1000) extends AppenderSkeleton {
193193
val loggingEvents = new ArrayBuffer[LoggingEvent]()
194194

195195
override def append(loggingEvent: LoggingEvent): Unit = {

0 commit comments

Comments
 (0)