Skip to content

Commit b1493d8

Browse files
LuciferYangmaropu
authored andcommitted
[SPARK-35398][SQL] Simplify the way to get classes from ClassBodyEvaluator in CodeGenerator.updateAndGetCompilationStats method
### What changes were proposed in this pull request? SPARK-35253 upgraded janino from 3.0.16 to 3.1.4, `ClassBodyEvaluator` provides the `getBytecodes` method to get the mapping from `ClassFile#getThisClassName` to `ClassFile#toByteArray` directly in this version and we don't need to get this variable by reflection api anymore. So the main purpose of this pr is simplify the way to get `bytecodes` from `ClassBodyEvaluator` in `CodeGenerator#updateAndGetCompilationStats` method. ### Why are the changes needed? Code simplification. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass the Jenkins or GitHub Action - Manual test: 1. Define a code fragment to be tested, for example: ``` val codeBody = s""" public java.lang.Object generate(Object[] references) { return new TestMetricCode(references); } class TestMetricCode { public TestMetricCode(Object[] references) { } public long sumOfSquares(long left, long right) { return left * left + right * right; } } """ ``` 2. Create a `ClassBodyEvaluator` and `cook` the `codeBody` as above, the process of creating `ClassBodyEvaluator` can extract from `CodeGenerator#doCompile` method. 3. Get `bytecodes` using `ClassBodyEvaluator#getBytecodes` api(after this pr) and reflection api(before this pr) respectively, then assert that they are the same. If the `bytecodes` not changed, we can be sure that metrics state will not change. The test code example as follows: ``` import scala.collection.JavaConverters._ val bytecodesFromApi = evaluator.getBytecodes.asScala val bytecodesFromReflectionApi = { val scField = classOf[ClassBodyEvaluator].getDeclaredField("sc") scField.setAccessible(true) val compiler = scField.get(evaluator).asInstanceOf[SimpleCompiler] val loader = compiler.getClassLoader.asInstanceOf[ByteArrayClassLoader] val classesField = loader.getClass.getDeclaredField("classes") classesField.setAccessible(true) classesField.get(loader).asInstanceOf[java.util.Map[String, Array[Byte]]].asScala } assert(bytecodesFromApi == bytecodesFromReflectionApi) ``` Closes #32536 from LuciferYang/SPARK-35253-FOLLOWUP. Authored-by: yangjie01 <[email protected]> Signed-off-by: Takeshi Yamamuro <[email protected]>
1 parent 186477c commit b1493d8

File tree

1 file changed

+2
-12
lines changed
  • sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen

1 file changed

+2
-12
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@
1818
package org.apache.spark.sql.catalyst.expressions.codegen
1919

2020
import java.io.ByteArrayInputStream
21-
import java.util.{Map => JavaMap}
2221

2322
import scala.collection.JavaConverters._
2423
import scala.collection.mutable
@@ -28,8 +27,7 @@ import scala.util.control.NonFatal
2827
import com.google.common.cache.{CacheBuilder, CacheLoader}
2928
import com.google.common.util.concurrent.{ExecutionError, UncheckedExecutionException}
3029
import org.codehaus.commons.compiler.{CompileException, InternalCompilerException}
31-
import org.codehaus.commons.compiler.util.reflect.ByteArrayClassLoader
32-
import org.codehaus.janino.{ClassBodyEvaluator, SimpleCompiler}
30+
import org.codehaus.janino.ClassBodyEvaluator
3331
import org.codehaus.janino.util.ClassFile
3432

3533
import org.apache.spark.{TaskContext, TaskKilledException}
@@ -1434,15 +1432,7 @@ object CodeGenerator extends Logging {
14341432
*/
14351433
private def updateAndGetCompilationStats(evaluator: ClassBodyEvaluator): ByteCodeStats = {
14361434
// First retrieve the generated classes.
1437-
val classes = {
1438-
val scField = classOf[ClassBodyEvaluator].getDeclaredField("sc")
1439-
scField.setAccessible(true)
1440-
val compiler = scField.get(evaluator).asInstanceOf[SimpleCompiler]
1441-
val loader = compiler.getClassLoader.asInstanceOf[ByteArrayClassLoader]
1442-
val classesField = loader.getClass.getDeclaredField("classes")
1443-
classesField.setAccessible(true)
1444-
classesField.get(loader).asInstanceOf[JavaMap[String, Array[Byte]]].asScala
1445-
}
1435+
val classes = evaluator.getBytecodes.asScala
14461436

14471437
// Then walk the classes to get at the method bytecode.
14481438
val codeAttr = Utils.classForName("org.codehaus.janino.util.ClassFile$CodeAttribute")

0 commit comments

Comments
 (0)