Skip to content

Commit d6a972b

Browse files
Kostas Sakellismarmbrus
authored andcommitted
[SPARK-4774] [SQL] Makes HiveFromSpark more portable
HiveFromSpark read the kv1.txt file from SPARK_HOME/examples/src/main/resources/kv1.txt which assumed you had a source tree checked out. Now we copy the kv1.txt file to a temporary file and delete it when the jvm shuts down. This allows us to run this example outside of a spark source tree. Author: Kostas Sakellis <[email protected]> Closes #3628 from ksakellis/kostas-spark-4774 and squashes the following commits: 6770f83 [Kostas Sakellis] [SPARK-4774] [SQL] Makes HiveFromSpark more portable
1 parent ab2abcb commit d6a972b

File tree

1 file changed

+11
-2
lines changed

1 file changed

+11
-2
lines changed

examples/src/main/scala/org/apache/spark/examples/sql/hive/HiveFromSpark.scala

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,17 +17,26 @@
1717

1818
package org.apache.spark.examples.sql.hive
1919

20+
import com.google.common.io.{ByteStreams, Files}
21+
22+
import java.io.File
23+
2024
import org.apache.spark.{SparkConf, SparkContext}
2125
import org.apache.spark.sql._
2226
import org.apache.spark.sql.hive.HiveContext
2327

2428
object HiveFromSpark {
2529
case class Record(key: Int, value: String)
2630

31+
// Copy kv1.txt file from classpath to temporary directory
32+
val kv1Stream = HiveFromSpark.getClass.getResourceAsStream("/kv1.txt")
33+
val kv1File = File.createTempFile("kv1", "txt")
34+
kv1File.deleteOnExit()
35+
ByteStreams.copy(kv1Stream, Files.newOutputStreamSupplier(kv1File))
36+
2737
def main(args: Array[String]) {
2838
val sparkConf = new SparkConf().setAppName("HiveFromSpark")
2939
val sc = new SparkContext(sparkConf)
30-
val path = s"${System.getenv("SPARK_HOME")}/examples/src/main/resources/kv1.txt"
3140

3241
// A hive context adds support for finding tables in the MetaStore and writing queries
3342
// using HiveQL. Users who do not have an existing Hive deployment can still create a
@@ -37,7 +46,7 @@ object HiveFromSpark {
3746
import hiveContext._
3847

3948
sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
40-
sql(s"LOAD DATA LOCAL INPATH '$path' INTO TABLE src")
49+
sql(s"LOAD DATA LOCAL INPATH '${kv1File.getAbsolutePath}' INTO TABLE src")
4150

4251
// Queries are expressed in HiveQL
4352
println("Result of 'SELECT *': ")

0 commit comments

Comments
 (0)