[SPARK-4774] [SQL] Makes HiveFromSpark more portable #3628
Conversation
| // Copy kv1.txt file from classpath to temporary directory | ||
| val kv1Stream = HiveFromSpark.getClass.getResourceAsStream("/kv1.txt") | ||
| val tmpDir = Files.createTempDir() | ||
| tmpDir.deleteOnExit() |
There was a problem hiding this comment.
I don't think this works for non-empty directories. You'd need to recursively delete it. Instead of creating a temp directory, I'd just use File.createTempFile(), unless the name of the file reeeeeally needs to be kv1.txt.
Also, the paranoid in me would close all open streams in a finally block, but not super important in this code.
|
Looks ok aside from the cleanup issue. |
|
Test build #24201 has started for PR 3628 at commit
|
|
Thanks for doing this! Mind adding |
ksakellis
changed the title
HiveFromSpark read the kv1.txt file from SPARK_HOME/examples/src/main/resources/kv1.txt which assumed you had a source tree checked out. Now we copy the kv1.txt file to a temporary file and delete it when the jvm shutsdown. This allows us to run this example outside of a spark source tree.
ksakellis
force-pushed
the
kostas-spark-4774
branch
from
fca97ee to
6770f83
Compare
|
Test build #24202 has started for PR 3628 at commit
|
|
Test build #24201 has finished for PR 3628 at commit
|
|
Test PASSed. |
|
Test build #24202 has finished for PR 3628 at commit
|
|
Test PASSed. |
|
LGTM |
HiveFromSpark read the kv1.txt file from SPARK_HOME/examples/src/main/resources/kv1.txt which assumed you had a source tree checked out. Now we copy the kv1.txt file to a temporary file and delete it when the jvm shuts down. This allows us to run this example outside of a spark source tree. Author: Kostas Sakellis <[email protected]> Closes #3628 from ksakellis/kostas-spark-4774 and squashes the following commits: 6770f83 [Kostas Sakellis] [SPARK-4774] [SQL] Makes HiveFromSpark more portable (cherry picked from commit d6a972b) Signed-off-by: Michael Armbrust <[email protected]>
HiveFromSpark read the kv1.txt file from SPARK_HOME/examples/src/main/resources/kv1.txt which assumed
you had a source tree checked out. Now we copy the kv1.txt file to a temporary file and delete it when
the jvm shuts down. This allows us to run this example outside of a spark source tree.