[SPARK-7738] [SQL] [PySpark] add reader and writer API in Python #6238

davies · 2015-05-18T19:39:45Z

cc @rxin, please take a quick look, I'm working on tests.

AmplabJenkins · 2015-05-18T19:42:14Z

Merged build triggered.

AmplabJenkins · 2015-05-18T19:42:21Z

Merged build started.

SparkQA · 2015-05-18T19:44:27Z

Test build #33011 has started for PR 6238 at commit bcc6668.

SparkQA · 2015-05-18T20:55:51Z

Test build #33011 has finished for PR 6238 at commit bcc6668.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameReader(object):
- class DataFrameWriter(object):

AmplabJenkins · 2015-05-18T20:55:56Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-18T20:55:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33011/
Test FAILed.

andrewor14 · 2015-05-19T00:22:47Z

retest this please

AmplabJenkins · 2015-05-19T00:27:11Z

Merged build triggered.

AmplabJenkins · 2015-05-19T00:27:18Z

Merged build started.

SparkQA · 2015-05-19T00:29:33Z

Test build #33035 has started for PR 6238 at commit bcc6668.

rxin · 2015-05-19T01:06:57Z

core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala

@@ -50,8 +50,15 @@ private[spark] object PythonUtils {
  /**
   * Convert list of T into seq of T (for calling API with varargs)
   */
-  def toSeq[T](cols: JList[T]): Seq[T] = {
-    cols.toList.toSeq
+  def toSeq[T](vs: JList[T]): Seq[T] = {


what does vs mean?

value -> v, values => vs :)

SparkQA · 2015-05-19T01:41:03Z

Test build #33035 has finished for PR 6238 at commit bcc6668.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameReader(object):
- class DataFrameWriter(object):

AmplabJenkins · 2015-05-19T01:41:09Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-19T01:41:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33035/
Test FAILed.

AmplabJenkins · 2015-05-19T04:02:13Z

Merged build triggered.

AmplabJenkins · 2015-05-19T04:02:20Z

Merged build started.

SparkQA · 2015-05-19T04:04:21Z

Test build #33054 has started for PR 6238 at commit f0c5a04.

AmplabJenkins · 2015-05-19T04:12:09Z

Merged build triggered.

AmplabJenkins · 2015-05-19T04:12:15Z

Merged build started.

SparkQA · 2015-05-19T04:12:58Z

Test build #33055 has started for PR 6238 at commit 9cbf01b.

SparkQA · 2015-05-19T05:50:37Z

Test build #33054 has finished for PR 6238 at commit f0c5a04.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameReader(object):
- class DataFrameWriter(object):

AmplabJenkins · 2015-05-19T05:50:42Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-19T05:50:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33054/
Test PASSed.

SparkQA · 2015-05-19T05:58:39Z

Test build #33055 has finished for PR 6238 at commit 9cbf01b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameReader(object):
- class DataFrameWriter(object):

AmplabJenkins · 2015-05-19T05:58:44Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-19T05:58:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33055/
Test PASSed.

rxin · 2015-05-19T06:16:04Z

python/pyspark/sql/readwriter.py

+        else:
+            jwrite.save(path)
+
+    def saveAsTable(self, name, format=None, mode="error", **options):


are there any test cases covering this?

It's covered by df.saveAsTable(), had updated the tests to use new API.

davies · 2015-05-19T19:03:25Z

@rxin @yhai How do we test JDBC?

jkbradley · 2015-05-19T19:03:53Z

@davies I think this has the wrong JIRA number in the title.

davies · 2015-05-19T19:04:51Z

@jkbradley, thanks, corrected.

AmplabJenkins · 2015-05-19T19:07:10Z

Merged build triggered.

AmplabJenkins · 2015-05-19T19:07:18Z

Merged build started.

SparkQA · 2015-05-19T19:09:24Z

Test build #33091 has started for PR 6238 at commit c7200eb.

SparkQA · 2015-05-19T20:52:44Z

Test build #33091 has finished for PR 6238 at commit c7200eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameReader(object):
- class DataFrameWriter(object):

AmplabJenkins · 2015-05-19T20:52:49Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-19T20:52:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33091/
Test PASSed.

cc rxin, please take a quick look, I'm working on tests. Author: Davies Liu <[email protected]> Closes #6238 from davies/readwrite and squashes the following commits: c7200eb [Davies Liu] update tests 9cbf01b [Davies Liu] Merge branch 'master' of github.com:apache/spark into readwrite f0c5a04 [Davies Liu] use sqlContext.read.load 5f68bc8 [Davies Liu] update tests 6437e9a [Davies Liu] Merge branch 'master' of github.com:apache/spark into readwrite bcc6668 [Davies Liu] add reader amd writer API in Python (cherry picked from commit 4de74d2) Signed-off-by: Reynold Xin <[email protected]>

cc rxin, please take a quick look, I'm working on tests. Author: Davies Liu <[email protected]> Closes apache#6238 from davies/readwrite and squashes the following commits: c7200eb [Davies Liu] update tests 9cbf01b [Davies Liu] Merge branch 'master' of github.com:apache/spark into readwrite f0c5a04 [Davies Liu] use sqlContext.read.load 5f68bc8 [Davies Liu] update tests 6437e9a [Davies Liu] Merge branch 'master' of github.com:apache/spark into readwrite bcc6668 [Davies Liu] add reader amd writer API in Python

add reader amd writer API in Python

bcc6668

Davies Liu added 2 commits May 18, 2015 14:23

Merge branch 'master' of github.com:apache/spark into readwrite

6437e9a

update tests

5f68bc8

rxin reviewed May 19, 2015
View reviewed changes

use sqlContext.read.load

f0c5a04

Merge branch 'master' of github.com:apache/spark into readwrite

9cbf01b

rxin reviewed May 19, 2015
View reviewed changes

davies changed the title ~~[WIP] [SQL] [PySpark] add reader amd writer API in Python~~ [SPARK-7338] [SQL] [PySpark] add reader amd writer API in Python May 19, 2015

update tests

c7200eb

davies changed the title ~~[SPARK-7338] [SQL] [PySpark] add reader amd writer API in Python~~ [SPARK-7738] [SQL] [PySpark] add reader amd writer API in Python May 19, 2015

davies changed the title ~~[SPARK-7738] [SQL] [PySpark] add reader amd writer API in Python~~ [SPARK-7738] [SQL] [PySpark] add reader and writer API in Python May 19, 2015

asfgit closed this in 4de74d2 May 19, 2015

[SPARK-7738] [SQL] [PySpark] add reader and writer API in Python #6238

[SPARK-7738] [SQL] [PySpark] add reader and writer API in Python #6238

Uh oh!

Conversation

davies commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

SparkQA commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

AmplabJenkins commented May 18, 2015

Uh oh!

andrewor14 commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

rxin May 19, 2015

Choose a reason for hiding this comment

Uh oh!

davies May 19, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

rxin May 19, 2015

Choose a reason for hiding this comment

Uh oh!

davies May 19, 2015

Choose a reason for hiding this comment

Uh oh!

davies commented May 19, 2015

Uh oh!

jkbradley commented May 19, 2015

Uh oh!

davies commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

AmplabJenkins commented May 19, 2015

Uh oh!

SparkQA commented May 19, 2015

Uh oh!