[SPARK-4574][SQL] Adding support for defining schema in foreign DDL commands. #3431

scwf · 2014-11-24T15:14:32Z

Adding support for defining schema in foreign DDL commands. Now foreign DDL support commands like:

CREATE TEMPORARY TABLE avroTable
USING org.apache.spark.sql.avro
OPTIONS (path "../hive/src/test/resources/data/files/episodes.avro")

With this PR user can define schema instead of infer from file, so support ddl command as follows:

CREATE TEMPORARY TABLE avroTable(a int, b string)
USING org.apache.spark.sql.avro
OPTIONS (path "../hive/src/test/resources/data/files/episodes.avro")

SparkQA · 2014-11-24T15:20:05Z

Test build #23787 has started for PR 3431 at commit c203ce2.

This patch merges cleanly.

SparkQA · 2014-11-24T16:32:11Z

Test build #23787 has finished for PR 3431 at commit c203ce2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ParquetRelation2(

AmplabJenkins · 2014-11-24T16:32:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23787/
Test PASSed.

SparkQA · 2014-12-02T15:00:06Z

Test build #24044 has started for PR 3431 at commit 12fab0a.

This patch merges cleanly.

SparkQA · 2014-12-02T16:04:15Z

Test build #24044 has finished for PR 3431 at commit 12fab0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ParquetRelation2(

AmplabJenkins · 2014-12-02T16:04:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24044/
Test PASSed.

marmbrus · 2014-12-02T22:33:56Z

Cool feature :) I wanted to include this in the first cut but ran out of time. It's too late for 1.2 but I'll try and review this soon.

SparkQA · 2014-12-12T07:20:13Z

Test build #24396 has started for PR 3431 at commit 6f1259c.

This patch merges cleanly.

SparkQA · 2014-12-12T08:21:30Z

Test build #24396 has finished for PR 3431 at commit 6f1259c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Analyzer(catalog: Catalog,
- case class ParquetRelation2(

AmplabJenkins · 2014-12-12T08:21:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24396/
Test PASSed.

marmbrus · 2014-12-16T22:53:12Z

sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala

 import org.apache.spark.sql.sources._

 private[sql] class DefaultSource extends RelationProvider {
  /** Returns a new base relation with the given parameters. */
  override def createRelation(
      sqlContext: SQLContext,
-      parameters: Map[String, String]): BaseRelation = {
+      parameters: Map[String, String],
+      schema: Option[StructType]): BaseRelation = {


We cannot change the function signature, otherwise we will break existing libraries. Instead I think we need to create a new interface SchemaRelationProvider maybe?

Or using a default value for schema: schema: Option[StructType] = None ?

Default values do not preserve binary compatibility, only source compatibility.

marmbrus · 2014-12-16T23:07:39Z

Thanks for working on this! I know a couple of people who want to use this in data sources they are writing.

marmbrus · 2014-12-30T07:41:23Z

ping. any progress here?

scwf · 2014-12-30T07:46:14Z

Hi @marmbrus, still working on this, tomorrow i will update this.

SparkQA · 2014-12-30T16:32:33Z

Test build #24897 has started for PR 3431 at commit 44eb70c.

This patch merges cleanly.

SparkQA · 2014-12-30T16:33:25Z

Test build #24897 has finished for PR 3431 at commit 44eb70c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DefaultSource extends SchemaRelationProvider
- case class ParquetRelation2(
- trait SchemaRelationProvider

AmplabJenkins · 2014-12-30T16:33:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24897/
Test FAILed.

SparkQA · 2014-12-30T16:42:30Z

Test build #24898 has started for PR 3431 at commit 02a662c.

This patch merges cleanly.

SparkQA · 2014-12-30T17:51:23Z

Test build #24898 has finished for PR 3431 at commit 02a662c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DefaultSource extends SchemaRelationProvider
- case class ParquetRelation2(
- trait SchemaRelationProvider

AmplabJenkins · 2015-01-09T01:21:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25279/
Test FAILed.

scwf · 2015-01-09T01:30:30Z

retest this please

SparkQA · 2015-01-09T01:32:31Z

Test build #25283 has started for PR 3431 at commit f336a16.

This patch merges cleanly.

yhuai · 2015-01-09T01:36:36Z

sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala

+  protected def cleanIdentifier(ident: String): String = ident match {
+    case escapedIdentifier(i) => i
+    case plainIdent => plainIdent
+  }


Seems when we use ident, the parser will automatically take care backticks. We can remove it. I am sorry I just noticed it.

Thank you:)

SparkQA · 2015-01-09T02:22:47Z

Test build #25287 has started for PR 3431 at commit a852b10.

This patch merges cleanly.

SparkQA · 2015-01-09T02:41:59Z

Test build #25283 has finished for PR 3431 at commit f336a16.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DefaultSource extends SchemaRelationProvider
- case class ParquetRelation2(
- trait SchemaRelationProvider

AmplabJenkins · 2015-01-09T02:42:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25283/
Test PASSed.

SparkQA · 2015-01-09T03:30:23Z

Test build #25287 has finished for PR 3431 at commit a852b10.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DefaultSource extends SchemaRelationProvider
- case class ParquetRelation2(
- trait SchemaRelationProvider

AmplabJenkins · 2015-01-09T03:30:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25287/
Test PASSed.

marmbrus · 2015-01-09T21:01:10Z

sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala

+  def createRelation(
+      sqlContext: SQLContext,
+      parameters: Map[String, String],
+      schema: Option[StructType]): BaseRelation


Why is this an option? we have two traits and option is not very friendly to java

My initial idea is to compatible with the old traits, since we will have two traits i will fix this.

yhuai · 2015-01-10T01:59:11Z

@scwf I have done it and will have a PR to your branch.

yhuai · 2015-01-10T02:00:28Z

scwf#22

Remove Option from createRelation.

scwf · 2015-01-10T02:01:58Z

ok, merged!

SparkQA · 2015-01-10T02:02:44Z

Test build #25354 has started for PR 3431 at commit 7e79ce5.

This patch merges cleanly.

SparkQA · 2015-01-10T03:09:09Z

Test build #25354 has finished for PR 3431 at commit 7e79ce5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait SchemaRelationProvider

AmplabJenkins · 2015-01-10T03:09:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25354/
Test PASSed.

marmbrus · 2015-01-10T21:54:14Z

Thanks for working on this guys! Merging to master.

@yhuai can you clarify the difference between SchemaRelationProvider and RelationProvider in the scala doc in your next PR?

yhuai · 2015-01-10T22:42:42Z

Yeah, no problem.

With changes in this PR, users can persist metadata of tables created based on the data source API in metastore through DDLs. Author: Yin Huai <[email protected]> Author: Michael Armbrust <[email protected]> Closes #3960 from yhuai/persistantTablesWithSchema2 and squashes the following commits: 069c235 [Yin Huai] Make exception messages user friendly. c07cbc6 [Yin Huai] Get the location of test file in a correct way. 4456e98 [Yin Huai] Test data. 5315dfc [Yin Huai] rxin's comments. 7fc4b56 [Yin Huai] Add DDLStrategy and HiveDDLStrategy to plan DDLs based on the data source API. aeaf4b3 [Yin Huai] Add comments. 06f9b0c [Yin Huai] Revert unnecessary changes. feb88aa [Yin Huai] Merge remote-tracking branch 'apache/master' into persistantTablesWithSchema2 172db80 [Yin Huai] Fix unit test. 49bf1ac [Yin Huai] Unit tests. 8f8f1a1 [Yin Huai] [SPARK-4574][SQL] Adding support for defining schema in foreign DDL commands. #3431 f47fda1 [Yin Huai] Unit tests. 2b59723 [Michael Armbrust] Set external when creating tables c00bb1b [Michael Armbrust] Don't use reflection to read options 1ea6e7b [Michael Armbrust] Don't fail when trying to uncache a table that doesn't exist 6edc710 [Michael Armbrust] Add tests. d7da491 [Michael Armbrust] First draft of persistent tables.

marmbrus reviewed Dec 16, 2014
View reviewed changes

scwf added 5 commits December 30, 2014 16:14

draft version

0ba70df

added SchemaRelationProvider

7787ec7

adding test case

9bf12f8

minor fix

83b6fc3

fix decimal parser issue

44eb70c

scwf force-pushed the ddl branch from 6f1259c to 44eb70c Compare December 30, 2014 16:31

style issue

02a662c

yhuai reviewed Jan 9, 2015
View reviewed changes

remove cleanIdentifier

a852b10

marmbrus reviewed Jan 9, 2015
View reviewed changes

yhuai added 2 commits January 9, 2015 15:52

Revert all changes since applying a given schema has not been testd.

65e9c73

Remove Option from createRelation.

38f634e

Merge pull request #22 from yhuai/pr3431yin

7e79ce5

Remove Option from createRelation.

marmbrus mentioned this pull request Jan 10, 2015

[SPARK-5009] [SQL] Long keyword support in SQL Parsers #3926

Closed

asfgit closed this in 693a323 Jan 10, 2015

scwf deleted the ddl branch January 10, 2015 23:23

[SPARK-4574][SQL] Adding support for defining schema in foreign DDL commands. #3431

[SPARK-4574][SQL] Adding support for defining schema in foreign DDL commands. #3431

Uh oh!

Conversation

scwf commented Nov 24, 2014

Uh oh!

SparkQA commented Nov 24, 2014

Uh oh!

SparkQA commented Nov 24, 2014

Uh oh!

AmplabJenkins commented Nov 24, 2014

Uh oh!

SparkQA commented Dec 2, 2014

Uh oh!

SparkQA commented Dec 2, 2014

Uh oh!

AmplabJenkins commented Dec 2, 2014

Uh oh!

marmbrus commented Dec 2, 2014

Uh oh!

SparkQA commented Dec 12, 2014

Uh oh!

SparkQA commented Dec 12, 2014

Uh oh!

AmplabJenkins commented Dec 12, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Dec 16, 2014

Uh oh!

marmbrus commented Dec 30, 2014

Uh oh!

scwf commented Dec 30, 2014

Uh oh!

SparkQA commented Dec 30, 2014

Uh oh!

SparkQA commented Dec 30, 2014

Uh oh!

AmplabJenkins commented Dec 30, 2014

Uh oh!

SparkQA commented Dec 30, 2014

Uh oh!

SparkQA commented Dec 30, 2014

Uh oh!

AmplabJenkins commented Jan 9, 2015

Uh oh!

scwf commented Jan 9, 2015

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!

AmplabJenkins commented Jan 9, 2015

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!

AmplabJenkins commented Jan 9, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yhuai commented Jan 10, 2015

Uh oh!

yhuai commented Jan 10, 2015

Uh oh!

scwf commented Jan 10, 2015

Uh oh!