[SPARK-2674] [SQL] [PySpark] support datetime type for SchemaRDD #1601
Conversation
|
QA tests have started for PR 1601. This patch merges cleanly. |
|
QA results for PR 1601: |
| @@ -395,6 +395,11 @@ class SchemaRDD( | |||
| arr.asInstanceOf[Array[Any]].map { | |||
| element => rowToMap(element.asInstanceOf[Row], struct) | |||
| } | |||
| case t: java.sql.Timestamp => { | |||
There was a problem hiding this comment.
This pair of extra brackets can be removed.
|
QA tests have started for PR 1601. This patch merges cleanly. |
|
QA results for PR 1601: |
java.util.{List,Set} => Seq
java.util.Map => Map
but it can not convert Seq into java.util.Set, so set() and tuple()
and array() can not been handled gracefully (back with the original
type).
We can not access items in ArrayType by position, but this is not defined
for set().
Do we still want to support set()/tuple()/array() ?
|
QA tests have started for PR 1601. This patch merges cleanly. |
|
QA results for PR 1601: |
|
Lets just remove it now. It should be as easy as adding an error and removing the tests in question. |
|
QA tests have started for PR 1601. This patch merges cleanly. |
|
QA results for PR 1601: |
|
cc @kanzhang |
|
I've merged this into master. |
Datetime and time in Python will be converted into java.util.Calendar after serialization, it will be converted into java.sql.Timestamp during inferSchema(). In javaToPython(), Timestamp will be converted into Calendar, then be converted into datetime in Python after pickling. Author: Davies Liu <[email protected]> Closes apache#1601 from davies/date and squashes the following commits: f0599b0 [Davies Liu] remove tests for sets and tuple in sql, fix list of list c9d607a [Davies Liu] convert datetype for runtime 709d40d [Davies Liu] remove brackets 96db384 [Davies Liu] support datetime type for SchemaRDD
Datetime and time in Python will be converted into java.util.Calendar after serialization, it will be converted into java.sql.Timestamp during inferSchema().
In javaToPython(), Timestamp will be converted into Calendar, then be converted into datetime in Python after pickling.